# Architecture and Product Decisions Log Record significant decisions here with date, context, decision, and consequences. Minor implementation details belong in task files or the changelog. ## How to add an entry ```markdown ### YYYY-MM-DD — Short title **Context:** Why we needed to decide. **Decision:** What we chose. **Consequences:** Tradeoffs, follow-ups, doc updates. ``` --- ## Initial decisions (project kickoff) ### 2026-06-02 — Database: MongoDB **Context:** CRM entities vary in shape (companies, people, custom import fields, campaign metadata). Relational migrations for early-stage iteration would slow delivery. **Decision:** Use **MongoDB** for flexible CRM and marketing data with Mongoose (or equivalent) schemas for validation. **Consequences:** Document all collections in [02-database-schema.md](./02-database-schema.md). Plan indexes for lookup-heavy fields (email, company name, lead status). --- ### 2026-06-02 — Email: internal sending engine **Context:** Campaign and transactional email must be reliable and auditable within the platform. **Decision:** Build an **internal email sending engine** (SMTP) instead of relying only on n8n for outbound mail. **Consequences:** Implement SMTP configuration via env vars; n8n may still trigger side effects (notifications, external systems). --- ### 2026-06-02 — Automation: n8n as layer, not CRM core **Context:** n8n is strong for workflows but weak as a customer database of record. **Decision:** Use **n8n as the automation layer** (webhooks in/out), not as the main CRM. **Consequences:** CRM state lives in MongoDB; webhooks are configurable via `N8N_WEBHOOK_*` env vars. See [09-n8n-integration.md](./09-n8n-integration.md). --- ### 2026-06-02 — AI: external preparation first **Context:** Market lists may need cleaning and classification before import. **Decision:** Use **GPT/AI outside the system initially** for smart data preparation; in-app classification comes in Phase 8. **Consequences:** Import pipeline accepts processed Excel; [08-ai-classification.md](./08-ai-classification.md) defines later in-app scope. --- ### 2026-06-02 — Data ingest: processed Excel import **Context:** Anwal market database work often starts in spreadsheets. **Decision:** **Import processed Excel data** into the system via a dedicated import module (Task 005). **Consequences:** Define import mapping, validation, and deduplication rules in docs and tasks before bulk loads. --- ### 2026-06-02 — Secrets: environment only **Context:** Multiple integrations (DB, SMTP, IMAP, OpenAI, n8n, Redis). **Decision:** Keep **all external credentials in `.env`** (and hosting provider secrets in production). Never commit secrets. **Consequences:** Maintain [10-environment-variables.md](./10-environment-variables.md) as the canonical list. --- ### 2026-06-02 — shadcn/ui deferred **Context:** Task 002 focuses on foundation only; shadcn init adds dependencies and config surface. **Decision:** **Defer shadcn/ui** until after app shell/auth (Task 003+) or first CRM UI task needing shared components. **Consequences:** `src/components/ui/` reserved; Tailwind-only UI for foundation home page. --- ### 2026-06-02 — Auth: NextAuth credentials + JWT **Context:** Task 003 requires internal login without external IdP yet. **Decision:** Use **NextAuth.js v4** with **Credentials provider** and **JWT sessions**; users stored in MongoDB with **bcrypt** password hashes. Initial admin via `POST /api/admin/bootstrap` with `BOOTSTRAP_SECRET` header. **Consequences:** Roles in session for future RBAC; OAuth/SSO can be added later as providers. --- ### 2026-06-02 — Mongoose field renames (Document conflicts) **Context:** TypeScript build failed when interface fields `model` and `errors` extended Mongoose `Document`. **Decision:** Store as `providerModel` on `AiClassification` and `errorMessages` on `ImportLog`; document mapping in schema docs. **Consequences:** API/import code must use these names; spec synonyms documented in `02-database-schema.md`. --- ### 2026-06-02 — Monolithic model layer under server/models **Context:** Task 004 requires 20 collections supporting import, CRM, campaigns, inbox, AI, and automation. **Decision:** All Mongoose models live in `src/server/models/` with `index.ts` barrel; enums centralized in `src/lib/enums.ts`. **Consequences:** Services in Task 005+ import from `@/server/models`; no duplicate enum strings in services. --- ### 2026-06-02 — Import: dual path + row-level resilience **Context:** Market data arrives as raw Excel or GPT multi-sheet workbooks. **Decision:** **Simple** (mapped single sheet) and **processed** (Entities/People/Contact_Points sheets) import services; row errors collected without aborting entire job; dedup by name/email/website with `DuplicateReview` when uncertain. **Consequences:** See [18-import-system.md](./18-import-system.md); `Campaign_Ready` deferred. --- ### 2026-06-02 — Market Database Module (replaces Companies Module direction) **Context:** Imported market data (companies, people, contacts, service matches, quality flags) is the primary asset after import. Full manual CRUD for companies was planned as Task 006 but does not match the immediate user need. **Decision:** Reframe Task 006 as **Market Database Module** — a browse/search/filter layer over imported data, with duplicate review and data quality workflows. Manual entity CRUD remains deferred to later tasks. **Consequences:** Routes live under `/market` and `/api/market/*`. Old `/companies` CRUD specs in docs remain planned for future tasks. Tasks 007–008 may be narrowed to manual CRUD only. --- ### 2026-06-02 — Segments moved before People CRUD **Context:** Campaign orchestration depends on target selection first. Imported data is already browseable through Market Database. **Decision:** Move Segments before manual People CRUD and execute Segments as Task 007. **Consequences:** `/segments` and `/api/segments/*` become available earlier; People CRUD is deferred to a later task. --- ### 2026-06-02 — Production-ready import engine **Context:** Imports felt stuck (long HTTP wait), duplicated records on re-import, and results UI dumped huge JSON. **Decision:** Layered dedupe + fill-missing enrichment, `ImportLog` progress fields, async execute (`after()` + status polling), compact summary UI. No Duplicate Review / Data Quality modules. **Consequences:** `GET /api/imports/status/:id`; execute returns `202` with `importLogId`. Re-import increments skipped/updated counts instead of creating duplicates when data unchanged. --- ### 2026-06-02 — Simple import: semantic auto-mapping **Context:** Simple Excel imports required manual JSON column mapping; real files use varied English/Arabic header names. **Decision:** Add layered semantic mapping (`import-mapping.service.ts`) with auto-detect on detect/preview/execute; keep editable override in UI. Processed GPT imports unchanged. **Consequences:** Mapping optional in API; minimum one identity field required. Docs updated in [18-import-system.md](./18-import-system.md). --- ### 2026-06-02 — Market data: profiles and smart segments **Context:** Companies, People, and Contact Points pages were passive tables with limited CRM actionability. Segment filters used free-text fields disconnected from imported data. **Decision:** Upgrade market pages to **actionable entity profiles** (company/person detail with related records and quick actions) and **smart segment builders** (filter-options from DB, usable static segments with bulk add from list tables). **Consequences:** New routes `/market/people/[id]`, segment member APIs, shared market components (`entity-quick-actions`, `bulk-segment-toolbar`). Dynamic segment behavior unchanged. Docs and task file `tasks/task-018-entity-profiles-smart-segments.md`. --- ## Future decisions (Add new entries below as the project evolves.)