--- title: Implement CluedIn with AI url: https://documentation.cluedin.net/playbooks/implement-cluedin-with-ai content_type: how-to section: Playbooks --- # Implement CluedIn with AI > how-to: Steps to accomplish one task. **Scope:** How to run CluedIn with AI at the center—using **AI Agents**, **AI Rules**, **AI Enricher**, **AI Mapping**, and **Copilot**—from onboarding to daily operations, with safety, observability, and governance. > TL;DR: Automate suggestions and safe auto-fixes end‑to‑end, keep humans in the approval loop for high‑impact changes, and measure everything. Start read‑only, then progress to **suggest → gated auto‑fix → auto‑promotion** with rollback. --- ## 1) Principles of an AI‑First CluedIn 1. **ELT + AI loops:** Land raw data, then let AI **map**, **profile**, **validate**, **clean**, **dedup**, and **enrich** inside CluedIn. 2. **Guardrails by default:** AI reads **masked views**; approvals required for **PII exposure** or **schema‑breaking** actions. 3. **Human‑on‑the‑loop:** AI proposes → you approve (or promote) with **clear thresholds** and **rollbacks**. 4. **Config as code:** Store **Agent playbooks**, **Rules**, **Enricher templates**, and **Mapping diffs** in Git with PR review. 5. **Measure everything:** Acceptance rate, precision/recall of fixes, incident rate, time‑to‑adopt suggestions, rollback frequency. 6. **Fail safe:** Prefer **flag/quarantine** > **auto‑fix**; require extra proof for structural changes (mappings, exports). --- ## 2) Reference Architecture (AI Control Loop) ``` Ingestion → AI Mapping → Entities ↘ ↗ AI Agents (profile/validate/dedup plan) ↓ AI Rules (validations, policies, fixes) ↓ AI Enricher (lookup/classify/summarize) ↓ Cleaning & Dedup (auto/queued) ↓ Exports → Consumers (BI/Apps) ↓ Copilot (chat + actions + change PRs) ↺ Feedback (metrics, audit, prompts) ``` - **AI Agents:** Orchestrate analysis, generate proposals, triage issues. - **AI Rules:** Convert proposals into executable validations/policies/transform steps. - **AI Enricher:** Add attributes (classifications, categories, standardizations). - **AI Mapping:** Propose/maintain source→entity mappings and diffs. - **Copilot:** Conversational control surface to run, inspect, and ship changes safely. --- ## 3) Quick Start (2 Days) **Day 1** - Enable AI features; point to your provider(s). - Run **AI Mapping** on one source; accept **minimal** mapping. - Run **AI Agents** (read‑only) to produce DQ & dedup findings. - Generate **AI Rules** draft pack; keep all as **flag/quarantine** only. **Day 2** - Turn on **AI Enricher** for 1–2 low‑risk enrichments (e.g., phone E.164, country codes). - Schedule Agents nightly; export findings to a review queue. - Wire **Copilot** to create PRs for mapping/cleaning changes; no direct prod writes. --- ## 4) AI Agents — Analysis & Automation Brain ### 4.1 What to use them for - Profiling: completeness, validity, uniqueness, timeliness. - Rule synthesis: propose validations, cleaning steps, dedup keys. - Risk detection: schema drift, PII leakage, broken joins. - Change notes: auto‑draft PR descriptions and runbooks. ### 4.2 Agent Run (pseudo‑API) ```json POST /api/ai/agents/run { "agent": "dq-analyzer", "target": { "entity": "Person" }, "mode": "analysis", "options": { "sample": 10000, "masked": true } } ``` ### 4.3 Outputs you expect ```json { "issues": [ {"field":"email","type":"invalid_regex","rate":0.031,"examples":["a@x","b@x"]}, {"field":"phone","type":"format_inconsistent","rate":0.22} ], "proposals": { "validations":[ "...yaml..." ], "cleaning":[ "...yaml..." ], "dedup_rules":[ "...yaml..." ] } } ``` ### 4.4 Promotion Policy - **Auto‑create PR** with the proposals → **staging** run → diff & metrics. - **Auto‑promote** only if: tests pass, metrics improve, and risk tag ≠ `high`. - Always retain **correlation_id** and **prompt hash** for audit. --- ## 5) AI Rules — From Idea to Executable Guardrails ### 5.1 What they encode - Validations (regex, domain lists, cross‑field checks). - Quarantine conditions & auto‑fix suggestions. - Policy hints (masking, row filters) for Governance to review. ### 5.2 Template (generated by AI, reviewed by humans) ```yaml # ai_rules/person-email.yaml rule: email_must_match_regex entity: Person severity: high when: [{ field: email, is_not_null: true }] check: { regex: "^[^@\\s]+@[^@\\s]+\\.[^@\\s]+$" } on_fail: action: flag # start with flag; escalate later labels: ["PII","contactability"] tests: - sample_invalids: ["a@x","b@x"] - sample_valids: ["ada@lovelace.org"] ``` ### 5.3 Lifecycle 1. Agent proposes → PR opens. 2. CI validates schema & runs sample tests. 3. Staging export diff; DQ metrics must **not regress**. 4. Human approves or requests changes. 5. Optional: **auto‑promote** next similar rules if precision > threshold. --- ## 6) AI Enricher — Add Signals Safely ### 6.1 Use cases - Normalize: phone → **E.164**, emails → lowercase/trim. - Classify: industry, product category, sentiment, language. - Extract: keywords, geocodes, brand names from free‑text. - Summarize: latest ticket or notes for Copilot context. ### 6.2 Enricher Config (example) ```yaml enricher: standardize_contacts target: Person schedule: "0 * * * *" steps: - name: email_normalize type: ai_enricher mode: transform field: email prompt: "Normalize to lowercase and trim whitespace." reversible: true - name: phone_e164 type: ai_enricher mode: transform field: phone prompt: "Convert to E.164 with default country 'AU' when missing." guardrails: deny_if_changes_pct_over: 0.6 observability: emit_metrics: true sample_before_after: 25 ``` **Guardrails** - **Reversible** writes; keep original in shadow fields. - **Rate limits** and **caching** for any external lookups. - **PII**: redact in prompts or use masked views. --- ## 7) AI Mapping — Source → Canonical, On Autopilot (With Brakes) ### 7.1 Capabilities - Suggest entity classification for a new source. - Propose field‑level mappings with confidence scores. - Generate diffs as sources evolve (rename/add/remove). ### 7.2 Example Proposal (diff) ```yaml mapping: Person source: "crm-contacts" proposal: add: - field: first_name ; from: "$.firstName" ; confidence: 0.94 - field: last_name ; from: "$.lastName" ; confidence: 0.94 change: - field: email ; from: "$.emailAddr" ; confidence: 0.88 unknown: - "$.nickname" ; note: "stash in attributes.* for now" risk: medium ``` ### 7.3 Acceptance Flow - Open PR with **mapping diff** + **staging export**. - Auto‑accept low‑risk adds (nullable fields) if tests pass. - Require human approval for **type changes**, **renames**, or **key semantics**. - Provide **one‑click rollback** and keep version history. --- ## 8) Copilot — Conversational Control Surface ### 8.1 What Copilot should do - “Profile `Person` and propose top 5 fixes.” - “Draft dedup rules deterministic first, then fuzzy.” - “Create an export `customers_wide_v1` (upsert, hourly) with these fields.” - “Explain last export failure and suggest a safe rollback.” ### 8.2 System Prompt (sketch) ``` You are CluedIn Copilot. Prefer minimal, reversible changes. Never expose raw PII in responses; use masked views or aggregates. For structural changes (mapping/exports/policies), open a PR with tests and staging runs. Include a rollback block. ``` ### 8.3 Action Binding (examples) - `/copilot create-export customers_wide_v1 …` → opens PR + staging run. - `/copilot dedup-plan Person` → posts rule YAML and sampling plan. - `/copilot dq-report Person --since 7d` → posts charts + issues. --- ## 9) Governance, Safety & Observability for AI ### 9.1 Policy hooks ```yaml policy: ai_read_masked_by_default target: ai:agents actions: [read] effect: allow_when when: "dataset.view == 'masked'" ``` ```yaml policy: ai_auto_promotion_guard target: ai:proposals actions: [promote] effect: require_approval when: "proposal.impact in ['schema','pii','survivorship']" approvers: ["Data Governance Manager","Administrator"] ``` ### 9.2 Metrics to track - **Suggestion acceptance rate** (weekly). - **Precision of auto‑fixes** (sampled QA). - **False‑positive rate** of validations. - **Rollback count & time to rollback**. - **DQ KPI trend** post‑adoption (validity, duplicates, completeness). - **Prompt drift**: changes in outputs for same inputs. ### 9.3 Audit & Evidence - Store **prompt**, **parameters**, **model/version**, **proposal diff**, **approver**, **correlation_id**. - Snapshot **before/after** samples for fixes. - Export audit packets for reviews. --- ## 10) Maturity Ladder (AI Adoption Path) 1. **Read‑Only Insights**: Agents analyze; Rules generate as drafts; Enricher off. 2. **Human‑Gated Fixes**: Enricher on for low‑risk transforms; Rules flag/quarantine; Mapping diffs PR‑only. 3. **Targeted Auto‑Fix**: Auto‑approve low‑risk Rules/Enrichers with >98% precision; Copilot opens PRs for structure. 4. **Auto‑Promotion**: Time‑boxed auto‑promotion for repetitive, proven changes (e.g., mapping adds); continuous monitoring & fast rollback. 5. **Self‑Optimizing**: Agents retune thresholds based on QA feedback; A/B test competing rule sets. --- ## 11) End‑to‑End Playbooks ### 11.1 New Source Onboarding (AI‑led) 1. **AI Mapping** proposes entity & field map → PR. 2. **Agent** profiles early load → proposes validations/cleaning. 3. **Rules** merged with flag/quarantine only. 4. **Enricher** normalizes low‑risk fields. 5. **Copilot** scaffolds an export contract and staging run. 6. **Promote** once DQ KPIs are stable; set alerts. ### 11.2 Quality Lift Sprint (1 week) - Day 1: Agent report; pick top 3 issues. - Day 2–3: Rules + Enricher steps; staging QAs. - Day 4: Partial rollout with alerts; sample QA. - Day 5: Measure lift; capture lessons; adjust thresholds. ### 11.3 Dedup Program - Agent proposes deterministic keys → queue. - Rules codified; auto‑approve high‑confidence merges. - Copilot generates survivorship spec & reviewer guide. - Weekly precision/recall sampling; refine rules. --- ## 12) CI/CD & Testing for AI Changes - **Unit tests** for prompts: stable I/O examples. - **Golden datasets** for validation drift. - **Contract tests** to prevent schema break. - **Staging export diffs** with row/null distribution comparisons. - **Release notes**: link to audit events and dashboards. **GitHub Actions sketch** ```yaml name: cluedin-ai-changes on: [pull_request] jobs: validate: runs-on: ubuntu-latest steps: - uses: actions/checkout@v4 - run: ./tools/validate-yaml.sh ./ai_rules ./ai_mapping ./ai_enricher - run: pytest -q # prompt/golden tests stage: needs: validate steps: - run: ./tools/apply.sh env/test --ai ``` --- ## 13) Operating Rhythm **Daily:** Agent reports glance, new proposals triage, failed jobs check. **Weekly:** Ship 2–3 AI Rules/Enricher improvements; measure acceptance/precision. **Monthly:** Review auto‑promotion thresholds; audit sample prompts; renew keys/secrets. --- ## 14) “What Good Looks Like” - >95% of low‑risk fixes applied automatically, with <1% rollback. - Mapping changes are **additive** and auto‑promoted with staged safety. - DQ KPIs trend up; duplicate rate drops steadily. - Copilot PRs are the default path for structural change. - Audit shelf is always ready: prompts, diffs, metrics, approvals. --- ## 15) Copy‑Paste Library ### 15.1 Agent Request (DQ) ```json { "agent":"dq-analyzer","target":{"entity":"Person"},"mode":"analysis","options":{"sample":10000,"masked":true} } ``` ### 15.2 Rule: Cross‑Field Consistency ```yaml rule: order_date_precedes_ship_date entity: Order check: { expression: "order_date <= ship_date" } severity: medium on_fail: { action: flag } ``` ### 15.3 Enricher: Country Codes ```yaml - name: normalize_country_code type: ai_enricher field: country_code prompt: "Map common names to ISO_3166_1_ALPHA2. If ambiguous, leave null." ``` ### 15.4 Mapping Diff Template ```yaml proposal_id: "map-2025-08-24-001" entity: Person changes: add: [{ field: "middle_name", from: "$.middleName", confidence: 0.91 }] change: [] remove: [] risk: low ``` ### 15.5 Copilot Command Examples ``` /copilot dq-report Person --since 7d /copilot propose-mapping crm-contacts --entity Person /copilot create-export customers_wide_v1 --upsert --hourly --fields id,email,updated_at /copilot dedup-plan Person --deterministic-first ``` --- **Bottom line:** A 100% AI approach to CluedIn works when you combine **Agent‑generated proposals**, **Rule & Enricher execution**, **Mapping diffs**, and **Copilot‑driven PRs**—all wrapped in policy guardrails, metrics, and fast rollback. Start small, ship often, and let AI handle the toil while humans set the rules of the game. Up: [Playbooks](https://documentation.cluedin.net/playbooks.md)