Ingest

How do I get data into CluedIn?

Data engineers and anyone connecting a source system.

Everything that brings records into CluedIn lives here: connecting a source, describing what you want to ingest, mapping source columns onto the model, and processing the result into golden records.

The usual order is Data sources -> Mapping -> Processing. Use Troubleshoot ingestion when records do not arrive or land in quarantine.

In this section

  • Tutorial: ingest data tutorial
  • Data sourcessectionA data source is where records come from.
  • MappingsectionMapping decides which source column becomes which vocabulary key, which value identifies the record, and what business domain it belongs to.
  • ProcessingsectionProcessing turns mapped records into data parts and golden records.
  • Troubleshoot ingestionsectionWhen records do not arrive, or arrive wrong, these are the three places to look: the quarantine holds records that failed validation, the logs say what the source did, and monitoring shows whether processing is keeping up.
  • Manual data entrysectionSome data has no source system - it lives in someone's head or a spreadsheet.
  • ConnectorssectionA connector crawls an external system and turns what it finds into clues.
  • Ingestion patternssectionWorked patterns for feeding CluedIn from common parts of a modern data stack, and for the case where you have many systems rather than a few.

Table of contents

  • Tutorial: ingest data
  • Data sources - A data source is where records come from.
  • Mapping - Mapping decides which source column becomes which vocabulary key, which value identifies the record, and what business domain it belongs to.
  • Processing - Processing turns mapped records into data parts and golden records.
  • Troubleshoot ingestion - When records do not arrive, or arrive wrong, these are the three places to look: the quarantine holds records that failed validation, the logs say what the source did, and monitoring shows whether processing is keeping up.
  • Manual data entry - Some data has no source system - it lives in someone's head or a spreadsheet.
  • Connectors - A connector crawls an external system and turns what it finds into clues.
  • Ingestion patterns - Worked patterns for feeding CluedIn from common parts of a modern data stack, and for the case where you have many systems rather than a few.