Ingest data

Files

You can ingest data from CSV, JSON, XLS, and XLSX files.

To ingest data from files

On CluedIn home page, in the Integrations section, select Import from files.
In the Add files section, either drag the file or choose it from your computer. You can upload up to 5 files at once, and the total file size should not exceed 1 GB.
Specify the Group where the data source will be stored. You can choose the existing group or create a new group. If you upload several files, they will be stored as separate data sources within a group.
In the lower-right corner, select Upload.
The data has been sent to CluedIn. You can now view it on the Preview tab of the data set. The next steps involve creating a mapping and processing the data.

Ingestion point

You can ingest a JSON array to an HTTP endpoint created by CluedIn. The process of ingesting data from an ingestion point involves two steps:

Adding an ingestion point
Sending data in HTTP POST request

Add ingestion point

An ingestion point is a channel through which CluedIn can receive data from external sources.

To add an ingestion point

On CluedIn home page, in the Integrations section, select Import from ingestion point.
The Import from ingestion point pane opens, where you can choose the group for storing the data source and define preliminary mapping configuration.
On the Configure tab, do the following:
1. Enter the Name of the data source.
2. Specify the Group where the data source will be stored. You can choose the existing group or create a new group.
3. In the lower-right corner, select Next.
On Add ingestion point tab, do the following:
1. Enter the Endpoint name that will be used as the name of the data set.
2. Select the Mapping configuration option:
  - New mapping – you can create a new mapping for the data set. If you choose this option, you need to select the existing entity type or create a new one. If you create a new entity type, select an icon to visually represent the entity type.
  - Existing mapping – you can reuse the mapping from the data set that has the same structure. If you choose this option, you need to indicate the data set with the required mapping configuration. To do that, choose the following items one by one: a data source group, a data source, and a data set.
3. In the lower-right corner, select Add.
The ingestion point is added to CluedIn. It has a label No data sent, which indicates that CluedIn has not received data for this ingestion point.
Now, you can send data to CluedIn by creating HTTP POST requests.

Send data

In this section, Postman is used as a tool to demonstrate how to send an HTTP POST request to CluedIn.

Prerequisites

To be accepted by CluedIn, your HTTP POST request should meet the following prerequisites:

The request’s authorization type should be set to Bearer Token and the value should be a valid API token from CluedIn. You can find the API token in CluedIn in Administration > API Tokens.
The request’s body should contain raw data in JSON format.
The content-type in the request’s header should be set to application/json.

To send data to CluedIn

In CluedIn, open the data set that was created in the previous procedure, and then select View instructions.
On the Ingestion point instructions pane, find and copy the POST URL that you can use to send data to CluedIn.
In Postman, paste the URL that you copied to the URL input field of your request. Then, send the request.
The data has been sent to CluedIn. You can now view it on the Preview tab of the data set. The next steps involve creating a mapping and processing the data.

Database

You can ingest the database tables to CluedIn if you have the read permission to these database tables. The process of ingesting data from the database involves two steps:

Adding connection to the database
Ingesting the database tables

Prerequisites

To be able to ingest the data from a database, go to Administration > Feature Flags and make sure that the Import databases in Data Source module feature is enabled.

Add connection to database

To be able to access the database tables in CluedIn, first establish a connection to the database.

To add a connection to the database

On CluedIn home page, in the Integrations section, select Import from database.
The Import from database pane opens, where you can provide the database connection details and choose the group for storing the data source.
On the Connection string tab, do the following:
1. Choose the SQL database technology to query the data (Microsoft SQL Server, MySQL, or Postgres).
2. Enter the database connection details such as Host, Database name, Username, and Password. Optionally, you may add Port number.
3. In the lower-right corner, select Test connection. After you receive a notification that the connection is successful, select Next.
On the Configure tab, do the following:
1. Enter the Name of the data source.
2. Specify the Group where the data source will be stored. You can choose the existing group or create a new group.
3. In the lower-right corner, select Add.
The database connection is added to CluedIn.
Now, you can add database tables to CluedIn.

Ingest database tables

With an established connection to the database, you can choose which database tables you want to ingest into CluedIn.

To ingest database tables

Open the data source, and then select Add new table.
The Add data table pane opens, where you can view all tables existing in the database.
Select the checkboxes next to the tables you want to ingest into CluedIn. Then, in the lower-right corner, select Add.
The tables are added to CluedIn. Each table is added as a separate data set. Next, create mapping and process data.

Preview

After you ingest the data, it is displayed on the Preview tab as a table.

If you want to focus on specific columns and hide the others, select Column Options, and then clear the checkboxes next to the columns that you want to hide from the table.