🚧
Requirements
A DataPipe can only be created if a Data Source exists. For more information on Data Sources, refer to the Connecting Data Sources documentation.Connecting Data Sources.

Step-by-Step Guide

Step 1: Access DataPipes in Orchestrator

On the Orchestrator Home page, navigate to the AI Workshop option.

Alternatively, access it using the toolbar.

Step 2: Create a New Workshop Object

🚧
Note
You must have an existing Workshop or create a new one. For more details, refer to the documentation.AI Workshop.

Fill out the form to create a new Workshop.

Enter the Workshop to configure additional settings and create objects.

Step 3: Create a New Object

Complete the form with these configurations:

Choose an Extract & Load Object
Select an AP DataPipe Engine
Save the configuration

Step 4: Set the DataPipe Fields – Source and Destination

A new screen will appear, where you can configure the DataPipe.

The following table explains each configuration in detail:

Data Origin	Description	Data Destination	Description
Data Source	Defines the connection to an external system used to obtain data	Data Source	Specifies where data extracted from the source will be stored
Select Table	Indicates which table from the data source will be loaded	Select Table	Indicates where the data from the source table will be stored
Create Table in Repository	Generates customizable SQL code based on the Data Origin table to create a destination table in the repository

Select the Data Source for both Origin and Destination.

Select the source table and create a destination table if a compatible one doesn't exist.

Confirm that all fields are properly mapped according to the source structure.

Step 5: Configure DataPipe Settings

Type Configuration

Defines how data will be handled based on the selected configuration type:

Table ➝ Table
Copies data directly from a source table to a target table. Ideal for 1:1 mappings with minimal transformation.
SQL ➝ Table
Uses a custom SQL query to extract, filter, or transform data before loading it into the target table. Best for complex logic or joins.

🚧
Important
When selecting the SQL ➝ Table option, you must provide a SQL query to fetch the desired data from the source.

Synchronization Type

Full Pull
Loads all data from the source each time, regardless of changes.
SQL ➝ Table
Loads only records that have changed since the last sync, based on a field such as updated_at.

DataPipe Settings

Insert
Adds new records to the target table without modifying existing data. This method assumes data is additive and does not check for duplicates.
Replace
Overwrites existing records based on a matching key (e.g., primary key or unique constraint). Ideal when data may change and requires refreshing in the target table.

Step 6: Save Settings and Run

After configuring everything, run the object to verify the setup and troubleshoot any issues.

Click the Run button.
Click the run icon in the console.

If there are no errors, the output should look like this:

Step 7: Scheduling the Process

Go back to the Workshop screen
Click on the Workshop options
Select Schedule Settings

Fill out the form and configure the desired execution schedule

How to create a Data Pipe to MySQL