How to create a Data Pipe
Step-by-Step Guide
Step 1: Access DataPipes in Orchestrator
On your Orchestrator Home page, go to the DataPipes option.
Step 2: Create a New DataPipe
Now, select the + icon to create a DataPipe.
Step 3: Fill Out the DataPipe Form
A new screen will open where you need to fill it out to create the DataPipe.
The following table explains each part of this form in detail:
| Section | Description |
|---|---|
| Name | In this section, enter the name for your DataPipe. |
| Origin Connection | Select the origin of the data for your DataPipe, which can come from a Data Source or a Repository. |
| Destination Connection | Indicate where you want the DataPipe data to be stored. |
| DataPipe Type | This defines whether you want to extract data from a table in the origin data source to a table in the selected repository or obtain an SQL result, the output of which will be extracted to a table in the repository. This is useful when extracting data that is in different tables in the source. |
| Synchronization Type | This option allows defining whether to extract complete data every time the DataPipe is run or to use an initial complete extraction, and then only extract changed or inserted data after a primary key, a timestamp field, or both. |
| Description | Optionally, you can add a description about the DataPipe you are creating. |
| Group Name | This option allows us to categorize DataPipes and group them by category. |
Requirements
You can only create a DataPipe if a Data Source exists. To learn more about the Data Source, see the documentation Connecting Data Sources.
Step 4: Create the DataPipe
After filling out the form, click the CREATE button.
Step 5: Continue DataPipe Configuration
Next, you will be directed to a new window to continue the DataPipe configuration. In this Name and Type section, you will see the name, configuration type, and description you provided in the previous step.
Step 6: Set DataPipe Data Settings
Select the DataPipe Data Setting option, where you need to choose the table from the Data Origin to copy that data to the Data Destination table.
Save data in the destination table
If you want to save your data in a Data Destination table and it doesn't exist yet, click the CREATE TABLE IN REPOSITORY button to create it in the Data Destination.
Step 7: Configure DataPipe Properties
Select the Properties option. You need to choose the type of SQL function (Insert or Replace), specify if you want the DataPipe to run automatically, and if you want to clear the table each time the DataPipe is executed.
Step 8: View Execution Logs and Save
By clicking Execution Logs, you can see all the executions of the DataPipe. After configuring your DataPipe, click the Save button.
Step 9: Verify the Created DataPipe
After saving, you can see your DataPipe listed.
This way, you have created and configured your DataPipe.
Updated 3 months ago