- Capabilities
- Getting started
- Architecture center
- Platform updates
Add datasets
To begin building a pipeline, add data to your graph using one of the following four methods:
- Data Connection application
- Select a dataset or media set from Foundry
- Manually upload data
- Manually enter data in your pipeline file
Add data to Pipeline Builder from Data Connection
To access data from a data source, navigate to the Data Connection app in the Foundry navigation sidebar. Find the data source you want to integrate, then click Start Pipelining. Choose a location for your new pipeline, then click Save. This will create a new pipeline, and all syncs connected to your data source will be imported to your Pipeline Builder graph.
You cannot save a new pipeline to your personal file folder. Set up the recommended Project structure so that data security and governance are organized from the beginning of your development process.
Add data from Foundry to Pipeline Builder
To import datasets or media sets that already exist in your Foundry filesystem, proceed to the Pipeline Builder application and select Add Foundry data in the center of your graph space. Search for and select an available dataset, then choose Add data.
You can add multiple datasets or media sets by adding each of them and choosing Add to selection; once all are selected, choose Add data.
Upload data from your computer to Pipeline Builder
You can also upload dataset or media set files from your computer. Select Upload from your computer to select the file you want to add, or drag and drop the file onto your graph.
Manually enter data in Pipeline Builder
Input datasets can also be created by defining a data table and manually populating it with data.
Define the new table's schema by selecting column names and types, then manually add values to the table. Manually entered tables can be modified at any point.
The following table lists some of the most common column types in a manual entry table:
| Column type | Format |
|---|---|
| String | All characters |
| Timestamp | mm/dd/yyy hh:mm:ss; additional timestamp formats can be used |
| Date | mm/dd/yyyy |
| Boolean | 0 → false, not 0 → true |
| Binary | All characters, will be shown as base64 |
| Integer, long | Positive and negative numbers, no decimal point |
| Double | Positive and negative numbers, including decimal point |
| Array | A list of values of the same type, entered as [value1, value2] |
After selecting the Array column type, double-click the cell or type [ to open a pop-up where you can add string values to your array. Array string values must be entered this way and cannot be pasted in.
You may experience longer loading times when viewing proposed changes to large manually entered tables. As an alternative, you can view the raw tables side by side.
Next steps
After adding datasets or media sets to Pipeline Builder, you can change their computation mode, choose to transform the data or add outputs.
