REST

Foundry can integrate with external systems that expose a REST (representational state transfer) API. You may need to use a different approach depending on whether you are syncing, exporting, or interactively calling REST APIs. On this page you can find several connection options for secure and efficient integration with REST APIs.

REST API source

The REST API source may be used for workflows requiring interactive HTTP requests to external systems directly from Foundry applications via Actions. For example, you can create a Workshop application with a button that uses a webhook to calls a REST endpoint when clicked, connecting that application to existing workflows and source systems.

Webhooks to HTTP endpoints should use the REST API source type in Data Connection. You will need to configure the base URL, authentication, and an optional port.

The example configuration below shows how to configure a connection to https://my-domain.com using bearer token authentication.

Learn more about Webhooks in Foundry.

External transforms in Code Repositories

Use external transforms to configure syncs and exports that require you to call REST APIs. Simply import a source in a Python Code Repository and write custom logic to query the API.

Learn more about calling APIs from code repositories.

Examples

The examples below show common patterns of complex external transforms.

OAuth Client Credentials grant

The OAuth Client Credentials grant ↗ is a common authentication pattern for non-interactive, service-to-service REST API calls. The client exchanges a client_id and client_secret at the OAuth2 server's token endpoint for a short-lived access token, then includes that token as a bearer token in the Authorization header of every subsequent request to the resource API.

Before writing the transform, configure the REST API source with the OAuth2 token endpoint domain and the resource API domain, and store the client_id and client_secret as additional secrets on the source. Do not hard-code secrets in your transform.

The request to the token endpoint must use the application/x-www-form-urlencoded content type with at least the following parameters:

Once you have an access token, include it in the Authorization header of every request to the resource API:

Authorization: Bearer <access_token>

Many APIs return results in pages. A common pattern uses a nextPageToken field in the response to indicate there are more results; your transform should loop until no nextPageToken is returned.

Basic client_id/client_secret example

The following example requests an access token and calls a paginated resource API. It uses generic placeholders so that it can be adapted to any OAuth2-protected REST API.

Copied!
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
import logging

import pandas as pd
from transforms.api import Output, transform_pandas
from transforms.external.systems import external_systems, Source, ResolvedSource

logger = logging.getLogger(__name__)


@external_systems(
    api_source=Source("<source_rid>")
)
@transform_pandas(
    Output("<output_dataset_rid>"),
)
def compute(api_source: ResolvedSource) -> pd.DataFrame:
    base_url = api_source.get_https_connection().url
    client = api_source.get_https_connection().get_client()

    client_id = api_source.get_secret("additionalSecretClientId")
    client_secret = api_source.get_secret("additionalSecretClientSecret")

    token_response = client.post(
        base_url + "/oauth/token",
        data={
            "grant_type": "client_credentials",
            "client_id": client_id,
            "client_secret": client_secret,
            "scope": "<space-separated-scopes>",
        },
        headers={"Content-Type": "application/x-www-form-urlencoded"},
    )
    token_response.raise_for_status()
    access_token = token_response.json()["access_token"]

    auth_headers = {"Authorization": f"Bearer {access_token}"}

    results = []
    page_token = None
    while True:
        params = {"pageSize": 100}
        if page_token:
            params["pageToken"] = page_token

        response = client.get(
            base_url + "/api/v1/<resource>",
            headers=auth_headers,
            params=params,
        )
        response.raise_for_status()
        body = response.json()

        results.extend(body.get("data", []))

        page_token = body.get("nextPageToken")
        if not page_token:
            break

        logger.info(f"Fetched {len(results)} items so far, continuing to next page.")

    return pd.DataFrame(results)

To adapt this pattern, replace the token endpoint path, the resource URL, and the scope values with those required by the target API. Some OAuth2 servers require additional parameters such as audience or resource — add them to the data dictionary of the token request.

If the target API is the Foundry API itself (a Foundry-to-Foundry call from a transform), see Call the Foundry API from code for the Foundry-specific token endpoint, scopes, and setup steps.

JWT client-assertion variant (NetSuite)

Some OAuth2 servers — including NetSuite — require the client to authenticate with a signed JWT client assertion instead of a plain client_secret. The following example updates account names in NetSuite from an input dataset of accounts, using the POST /account ↗ endpoint. To enable the grant type, the client_id, certificate_id, and certificate_private_key are added to the source as additional secrets.

Copied!
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
100
101
102
103
104
105
106
107
108
109
110
111
from transforms.api import (
    transform,
    Output,
    Input,
    TransformInput,
    TransformOutput,
    TransformContext,
)
from transforms.external.systems import external_systems, Source, ResolvedSource
import datetime
import jwt
from urllib.parse import urljoin
import logging

logger = logging.getLogger(__name__)

@external_systems(
    netsuite_rest_source=Source("<source_rid>")
)
@transform(
    output=Output("<dataset_rid>"),
    account_updates=Input("<dataset_rid>"),  # Dataset with schema [account_id: String, account_name: String]
)
def update_account_names(
    netsuite_rest_source: ResolvedSource,
    account_updates: TransformInput,
    output: TransformOutput,
    ctx: TransformContext,
):
    # --- Set up connections and secrets ---
    base_url = netsuite_rest_source.get_https_connection().url
    client = netsuite_rest_source.get_https_connection().get_client()
    client_id = netsuite_rest_source.get_secret("additionalSecretClientId")
    certificate_id = netsuite_rest_source.get_secret("additionalSecretCertificateId")
    certificate_private_key = netsuite_rest_source.get_secret("additionalSecretPrivateCertificate")

    # --- Helper: Make JWT token ---
    def make_jwt_token(
        url, client_id, certificate_id, certificate_private_key, lifetime_in_minutes=59
    ):
        current_timestamp = datetime.datetime.now()
        expiration = current_timestamp + datetime.timedelta(minutes=lifetime_in_minutes)

        payload = {
            "iss": client_id,
            "scope": "rest_webservices",
            "aud": url,
            "iat": current_timestamp,
            "exp": expiration,
        }

        additional_headers = {
            "kid": certificate_id,
        }

        return jwt.encode(
            payload,
            certificate_private_key,
            algorithm="ES256",
            headers=additional_headers,
        )

    # --- Helper: Get OAuth2 access token ---
    def get_oauth2_access_token():
        url = urljoin(base_url, "/services/rest/auth/oauth2/v1/token")
        payload = {
            "grant_type": "client_credentials",
            "client_assertion_type": "urn:ietf:params:oauth:client-assertion-type:jwt-bearer",
            "client_assertion": make_jwt_token(
                url,
                client_id,
                certificate_id,
                certificate_private_key,
            ),
        }
        headers = {"Content-Type": "application/x-www-form-urlencoded"}
        response = client.post(url, data=payload, headers=headers)
        return response.json()["access_token"]

    # --- Prepare data for update ---
    account_update_data = [
        {
            "account_id": row.account_id,
            "payload": f'{{"acctName": "{row.account_name}"}}',
        }
        for row in account_updates.dataframe().collect()
    ]

    # --- Update accounts ---
    token = get_oauth2_access_token()
    headers = {
        "Content-Type": "application/json",
        "Authorization": f"Bearer {token}",
    }

    responses = []
    for account in account_update_data:
        account_id = account["account_id"]
        payload = account["payload"]
        logger.info(f"Updating account: {account_id} with payload {payload}")
        url = urljoin(base_url, f"/services/rest/record/v1/account/{account_id}")
        response = client.patch(url, data=payload, headers=headers)
        responses.append(
            {
                "account_id": account_id,
                "response_status": response.status_code,
                "response": response.text,
            }
        )

    output.write_dataframe(ctx.spark_session.createDataFrame(responses))

Use self-signed server certificates

On-premise systems sometimes use self-signed server certificates that must be added to the source for the connection to be trusted. These certificates are typically added automatically to the built-in HTTPS client provided in external transforms. However, some Python clients might rely on the REQUESTS_CA_BUNDLE environment variable. In these cases, you will need to override the variable.

The example below demonstrates how to override the REQUESTS_CA_BUNDLE to read data from an on-premise SharePoint source using the Python client for SharePoint ↗ Office365-REST-Python-Client, which is a required step to use the client.

Copied!
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
from pyspark.sql import DataFrame
from transforms.api import Output, transform, lightweight
from transforms.external.systems import external_systems, Source
import pandas as pd
import polars as pl
import tempfile
import os
from office365.sharepoint.client_context import ClientContext

@lightweight
@external_systems(
    sharepoint_rest=Source("<source_rid>")
)
@transform(
    output=Output("<dataset_rid>"),
)
def compute(ctx,  output, sharepoint_rest) -> DataFrame:

    # 1. Add custom certificates to default certificates environment variable
    cert_file = tempfile.NamedTemporaryFile(delete=False)
    with open(cert_file.name, 'w') as tmp_f:
        with open(os.environ.get("REQUESTS_CA_BUNDLE"), 'r') as ca_f:
            with open(sharepoint_rest.server_certificates_bundle_path, 'r') as source_ca_f:
                tmp_f.write(ca_f.read())
                tmp_f.write(source_ca_f.read())
    cert_file.close()

    os.environ["REQUESTS_CA_BUNDLE"] = cert_file.name # the REQUESTS_CA_BUNDLE now contains the source self-signed certificate


    # 2. Connect to Sharepoint using client certificate authentication.
    client = ClientContext("<sharepoint_url>").with_client_certificate(
        tenant="<tenant_id>",
        client_id="<client_id>",
        thumbprint="<thumbprint>",
        private_key=sharepoint_rest.get_secret("additionalSecretPrivateKey"),
        passphrase=sharepoint_rest.get_secret("additionalSecretPrivateKeyPassphrase"), # optional, if the private key is password encrypted
    )

    # 3. Grab web title and return it as DataFrame
    current_web = client.web
    client.load(current_web)
    client.execute_query()

    data = [{"web_title": current_web.properties['Title']}]
    output.write_table(pl.from_pandas(pd.DataFrame.from_records(data)))

Foundry REST API

For cases where you want to build applications on top of the Foundry platform, use the Foundry REST API. The Foundry API uses the OAuth 2.0 protocol for authentication, primarily uses JSON requests and responses, and provides support for Ontology and Modeling resources.

Learn more about the Foundry API.