Configure logging [Beta]

Applications across Foundry emit logs that provide visibility into their operation, including transform runs, build failures, and other system events.

Foundry logs contain information about:

• Application and function execution events
• Transform and pipeline runs
• Build and deployment activities
• System errors and warnings

Additionally, any user-added logs written in Code Repositories will also be emitted.

These logs can be exported to a Foundry streaming dataset for real-time monitoring and analysis. This feature is only available to Organization Administrators, who can configure a destination where logs from Foundry applications will flow into a Foundry streaming dataset.

Along with log export management, Ontology and AIP in platform log access is managed by Organization Administrators and Information Security Officers.

To start managing the log observability settings for your organization:

1. Navigate to the Control Panel application.
2. In the top banner, select the relevant organization from the dropdown.
3. In the left-hand sidebar, select Log observability settings under the Organization section. Alternatively, in the home page, search for Log observability settings and select the relevant page in the search results.

Export Foundry logs

Foundry logs can be exported, per-organization, directly into a Foundry streaming dataset.

After defining an export configuration, log entries will be continuously written to the streaming dataset from the time the configuration is created. You can process and analyze the data in real-time using Foundry's extensive suite of data transformation and visualization tools or export the logs to external monitoring systems.

Export permissions

To export Foundry logs, a user must be an Organization Administrator. This role provides the necessary permissions to configure log exporting destinations for the organization. See organization permissions for more details about organization-level permissions.

Log data might contain sensitive information. We recommend applying security markings to the output streaming dataset to control access to this data. You can apply markings directly during export creation or add them to the dataset afterward.

Define an export configuration

Foundry logs can be exported from resources belonging to a set of projects, or from all resources in the organization. Some limitations apply when defining an export configuration:

• Each organization can have a maximum of 50 export configurations.
• Each project can be included in at most three export configurations.
  • This limit includes configurations that export logs from all projects.
• When selecting a set of projects for a configuration, you can include up to 100 projects per configuration.

Follow the steps below to define an export configuration:

1. Select Create log export, which will open a dialog to define a configuration.

2. Select the projects to export relevant logs from. You can filter the project list by space using the dropdown, or search for projects by name. Projects that are already included in other export configurations display a count indicating the number of existing exports.

3. Specify the name and location of the streaming dataset where logs will be exported. The location must be within the Foundry file system and cannot be a user-created folder.

4. Select a log schema format. Foundry provides two schema types: an internal Palantir format that provides the payload as a serialized JSON string, or the OpenTelemetry (OTel) protocol format with the payload serialized as a protocol buffer binary. In the same step, use the Unredact user IDs toggle to control whether user IDs are redacted in exported logs. User IDs are redacted by default.

5. In the Apply markings step, select the security markings to apply to the output streaming dataset. This step is optional but recommended to control access to log data, as that data may contain sensitive information. Selected markings will be applied to the resulting log dataset after the export configuration is created.

6. Select Create configuration after reviewing the summary and access requirements. The summary displays the selected markings along with the other configuration details, so you can confirm your choice before creating the export.

7. After the configuration is created, it will appear in the list of existing configurations. Any applied markings are displayed alongside the export configuration details.

Note that when first configuring a log export, it may take up to five minutes for logs to begin streaming into the dataset at the export location.

Disable log exporting

To disable log exporting, delete any listed log storage configurations by selecting the trash icon on the right side of the configuration. The resulting dataset will continue to exist, but no new logs will be exported to it. Additionally, if the streaming dataset is in the trash or permanently deleted, logs will stop being exported.

Analyze log data

Log storage datasets can contain high volumes of streaming data, so we recommend filtering the dataset appropriately before performing analysis. Consider time-based filtering to focus on relevant log entries for your monitoring and troubleshooting needs.

Foundry provides many powerful tools for performing log analysis, such as Transforms and Pipeline Builder.

Log schema

Foundry application logs are structured logs that follow a consistent schema, making them suitable for programmatic analysis and monitoring.

Specifically, Foundry logs contain information about various application events, including transform executions, build processes, system errors, and other operational activities. The log structure varies depending on the type of application or service that generates the log entry.

Application-specific information is captured within structured fields that provide context about the operation being performed, including execution details, error information, and performance metrics.

Foundry provides two schema types:

• An internal Palantir format that provides the payload as a serialized JSON string
• The OpenTelemetry (OTel) protocol provides logs in OTel protocol ↗ format, with the payload serialized as a protocol buffer binary for ingestion downstream to an OTel collector.

Log attribution

Foundry application logs are generated by various services and applications within the platform. When logs are streamed to the configured dataset, they are filtered to include only logs relevant to the organization that configured the log storage configuration.

Log exports configured for a given organization will receive logs from applications and services used by that organization. This includes transform executions, builds, and other activities performed within the organization's context.

The RID of the resource that generated the log is included in the log entry and can be used to trace the log back to the resource.

Log guarantees

Foundry logs are not audit logs. There is no guarantee of 100% reliable log delivery.

Log contents produced by Foundry are subject to change without notice.

In platform log access for Ontology and AIP workflows

Ontology and AIP in platform log access is managed at the project level. The log access controls for in platform viewing are distinct from the permissions configured for the export streaming dataset. These controls pertain specifically to functions, actions, automation workflows, and language models called from enabled projects.

You can configure in-platform log access from Control Panel or directly from a resource node in Workflow Lineage.

A user must have the following to view logs (with 30-day retention) they did not invoke:

• Log reading enabled for the project the source executor resides in, or through a resource-level override.
• Markings access to any markings configured in the project log access settings or on a resource-level override.
• Edit permission on the source executor the logs were emitted under.

Specifically, the Edit permission requirement corresponds to the foundry-telemetry-service:view-execution-history operation, which the Editor role grants by default. You can grant this operation through a different role using custom roles.

For the authoritative list of viewing requirements and the roles they map to, see Log permissions.

Configure log access from Control Panel

To manage the in platform log access controls by project:

1. Navigate to the Log access tab. This tab displays the projects with log reading enabled for your organization, if any. By default, log access is disabled for all projects within an organization.

2. To enable log access for a project, select Add project, which will open a dialog to configure telemetry log visibility for a project.

3. Once a project is selected, a second dialog will appear to enable log reading for the project and apply necessary markings.

4. After log access is enabled for the project, it will appear in the list of enabled projects.

5. You should then see the updated policy reflected throughout the platform, with log viewing enabled for all source executor resources residing in the project.

Configure log access from Workflow Lineage

In addition to the Control Panel flow, an administrator can review and adjust a resource's log access from the Log access overview dialog in Workflow Lineage. Open the dialog as described in Log permissions, then select Edit on the Log access policy requirement to open the configuration step.

We recommend project-level log access. Select Configure in Control Panel to enable it using the Control Panel flow.

The configuration step shows a table summarizing where log access is resolved for the resource:

Update project attribution

A source executor is attributed to a project for log access purposes. By default, the attributed project is the project where the source executor was located when it first wrote a log. If a resource is moved after writing its first log, the current project and the attributed project differ, and log access is enforced against both.

When the two projects differ, an administrator can re-point attribution to the current project: hover over the previous project's status tag and select Clear and inherit.

Configure resource overrides for legacy Ontology permissions

You can manage an action at the project level only after migrating it to Ontology project-based permissions. Actions still managed by legacy Ontology permissions must first be migrated to project-based permissions. Once an action is migrated, an administrator updates its project attribution by clearing the legacy Ontology resource identifier.

For an action on legacy Ontology permissions, the configuration step offers two paths: migrate the action to a project (recommended), or configure a resource override. The action's Ontology resource identifier appears in place of the project; administrators cannot enable it and have no access to it through Foundry's project and file system.

Until the action is migrated, you can enable log access by activating a resource override and applying the Required markings that viewers must hold to read the resource's logs. The markings are applied to the logs on top of any markings already present on the resource and project log settings.

Disabled resource overrides are deprecated. If the table shows a disabled override, remove it and rely on the project-level policy; the dialog blocks Next until you do.

After configuring the policy, select Next to continue to the acknowledgment step.

Acknowledge changes

Before changes take effect, you must select an acknowledgment checkbox confirming the security trade-off. When enabling an override, you confirm that sensitive object data may be logged and that you applied the correct markings. When clearing an override, you confirm that the resource reverts to project-level log visibility. After selecting the checkbox, choose Apply Changes to save.

Delete log history

Users with the Information security officer or Enrollment administrator role can delete logs at any time. From the Run history or Log search panel in Workflow Lineage, select Edit permissions > Delete log history > Delete logs.