Configure logging
Applications across Foundry emit logs that provide visibility into their operation, including transform runs, build failures, and other system events.
Foundry logs contain information about:
- Application execution events
- Transform and pipeline runs
- Build and deployment activities
- System errors and warnings
Additionally, any user-added logs written in Code Repositories will also be emitted.
These logs can be exported to a Foundry streaming dataset for real-time monitoring and analysis. This feature is only available to Organization Administrators, who can configure a destination where logs from Foundry applications will flow into a Foundry streaming dataset.
Log export to Foundry streaming dataset
Foundry logs can be exported, per-organization, directly into a Foundry streaming Dataset. As part of the configuration setup, an Organization Administrator chooses where within the Foundry file system this dataset will be generated.
Once log data is flowing into the streaming dataset, you can process and analyze the data in real-time using Foundry's extensive suite of data transformation and visualization tools or export the logs to external monitoring systems.
Export permissions
To export Foundry logs, a user must be an Organization Administrator. This role provides the necessary permissions to configure log exporting destinations for the organization. See Organization Permissions for more details about organization-level permissions.
Additionally, sensitive details in logs are obfuscated and/or removed to protect user privacy and security.
Export setup
To set up log exporting in Foundry, follow these steps:
- Navigate to the Control Panel application in Foundry.
- In the left-hand sidebar, select the relevant organization via the dropdown menu.
- In the left-hand sidebar, select Log exporting under Organization.
- Select Create log storage configuration to begin the process of configuring a log storage dataset. Select a location in Foundry for this streaming dataset. The location cannot be a user-created folder.
Note that when first configuring log exporting, it may take some time for logs to begin flowing into the streaming dataset.
Disable log exporting
To disable log exporting, delete any log storage configurations listed by selecting the trash icon on the right side of the configuration.
Analyze log data
Log storage datasets can contain high volumes of streaming data, so we recommend filtering the dataset appropriately before performing analysis. Consider time-based filtering to focus on relevant log entries for your monitoring and troubleshooting needs.
Foundry provides many powerful tools for performing log analysis, such as Transforms and Pipeline Builder.
Log schema
Foundry application logs are structured logs that follow a consistent schema, making them suitable for programmatic analysis and monitoring.
Specifically, Foundry logs contain information about various application events, including transform executions, build processes, system errors, and other operational activities. The log structure varies depending on the type of application or service that generates the log entry.
Application-specific information is captured within structured fields that provide context about the operation being performed, including execution details, error information, and performance metrics.
Log attribution
Foundry application logs are generated by various services and applications within the platform. When logs are streamed to the configured dataset, they are filtered to include only logs relevant to the organization that configured the log storage configuration.
Log exports configured for a given organization will receive logs from applications and services used by that organization. This includes transform executions, builds, and other activities performed within the organization's context.
The RID of the resource that generated the log is included in the log entry and can be used to trace the log back to the resource.