• Documentation
    • Search documentation
    karat

    +

    K

    API Reference ↗
    Data connectivity & integrationPythonBasic transformsMetrics

    Metrics

    Use the Metrics telemetry tab to debug and monitor your builds.

    The metrics telemetry tab is available for transforms that use a "lightweight" compute engine. The metrics Telemetry tab is not available for transforms built with Spark compute.

    Access metrics

    Follow the steps below to view metric details. Note that metrics are not available for transforms that use Spark compute.

    1. View the build report:
      • From a Dataset Preview or from Data Lineage, select the History tab, select an individual build from the list, then select View build report.
      • Select a build from the list of all builds.
    2. Select a job: A build consists of one or more jobs, which are visible in the list below the Gantt chart. Choose a job from the list, select the Telemetry option, and select the Metrics tab to see metrics emitted by the build container plotted on time-series charts.

    The "Metrics" tab.

    Memory utilization

    The Memory utilization graph shows your build's memory usage. The red dashed line signifies the memory limit requested from the build. Any build that exceeds this will cause the container to throw an out-of-memory (OOM) error.

    The following example shows a build that has thrown an OOM error, which means that the build is memory-bound. Memory usage rises, and eventually leads to an OOM. If you encounter an OOM error, increase the container's memory limits by setting container memory to 32 GB or another appropriate value.

    Build metrics showing out-of-memory error.

    The example below shows a build that underutilizes the requested memory. This build does not come close to reaching memory limits, so setting the memory request to 32 GB would reduce resource consumption for this particular build.

    Underutilized memory build metrics.

    CPU utilization

    The CPU utilization graph shows your build's CPU usage. The orange dashed line shows your build's requested CPUs. Two cores are requested by default. The build scheduler will wait until it can allocate your job on a host that has sufficient cores available to fill your request. Any extra unused cores on the host will automatically be provisioned to help run the build.

    CPU build metrics.

    In the example above, two CPUs were requested, but 22 were used to help speed up the build. Be aware that the host is not guaranteed to always have additional CPUs to allocate during the build.

    RequestResult
    1 CPU coreBuilds will start up faster as the scheduler will be able to find free hosts easier. Build times may be inconsistent as available CPU resources may vary.
    8 CPU coresBuilds will take longer on average to start as the scheduler may struggle to find hosts with free cores. Build times may be more consistent as 8-core availability will be guaranteed during the build.