Lightweight transforms API evolution

The transforms API for lightweight compute has evolved over time to support more streamlined syntax options.

Reference this page for details on the evolution of the lightweight API.

To start writing lightweight transforms, see Getting started or the Polars and pandas examples in Python transforms. These pages explain how to use lightweight transforms as the default compute option.

Legacy syntax: @lightweight decorator

The original syntax for lightweight compute is the @lightweight decorator. This syntax option remains fully supported.

Copied!
1
2
3
4
5
6
7
8
9
10
11
from transforms.api import transform, lightweight, Input, Output


@lightweight
@transform(
    output=Output("/path/data/output"),
    input=Input("/path/data/input"),
)
def clean(output, input):
    df = input.pandas()
    output.write_table(df)

Updated syntax for Lightweight

The new recommended syntax for accessing lightweight transforms is @transform.using. This API removes the need for additional lightweight imports and streamlines the creation of lightweight transforms as the default.

The new API is available from transforms version 3.68.0 and higher. To make this available in Code Repositories, upgrade your repository with the repository upgrade guide. Ensure that the transformsLangPythonPluginVersion is equal to 1.978.0 or higher.

To learn more about transforms versions, see the transforms versions overview.

Copied!
1
2
3
4
5
6
7
8
9
10
from transforms.api import transform, Input, Output


@transform.using(
    output=Output("/path/data/output"),
    input=Input("/path/data/input"),
)
def clean(output, input):
    df = input.pandas()
    output.write_table(df)

Updated syntax for Spark

Alongside the new, default lightweight API, there is a new transforms API for Spark. The recommended syntax is @transform.spark.using.

The new API is available from transforms version 3.95.0 and higher. To make this available in Code Repositories, upgrade your repository with the repository upgrade guide. Ensure that the transformsLangPythonPluginVersion is equal to 1.1003.0 or higher.

Copied!
1
2
3
4
5
6
7
8
9
10
from transforms.api import transform, Input, Output


@transform.spark.using(
    output=Output("/path/data/output"),
    input=Input("/path/data/input"),
)
def clean(output, input):
    df = input.dataframe()
    output.write_dataframe(df)

Updated syntax for Databricks Connect

To define transforms that push down compute to Databricks via Databricks Connect, use the combined decorator @transform.databricks.using. This standardized syntax replaces patterns that previously mixed @databricks and @transform decorators.

You can chain configuration methods to specify compute and dependencies:

• with_compute(cluster_id="<cluster-id>") to connect to a specific Databricks compute cluster
• with_dependencies(dependencies=["<dependency>"]) to install Python packages required by UDFs on Databricks compute

By default, Databricks Connect establishes a session using serverless compute.

Copied!
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
from pyspark.sql.functions import col
from transforms.api import transform
from transforms.tables import DatabricksInput, DatabricksOutput, TableInput, TableOutput

@transform.databricks.using(
    source_table=TableInput('/Project/folder/input'),
    output_table=TableOutput(
        '/Project/folder/output',           # Register virtual table in Foundry
        'ri.magritte..source.1234',         # Databricks source connection RID
        'CATALOG.SCHEMA.TABLE',             # Unity Catalog table target in Databricks
    ),
).with_compute(cluster_id="<cluster-id>")
def compute(source_table: DatabricksInput, output_table: DatabricksOutput):
    df = source_table.dataframe()
    df = df.filter(col('id') > 1)
    output_table.write_dataframe(df)

For UDFs that require additional Python dependencies, configure them with with_dependencies:

Copied!
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
from pyspark.sql.functions import udf
from pyspark.sql.types import StringType
from transforms.api import transform
from transforms.tables import DatabricksOutput, TableOutput

@udf(returnType=StringType())
def get_joke():
    from pyjokes import get_joke
    return get_joke()

@transform.databricks.using(
    output_table=TableOutput(
        '/Project/folder/output',
        'ri.magritte..source.1234',
        'CATALOG.SCHEMA.TABLE',
    ),
).with_dependencies(dependencies=["pyjokes<1"])
def compute(output_table: DatabricksOutput):
    df = output_table.spark_session.range(1, 10)
    df = df.withColumn('jokes', get_joke())
    output_table.write_dataframe(df)

Troubleshoot version errors with @transform.using

If you receive an error message similar to the following, you may be on an older version of the transforms library that does not support the updated syntax.

• Usage of transform.using() on outdated repository
• A function object does not have an attribute using

To resolve this issue, try the following options:

• Upgrade your repository to the latest version of the transforms library and repository template following the repository upgrade guide.
• Switch to legacy syntax.