The Transforms Python API provides classes and decorators for constructing a Pipeline.
| Name | Description |
|---|---|
configure([profile, allowed_run_duration, ...]) | A decorator that modifies the configuration of a Spark transform. |
incremental([require_incremental, ...]) | A decorator to convert inputs and outputs into their transforms.api.incremental counterparts. |
lightweight([_maybe_function, cpu_cores, ...]) | |
transform_df(output, **inputs) | Register the wrapped compute function as a DataFrame transform. |
transform_pandas(output, **inputs) | Register the wrapped compute function as a pandas transform. |
transform_polars(output, **inputs) | Register the wrapped compute function as a Polars transform. |
| Name | Description |
|---|---|
BooleanParam(default, *[, description]) | Specification for the ParameterSpec definition used as an input to a transform. |
Check(expectation, name[, on_error, description]) | Wraps up an expectation such that it can be registered with Data Health. |
ComputeBackend(*values) | Enum class for representing the different compute backends for use in configure(). |
ContainerTransform(transform, *[, ...]) | A callable object that describes a single step of a lightweight, single-node computation. |
ContainerTransformsConfiguration(transform, *) | A callable object that describes a single step of a lightweight, single-node computation. |
FileStatus(path, size, modified) | A collections.namedtuple capturing details about a FoundryFS file in Spark transforms. |
FileSystem(foundry_fs[, read_only]) | A filesystem object for reading and writing raw dataset files in Spark transforms. |
FloatParam(default, *[, description]) | Specification for the ParameterSpec definition used as an input to a transform. |
FoundryDataSidecarFile(param, path, ...) | A file object for reading and writing raw dataset files in lightweight, single-node transforms. |
FoundryDataSidecarFileSystem(param[, ...]) | A file system for reading and writing raw dataset files in lightweight, single-node transforms. |
FoundryInputParam(aliases[, branch, type, ...]) | A base class for transforms input parameters. |
FoundryOutputParam(aliases[, type, ...]) | A base class for transforms output parameters. |
IncrementalLightweightInput(alias, rid[, branch]) | The input object passed into incremental ContainerTransform objects at runtime. |
IncrementalLightweightOutput(alias, rid[, ...]) | The output object passed into user code at runtime for incremental ContainerTransform objects. |
IncrementalTableTransformInput(table_tinput, ...) | TableTransformInput with added functionality for incremental computation. |
IncrementalTransformContext(is_incremental, ...) | TransformContext with added functionality for incremental computation. |
IncrementalTransformInput(tinput[, ...]) | TransformInput with added functionality for incremental computation. |
IncrementalTransformOutput(toutput[, ...]) | TransformOutput with added functionality for incremental computation. |
Input([alias, branch, description, ...]) | Specification for a transform dataset input. |
InputSet([aliases, description]) | Specification for a list of transform inputs. |
IntegerParam(default, *[, description]) | Specification for a ParameterSpec definition used as an input to a transform. |
LightweightContext() | A context object that can optionally be injected into the compute function of a lightweight transform. |
LightweightInput(alias, rid[, branch]) | The input object passed into ContainerTransform objects at runtime. |
LightweightInputParam() | Base type for input parameters compatible with lightweight, single node transforms. |
LightweightOutput(alias, rid[, branch]) | The output object passed to user code at runtime. |
LightweightOutputParam() | Base type for output parameters compatible with lightweight, single node transforms. |
Markings(marking_ids, on_branches) | Specification for a marking that stops propagating from input. |
OrgMarkings(marking_ids, on_branches) | Specification for a marking that is no longer required on the output. |
Output([alias, sever_permissions, ...]) | Specification for a transform output. |
OutputSet([aliases, sever_permissions, ...]) | Specification for a list of transform outputs. |
Param([description]) | Base class for any parameter taken by the transform compute function. |
ParamContext(foundry_connector, input_specs, ...) | A context object injected in the instance method of a parameter. |
ParamValueInput(value) | A wrapper around the value of a parameter spec. |
Pipeline() | An object for grouping a collection of Transform objects. |
StringParam(default, *[, description, ...]) | Specification for the ParameterSpec definition used as an input to a transform. |
TableTransformInput(rid, branch, table_dfreader) | The input object passed into transform objects at runtime for virtual table inputs. |
Transform(compute_func[, inputs, outputs, ...]) | A callable object that describes a single step of a Spark computation. |
TransformContext(foundry_connector[, ...]) | A context object that can optionally be injected into the compute function of a transform. |
TransformInput(rid, branch, txrange, ...[, ...]) | The input object passed into Transform objects at runtime. |
TransformOutput(rid, branch, txrid, ...[, mode]) | The output object passed into Transform objects at runtime. |
transform(**kwargs) | Wrap a compute function as a Transform object. |
| Name | Description |
|---|---|
LightweightException | Base exception for lightweight compatibility checks. |
LightweightNotImplementedError(message) | Lightweight-specific NotImplementedError ↗ for unsupported features. |
LightweightTypeError(message) | Exception for type errors in lightweight compatibility checks. |
LightweightValueError(message) | Exception for value errors in lightweight compatibility checks. |