BigQuery compute pushdown

To use compute pushdown with BigQuery, create a lightweight Python repository and install the most recent version of the transforms-tables library. An Ibis ↗ connection to BigQuery will be established based on the connection details of the BigQuery tables configured as inputs and/or outputs. The data can be transformed using the Ibis DataFrame API. For full guidance on the Ibis API, consult the Ibis documentation ↗.

An example of a BigQuery-backed Ibis transform is shown below:

Copied!
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
from transforms.api import lightweight, transform
from transforms.tables import (
    TableLightweightInput,
    TableLightweightOutput,
    TableInput,
    TableOutput,
    BigQueryTable
)


@lightweight
@transform(
    source_table=TableInput("ri.tables.main.table.1234"),
    output_table=TableOutput(
        "ri.tables.main.table.5678",
        "ri.magritte..source.1234",
        BigQueryTable("PROJECT", "DATASET", "TABLE"),
    ),
)
def compute(source_table: TableLightweightInput, output_table: TableLightweightOutput):
    table = source_table.ibis().table()
    # data transformation with Ibis
    output_table.ibis().write(table)