Rolling back a pipeline is currently in the beta phase of development. Functionality may change during active development.
When building your pipeline, you may need to roll back a dataset and all of its downstream dependents to an earlier version. There can be many reasons for this, including the following:
The pipeline rollback feature allows you to revert back to a transaction of an upstream dataset. When performing a rollback, the data provenance of the upstream dataset transaction is used to identify its downstream datasets and their corresponding transactions to create a final pipeline rollback state. Typically, this process would require several steps to properly roll back each affected dataset. With pipeline rollback, this is reduced to a few simple steps discussed below, along with the ability to preview the final pipeline state before confirming and proceeding with the rollback. Pipeline rollback also ensures that the incrementality of your pipeline is preserved.
As you set up your rollback, you can choose to exclude any downstream datasets; these datasets will remain unchanged as the pipeline is rolled back to the selected transaction.
If a dataset backs an object type stored using Object Storage V2, manual intervention is required to ensure that the object type is reindexed with a successful run of the replacement pipeline to reflect the state after the rollback.
After choosing the transaction, downstream datasets will automatically be found and the states they will revert to if the rollback is actioned will be displayed.
Resource types that are unable to be rolled back, including streaming datasets, media sets, and restricted views, will be displayed under the unsupported resources section. Transactional datasets on which you do not have Edit
access will also be included in this list.
Depending on how your pipeline changed between the current state and the rollback transaction, you may receive one of the warnings shown below on datasets selected for downstream rollback:
The dataset logic was changed after the selected transaction time. Rolling back this dataset will not revert logic changes.
If you receive this warning, you may need to inspect how the logic changed over time and determine if you can apply the logic again after the rollback and before any new dataset builds.This dataset did not exist at the time of the transaction selected for rollback. Its incremental state will be reset, though transactions and data will be unchanged after rollback. It will run as a snapshot in the next build.
If you receive this warning, note that the rollback will reset the dataset's job history as if it was never built. The next build on the dataset will not be run incrementally (if the dataset is an incremental dataset) since it will be treated as a completely new dataset being built for the first time.When rolling back a pipeline, keep the following considerations in mind:
DELETE
retention transaction.Edit
access on an upstream dataset that is then used to build a downstream dataset to which you only have View
access. In this case, the downstream dataset will be considered an "unsupported resource".