Transaction selectors

The following list describes the transaction selectors available for use when configuring retention policies in the Retention application.

Text in this format represents a parameter that could be defined on each policy.

Only in branch

Delete transactions that appear only in the given branch.

Takes 1 argument: branch

Example: "Only in branch main"

Setting "Only in branch main" will not delete the first two SNAPSHOT transactions as they are also the root transaction for the SNAPSHOT transaction on the feature branch.

Not in branch

Delete all transactions except ones in the given branch.

Takes 1 argument: branch

Example: "Not in branch main"

Transaction count

Retains only the transactions that are among the number of transaction to retain most recent data-containing transactions data on any branch. A transaction is defined to be data-containing if, and only if, the following statements are true:

• The transaction is committed.
• The transaction is not a DELETE transaction.

All aborted transactions are not data-containing and will be deleted. As this selector does not differentiate between SNAPSHOT, APPEND, or UPDATE transactions, we recommend using the viewCount selector in incremental pipelines.

Takes 1 argument: number of transactions to retain

Example: "transaction count 2"

This selector ensures that at least 2 SNAPSHOT transactions are available on each branch. The feature branch could have 3 transactions; the oldest transaction is also the 2nd transaction on branch main, so it is not deleted.

View count

The view count selector retains only transactions in the last number of views to retain dataset views. As a view is defined to comprise of only committed transactions, any aborted transactions are also deleted. For example, numViewsToRetain: 1 means that all transactions prior to the latest view (that is, all transactions prior to the latest SNAPSHOT transaction) and all aborted transactions are deleted.

Takes 1 argument: number of views to retain

Example: "view count 1"

Older than a certain duration

This selects transactions older than the given duration.

Takes 1 argument: duration

Has been projected (advanced)

For datasets with projections, this selector selects the transactions that have been propagated to all projections.

No files in active view (advanced)

This selector selects all transactions that are not in the latest view. It also selects transactions in the latest view whose files have all been superseded by files in newer transactions. Use it with Allow deletion from latest view to allow retention to delete superseded transactions that are still part of that view.

Ignore hidden files

Hidden files are files that do not contain dataset data. A file is considered hidden if its name begins with ., or begins with _ and does not contain an =, such as _driver.log. When Ignore hidden files is disabled, No files in active view considers these files when determining whether a transaction still has files in the active view.

Ignore hidden files is an option of the No files in active view selector. Enable it to consider only data files when evaluating the selector. A transaction can then be selected when none of its data files remain in the active view, even if its hidden files are still present.

Only present in views older than (advanced)

Selects transactions only present in views older than a given duration. A view age is defined as time between the close time of the latest transaction in the view and now. If a view has an open transaction, none of the transactions in that view will be deleted.

Takes 1 argument: duration

Is derived (advanced)

Selects transactions which are derived. These are transactions generated from running a build. Transactions created from manually uploading data or through Data Connection are not considered derived.