Assign timestamps and watermarks

Supported in: Streaming

Assigns timestamps and watermarks to the input, filtering out records where the timestamp is null.

Transform categories: Other

Declared arguments

• Dataset: Dataset to assign timestamps and watermarks.
  Table
• Timestamp expression: Expression evaluating to timestamp to assign.
  Expression<Timestamp>
• optional Emit watermark on every record: If true, the watermark will be propagated for every record in the stream. This is generally inefficient and will add performance overhead but gives deterministic behaviour if the stream is replayed. It is recommended to set this value to false in most cases.
  Literal<Boolean>
• optional Idleness timeout unit: Unit for the duration of time to consider a subtask idle.
  Enum<Days, Hours, Milliseconds, Minutes, Seconds, Weeks>
• optional Idleness timeout value: Value for the duration of time after which a subtask not receiving records will be considered idle, at which point it will not prevent the global watermark from progressing. This can help address unbounded state growth for time-based transforms or hanging (non-emitting) windows, but late records from slow subtasks may be dropped unexpectedly for downstream operators with allowed lateness. Please check Foundry Streaming docs to understand event-time handling.
  Literal<Long>

Examples

Example 1: Base case

Argument values:

• Dataset: ri.foundry.main.dataset.a
• Timestamp expression: timestamp
• Emit watermark on every record: null
• Idleness timeout unit: null
• Idleness timeout value: null

Input:

Output:

---

Example 2: Null case

Argument values:

• Dataset: ri.foundry.main.dataset.a
• Timestamp expression: timestamp
• Emit watermark on every record: null
• Idleness timeout unit: null
• Idleness timeout value: null

Input:

Output:

---