Object Storage v2 (OSv2) enforces data restrictions to ensure the quality of data going into the ontology, provide more deterministic behavior, and increase legibility across the platform. These restrictions are validated during indexing. For object types backed by batch datasources, violations will cause indexing jobs to fail. For object types backed by streaming datasources, records that violate these restrictions are dropped.
OSv2 requires unique object primary keys, but how it evaluates uniqueness depends on the datasource type. For dataset datasources, primary keys must be unique within each transaction; if a single transaction contains duplicate primary keys, indexing will fail and throw an error. When indexing incremental datasets, if a primary key appears in multiple transactions, OSv2 uses the row from the most recent transaction.
For Iceberg table datasources, primary keys must be unique across the current table contents, including rows added in separate writes. Duplicate keys are unsupported and can cause incorrect indexed data or failures on later full reindexes, even if incremental indexing initially succeeds. See Iceberg table datasources.
OSv2 prevents certain data types from being used as primary keys in order to encourage Ontology modeling best practices. The following types cannot be used as primary keys:
Double to Integer), all existing values for that property must be strictly compatible with the target type. If any data entries include values incompatible with the new type (such as fractional numbers when changing to Integer, or currency symbols), the migration will fail with an error such as A property could not be cast to the new type. Schema migrations will not proceed if incompatible values exist, and the migration process cannot automatically clean or coerce these values.NaN or ±infinity as property values.Lat, Long should be a comma-separated string with no parentheses, for example -29.123, 150.982.Not conditions in granular permissioning policies of restricted view datasources where the negated field is a collection and has a non-empty constraint. This can be configured in Ontology Manager by marking the relevant property as required.OSv2 enforces size limits on individual properties to ensure reliable indexing performance and stability. These limits address serialization constraints and memory pressure that can occur when processing large property values.
| Property type | Maximum size |
|---|---|
| String properties | 12 MB |
| Array properties | 100,000 elements |
Properties exceeding these limits will cause indexing jobs to fail. These limits are initially enforced for object types backed by batch datasources; similar limits will be applied to object types with streaming datasources in the future.