Skip to main content
Every dataset can be read two ways: a full scan (default) and, where supported, incremental feeds that return only what changed.

​
Full scan

With no incremental filter, a list endpoint returns every current object, paginated by cursor. Use this for the initial load or a periodic full refresh.

​
Incremental feeds

Assets, columns, column lineage, and tests expose two separate feeds so upserts and deletes stay clean and each can use its own index:
Timestamps are UTC ISO-8601.
  1. Store the wall-clock time T when you start a sync.
  2. Re-query from a small safety overlap behind the previous watermark (for example, synced_since=<previous T - overlap>). This covers clock skew and commits that were in flight when the previous sync ran.
  3. Page the synced_since feed to upsert changed objects and the deleted_since feed to remove tombstoned objects.
  4. After both feeds complete, persist T as the new watermark for the next run.
Drain each dataset once, then stay incremental. A full column-lineage scan is significantly more expensive per page than the other feeds — its edges are derived from each column’s stored dependencies, so a full scan reads every column in the environment — while synced_since and deleted_since reads are index-backed and stay fast as the environment grows. Persist next_cursor and the sync timestamp so a scheduled job resumes incrementally instead of re-running the initial snapshot on every run.
Feeds are at-least-once: an object may reappear across runs or in the safety overlap. Upserts and tombstones are keyed by id, so re-applying overlapping rows is idempotent.

​
Support matrix

A new test run does not bump Test.synced_at — current status lives on latest test executions. Disabled tests are omitted from a full scan unless disabled=true; the incremental feeds always include them so a sync observes a disable. Latest test executions have no incremental feed. Per-test history is filtered by start_time, not synced_since.
Column lineage is returned one record per edge, but edges do not have their own timestamps. The incremental filters use the downstream column’s timestamps. When a downstream column appears in the upserts feed, replace all of its edges rather than merging them. An edge removed while its downstream column still exists has no individual tombstone; a downstream column tombstone in deleted_since removes all of its edges.