Full scan
With no incremental filter, a list endpoint returns every current object, paginated by cursor. Use this for the initial load or a periodic full refresh.Incremental feeds
Assets, columns, and column lineage expose two separate feeds so upserts and deletes stay clean and each can use its own index:Recommended loop
- Store the wall-clock time
Twhen you start a sync. - Re-query from a small safety overlap behind the previous watermark (for
example,
synced_since=<previous T - overlap>). This covers clock skew and commits that were in flight when the previous sync ran. - Page the
synced_sincefeed to upsert changed objects and thedeleted_sincefeed to remove tombstoned objects. - After both feeds complete, persist
Tas the new watermark for the next run.
Drain each dataset once, then stay incremental. A full
column-lineage scan
is significantly more expensive per page than the other feeds — its edges are
derived from each column’s stored dependencies, so a full scan reads every
column in the environment — while synced_since and deleted_since reads are
index-backed and stay fast as the environment grows. Persist next_cursor and
the sync timestamp so a scheduled job resumes incrementally instead of
re-running the initial snapshot on every run.Feeds are at-least-once: an object may reappear across runs or in the
safety overlap. Upserts and tombstones are keyed by
id, so re-applying
overlapping rows is idempotent.Support matrix
Column lineage is returned one record per edge, but edges do not have their
own timestamps. The incremental filters use the downstream column’s
timestamps. When a downstream column appears in the upserts feed, replace all
of its edges rather than merging them. An edge removed while its downstream
column still exists has no individual tombstone; a downstream column tombstone
in
deleted_since removes all of its edges.