> ## Documentation Index
> Fetch the complete documentation index at: https://docs.elementary-data.com/llms.txt
> Use this file to discover all available pages before exploring further.

# Incremental sync

> Full scans plus incremental upsert and delete feeds.

Every dataset can be read two ways: a **full scan** (default) and, where
supported, **incremental feeds** that return only what changed.

## Full scan

With no incremental filter, a list endpoint returns every current object,
paginated by cursor. Use this for the initial load or a periodic full refresh.

```
GET /{env_id}/assets?limit=500
```

## Incremental feeds

Assets, columns, column lineage, and tests expose two separate feeds so upserts
and deletes stay clean and each can use its own index:

| Filter | Returns | Use for |
| - | - | - |
| `synced_since` | Objects created or updated at/after the timestamp | Upserts |
| `deleted_since` | Soft-delete tombstones at/after the timestamp | Deletions |

```bash theme={null}
# New/changed assets since your last sync
curl -H "Authorization: Bearer $ELEMENTARY_TOKEN" \
  "$BASE/$ENV_ID/assets?synced_since=2026-08-01T00:00:00Z&limit=500"

# Assets deleted since your last sync
curl -H "Authorization: Bearer $ELEMENTARY_TOKEN" \
  "$BASE/$ENV_ID/assets?deleted_since=2026-08-01T00:00:00Z&limit=500"
```

Timestamps are UTC ISO-8601.

<Note>
  **What `synced_at` means.** `synced_at` marks when an object was **created or
  last changed** in Elementary — not when it was last *seen*. Unchanged objects
  keep their existing `synced_at`, and some objects that predate this field may
  show a `1970-01-01T00:00:00` placeholder. This is expected and safe for
  incremental loading: every object is returned by the initial full scan
  (regardless of `synced_at`), and any later change gives it a fresh `synced_at`,
  so it reappears in the `synced_since` feed. Treat `synced_at` as a change
  marker for driving upserts, not as an absolute "last refreshed" timestamp.
</Note>

## Recommended loop

1. Store the wall-clock time `T` when you start a sync.
2. Re-query from a small safety overlap behind the previous watermark (for
   example, `synced_since=<previous T - overlap>`). This covers clock skew and
   commits that were in flight when the previous sync ran.
3. Page the `synced_since` feed to upsert changed objects and the
   `deleted_since` feed to remove tombstoned objects.
4. After both feeds complete, persist `T` as the new watermark for the next run.

<Note>
  Drain each dataset once, then stay incremental. A full `column-lineage` scan
  is significantly more expensive per page than the other feeds — its edges are
  derived from each column's stored dependencies, so a full scan reads every
  column in the environment — while `synced_since` and `deleted_since` reads are
  index-backed and stay fast as the environment grows. Persist `next_cursor` and
  the sync timestamp so a scheduled job resumes incrementally instead of
  re-running the initial snapshot on every run.
</Note>

<Note>
  Feeds are **at-least-once**: an object may reappear across runs or in the
  safety overlap. Upserts and tombstones are keyed by `id`, so re-applying
  overlapping rows is idempotent.
</Note>

## Support matrix

| Dataset | Full scan | Incremental |
| - | - | - |
| Assets | ✅ | ✅ (`synced_since`, `deleted_since`) |
| Columns | ✅ | ✅ (`synced_since`, `deleted_since`) |
| Asset lineage | ✅ | Full replace |
| Column lineage | ✅ | ✅ (`synced_since`, `deleted_since`) |
| Tests | ✅ | ✅ (`synced_since`, `deleted_since`) |
| Latest test executions | ✅ | Full snapshot |
| Test execution history | ✅ | Time window (default last 30 days) |
| Asset health scores | ✅ | Full snapshot |

<Note>
  A new test run does not bump `Test.synced_at` — current status lives on
  latest test executions. Disabled tests are omitted from a full scan unless
  `disabled=true`; the incremental feeds always include them so a sync observes
  a disable. Latest test executions have no incremental feed. Per-test history
  is filtered by `start_time`, not `synced_since`.
</Note>

<Note>
  Column lineage is returned one record per edge, but edges do not have their
  own timestamps. The incremental filters use the downstream column's
  timestamps. When a downstream column appears in the upserts feed, replace all
  of its edges rather than merging them. An edge removed while its downstream
  column still exists has no individual tombstone; a downstream column tombstone
  in `deleted_since` removes all of its edges.
</Note>


This documentation is built and hosted on [Mintlify](https://mintlify.com), a developer documentation platform.