Change feeds & jobs

Change feeds and jobs are the engine behind standing questions: a change feed detects change with zero stored data, a job acts on it on a schedule or a gate — so an agent's question keeps answering itself as new data lands.

The Snowflake-dialect keywords are accepted as aliases, so CREATE STREAM

still creates a change feed and CREATE TASK still creates a job. The

Obelisk names below are the primary spelling.

Change feeds — CDC as version diffs

A change feed is a tiny metadata object: an offset into the source table's version timeline. No change data is ever stored — querying the feed computes the delta between the offset version and the current version from the immutable file sets:

• - rows in segments added since the offset → candidate inserts
• - rows in segments removed since the offset → candidate deletes
• - identical rows on both sides cancel out (multiset EXCEPT ALL) — they were

merely rewritten by copy-on-write DML, not changed

CREATE CHANGE FEED f ON TABLE t;        -- offset = t's current version
SELECT * FROM f;                        -- source columns + metadata columns
SELECT SYSTEM$CHANGE_FEED_HAS_DATA('F');     -- cheap: compares two version numbers

Metadata columns: METADATA$ACTION (INSERT/DELETE), METADATA$ISUPDATE (always FALSE here — see divergences), METADATA$ROW_ID (content hash).

Consumption: the offset advances only when a DML statement reading the feed commits — INSERT INTO tgt SELECT … FROM f or CREATE TABLE x AS SELECT … FROM f. Reading alone never advances it, so a consumer that fails leaves the feed intact.

Append-only feeds (APPEND_ONLY = TRUE) see every inserted row — even rows deleted afterwards — and ignore deletes entirely. For insert-only tables (the usual EL landing pattern) they are exact and cheapest.

One divergence worth knowing: without per-row lineage, an UPDATE appears in a change feed as a DELETE+INSERT pair rather than a paired update row.

Jobs — scheduled statements and DAGs

CREATE JOB load_raw
  SCHEDULE = '5 MINUTES'
  WHEN SYSTEM$CHANGE_FEED_HAS_DATA('F')
  AS INSERT INTO staged SELECT * FROM f;

CREATE JOB refresh_marts AFTER load_raw
  AS EXECUTE PIPE nightly_sync;          -- jobs can run pipes too

ALTER JOB load_raw RESUME;               -- jobs are created SUSPENDED
EXECUTE JOB load_raw;                    -- manual run of the whole graph

• - SCHEDULE — '<n> SECONDS|MINUTES|HOURS' intervals (root jobs only).
• - AFTER parent — the child runs when the parent succeeds; graphs cascade.
• - WHEN — a SYSTEM$CHANGE_FEED_HAS_DATA gate evaluated as pure metadata; a

closed gate records a SKIPPED run and never spins up an engine.

• - Runs are recorded in task_runs (visible in the console's Jobs pane).

The scheduler

The scheduler runs inside obelisk ui (a tick every 5 seconds). Without the UI running, EXECUTE JOB and session.tasks.tick() still work — cron can call obelisk ingest run / a Python one-liner if you want headless scheduling.

Standing questions

Chain a pipe, a change feed, and a WHEN-gated job and a question answers itself: new rows land, the feed reports the delta as metadata, the gate opens, the job refreshes the downstream transform. The gate stays metadata-cheap when nothing changed, so an always-on standing question costs almost nothing at rest. Persist the answers it produces under Insights & memory, and keep the definitions it computes consistent through the Metrics & semantic layer.