Push & Ingest

Push registers your sl.config.ts schema with the service. Ingest is the background process that reads your source, embeds records into vectors, and indexes them so queries can run.

The two operations are intentionally separate — you can push a schema change without immediately triggering a re-index.

Push Your Config

semilayer push

This registers (or updates) your lenses. New lenses start with status paused — no ingest runs yet.

# Push AND immediately start ingest for new/changed lenses:
semilayer push --resume-ingest

# Push AND do a full re-index of everything (clears existing data):
semilayer push --rebuild

--rebuild prompts for confirmation before clearing. Pass --skip-confirmation-dangerously to bypass in CI.

What Push Does

For each lens in your config compared to the current remote state:

• Created — new lens registered, status set to paused
• Updated — schema change applied; existing vectors remain until re-ingest
• Unchanged — no-op
• Deleted — lens removed along with its indexed data

Drift Detection

Before pushing, the CLI checks whether any lenses were modified in the Console since your last push. If so, you'll see a warning:

⚠  These lenses were modified in the Console since your last push:

    products    2h ago    console

Pushing will overwrite these changes.
Continue? (y/N)

Run semilayer export config to pull Console changes back into your local sl.config.ts before pushing, or pass --force to skip the check.

Start Ingest

semilayer resume              # resume all paused lenses
semilayer resume --lens products

Or at push time:

semilayer push --resume-ingest

Monitor Progress

semilayer status

  Lenses:
    products    indexing    18,092 vectors
    articles    ready       42,301 vectors
    users       paused      0 vectors

  Queue:
    Active: 1  Pending: 0  Failed: 0  Completed (24h): 7

You can also watch progress in the Console under Jobs.

Ingest Modes

Full ingest (default)

Reads all records from the source from scratch. Used on first push and when you pass --rebuild.

Incremental ingest

Reads only records changed since the last ingest cursor. Requires changeTrackingColumn on the lens (defaults to updated_at if present):

lenses: {
  products: {
    source: 'main-db',
    table: 'public.products',
    primaryKey: 'id',
    changeTrackingColumn: 'updated_at',  // optional, auto-detected
    // ...
  },
}

Trigger incremental manually:

semilayer sync
semilayer sync --lens products

Periodic sync

Keep the index fresh automatically with syncInterval:

lenses: {
  products: {
    // ...
    syncInterval: '15m',  // '1m' | '5m' | '15m' | '30m' | '1h' | '6h' | '24h'
  },
}

Webhook ingest

Push specific records in real time from your application using the ingest API and an ik_ key:

curl -X POST https://api.semilayer.com/v1/ingest/products \
  -H "Authorization: Bearer ik_live_..." \
  -H "Content-Type: application/json" \
  -d '{
    "mode": "records",
    "records": [
      { "id": "42", "data": { "name": "Updated name", "price": 99 } }
    ]
  }'

Modes: full (re-index all), incremental (since last cursor), records (specific IDs).

Pause Ingest

semilayer pause
semilayer pause --lens products

Ingest stops after finishing the current page. The cursor is saved so it can be resumed from where it left off.

Troubleshooting

Lens stuck in indexing — Check semilayer status for error counts, or open Jobs in the Console for the full error message.

error status — The last ingest job failed. The most common causes are:

• Connection refused (source not reachable from the worker)
• Auth failure (connection string changed or expired)
• Schema mismatch (a declared field doesn't exist in the source)

After fixing the underlying issue, run semilayer resume --lens <name> to retry.