Elements runtime reference / Datasource access

`appendDataSourceRows`

Submit JSON rows from an Element to an uplink/parquet-backed datasource; acknowledgement durability depends on Uplink configuration.

Terug naar builtin index Signatuur Payloadformaat Voorbeeld Scope-notities

Signatuur

appendDataSourceRows(dataSourceId, rows=None, merge=None, forcedFilterHash=None, meta=None, timeoutSeconds=None, idempotencyKey=None)

Doel

Submit JSON rows from an Element to an uplink/parquet-backed datasource; acknowledgement durability depends on Uplink configuration.

Aliases and imports

append_datasource_rows
dataSources.appendRows
dataSources.append_rows

Payloadformaat

Use these fields as the Python call arguments or keyword payload for this helper.

{
  "dataSourceId": "element_event_store",
  "rows": [
    {
      "ts": "2026-07-06T12:00:00Z",
      "status": "ok"
    }
  ],
  "meta": {
    "dataSourceName": "Element event store",
    "dataSourceType": "uplink",
    "dataFormat": "parquet-remote",
    "parquetPartitionColumn": "ts",
    "parquetPartitionGranularity": "year/month/day"
  },
  "idempotencyKey": "element_event_store:2026-07-06T12:00:00Z"
}

Field	Description
`dataSourceId`	Target writable datasource id. The runtime must be granted datasource write capability by dashview-be.
`rows`	One row dict or a list of row dicts.
`merge`	Optional bool for legacy non-Uplink types. type=uplink always buffers Parquet batches; scheduled or manual Parquet refresh controls compaction.
`forcedFilterHash`	For type=uplink omit or use default. Queries enforce user forced filters and column permissions on the combined snapshot.
`meta`	Optional auto-create metadata such as dataSourceName, dataFormat, parquetPartitionColumn, and parquetPartitionGranularity.
`timeoutSeconds`	Optional write timeout.
`idempotencyKey`	Optional caller-supplied key reserved for future audit/idempotency behavior. Do not assume retry deduplication.

Return payload

{
  "ok": true,
  "dataSourceId": "element_event_store",
  "inserted": 1,
  "parts": 0,
  "uris": [],
  "buffered": true,
  "acknowledgement": "memory",
  "durable": false,
  "revision": null,
  "createdDataSource": false
}

Voorbeeld

def run():
    result = dataSources.appendRows(
        "element_event_store",
        [{"ts": "2026-07-06T12:00:00Z", "status": "ok"}],
    )
    return {"inserted": result["inserted"]}

Scope-notities

This ingests datasource rows. appendRows() only emits live stream commands.
The example response represents memory acknowledgement: inserted counts accepted rows, parts is zero, and uris is empty before background publication. Durable mode returns the published Parquet object URI. Memory-accepted rows can be lost if the instance stops; retries can duplicate rows.
Runtime defaults are durable acknowledgement and a one-second batching window. uplink/deploy.sh defaults to memory acknowledgement and a flush after 60 seconds total or 5 seconds of inactivity for HTTP and gRPC unless overridden. Each accepted request resets only the inactivity timer within its tenant/datasource batch. Soft flush thresholds are 10 MiB or 10,000 requests per tenant/datasource batch.
Queries combine Parquet batches and historical parts, plus any legacy NDJSON buffers. Each datasource's refresh schedule or manual refresh controls Parquet compaction; hourly refresh is a configuration choice.
The helper is available only when dashview-be injects datasource write capability for the executing user/context.
Use queryDataSources to read published rows. Memory acknowledgement does not guarantee immediate read visibility; publication follows the background flush.

appendDataSourceRows