Telemetry and History

RAPID keeps the latest channel state and can also record accepted value writes. TEMPEST keeps a separate latest-state telemetry snapshot for application clients; history stores samples for later queries. Neither is a substitute for the other.

The three channel paths

Every accepted channel commit has one source of truth and two downstream consumers:

RAPID live channel store
  ├─ task/calculation/direct-query state
  ├─ recording ingress → historical storage
  └─ TEMPEST telemetry snapshot → subscribed application clients

The live store is what RAPID tasks and explicit query operations use. Recording is a bounded historical stream. TEMPEST telemetry is a coalesced latest-state view: it is updated from committed RAPID publications and does not periodically query live fixed slots. This keeps a slow browser, telemetry serialization, or client fanout out of the fixed control-store locking path.

Choose a recording policy

Set data_frame to group subsequent samples. Changing it does not move old samples. A historical frame can remain after the live channel is removed.

Understand delivery

A local value commit runs its dependent calculations inline. Recording queues and telemetry publishers then process data asynchronously. This separation is deliberate: display, recording, Share, and ordinary observer delivery must not run arbitrary consumer work on a CAESAR callback or become a control acknowledgement. A slow UI does not prove the calculation was late; a current UI does not prove the newest sample is already on disk.

For a fixed multi-channel task commit, TEMPEST stages the accepted generation before its next telemetry tick. Existing application delivery remains one rapid/channels/{channel} topic per channel, so clients can receive those messages sequentially; each message is sourced from the same staged committed state, not a live RAPID poll.

Inspect the recording queue and persistence diagnostics when recent history is missing. Telemetry can coalesce intermediate updates for display, so a browser graph is not a lossless record of every engine write.

Query historical samples

Range bounds are Unix-epoch nanoseconds, including from and to in rapid/query-channel-range. Do not send seconds or JavaScript millisecond timestamps without conversion.

import time
end_ns = time.time_ns()
result = client.query_channel_range(
    series=["|tank_pressure|"],
    from_timestamp=end_ns - 60_000_000_000,
    to_timestamp=end_ns,
    data_frame="tank_test",
    bucket_count=500,
)

Use the Python guide for setup and result handling. For large work, the engine also exposes detached historical query and export operations; Using TEMPEST explains their lifecycle.

Preview versus raw data

Bucketing reduces returned points for a graph or analysis preview. Omit the bucket count when you need raw range-query samples. The Telemetry Exporter makes this distinction explicit: graph previews can be bucketed, while CSV export uses raw recorded data.

Before widening a query, confirm one channel, its data frame, and recorded time coverage in DataFrames. Empty history usually points to the recording policy, frame filter, or time range; it is different from a stale live value.

Recording queue capacity and loss

Recording and asynchronous channel notifications have separate queues and loss counters. Recording loss concerns historical samples; notification loss concerns delivery to asynchronous channel observers.

RAPID recording ingress has separate finite scalar and bulk queues. The defaults are 262,144 scalar rows, 8,388,608 bulk rows, 268,435,456 bulk payload bytes, and 1,024 bulk block descriptors. Capacity applies to pending ingress data, not all engine memory. Writer batches, RapidLog indexing, and historical indexes are separate consumers of memory.

When a queue fills, admission discards its oldest pending rows or whole blocks. An individually oversized block is rejected whole. Ordering is queue admission order, not sorting by sample timestamp; scalar and bulk queues remain separate. Live channel values and authority are unchanged by history loss. Stopping recording drains pending data without inserting capacity-consuming sentinel records.

Discarded bulk payloads go to a bounded retirement queue for recording workers to free. Its descriptor, row, and byte limits equal the configured bulk ingress limits; this can retain up to another bulk queue’s worth of payloads during overload. If retirement capacity is unavailable, the incoming block is rejected before constructing a recording payload, and existing queued blocks are preserved. The loss counter includes both outcomes. Bulk admission reserves capacity before copying samples outside the queue lock; concurrent construction counts against ingress capacity, and completed blocks enter the FIFO in publication order.

The existing persistence diagnostics expose recording_queue_rows, recording_queue_bytes, recording_queue_row_capacity, recording_queue_byte_capacity, and recording_dropped_rows_total. queue_depth remains the legacy descriptor count. The engine’s rapid_recording_queue_depth channel now counts rows, including rows in writer batches. Background reporting emits a recording-loss message and a correlated ARGUS error.

recording_retired_rows, recording_retired_bytes, recording_retired_row_capacity, and recording_retired_byte_capacity report the separate cleanup backlog. These discarded rows are already counted as lost; they are not pending history writes. Each recording worker can briefly own one additional block while freeing it outside queue locks.

RapidLog also bounds its pending index handoff to 1,024 blocks, 8,388,608 rows, and 268,435,456 retained payload bytes. These rows have already been written to disk, so a full index handoff waits on background recording workers instead of discarding index entries. The index_queue_rows, index_queue_bytes, index_queue_row_capacity, and index_queue_byte_capacity diagnostics describe that separate queue. If indexing remains stalled, recording ingress can fill and lose pending history under its policy above.

The active index batch can hold another queue-sized set of blocks. Writer batches, temporary splitting of oversized blocks on recording workers, and the historical in-memory index are separate; the index of retained history still grows. These queue limits are not a cap on total RAPID memory.

To change ingress limits, add a recording_queue object to the RAPID persistence configuration and restart the engine:

{
  "recording_queue": {
    "scalar_rows": 262144,
    "bulk_rows": 8388608,
    "bulk_bytes": 268435456,
    "bulk_blocks": 1024
  }
}

Explicit capacities must be positive integers. Queue descriptors are prepared before workers start, but queues still use mutexes. Bulk eviction defers payload destruction to recording workers; accepted bulk payload construction still allocates and copies samples on the caller, and construction failures can destroy temporary storage there. Complete history processing is not certified as bounded real-time work. Monitor losses and qualify your workload before relying on a chosen capacity.

Asynchronous channel notification capacity

The channel notification queue holds at most 8,192 notifications. Eligible prepared numeric fixed writes also have a shared 262,144-write capacity. One committed batch occupies one notification and one write entry per accepted write, including repeated writes to the same channel. Queue storage belongs to the asynchronous delivery worker rather than the task, so a stalled consumer does not exhaust the task’s reusable publication buffers. These are delivery-backlog limits, not a general claim that every queue operation is allocation-free or a substitute for a control-path resource-exhaustion policy.

When either limit fills, admission drops the oldest whole asynchronous notifications until the incoming batch fits. This preserves the order and contents of retained batches. Live values, command authority, and synchronous calculations are unaffected. Mixed payloads and batches larger than the fixed-write capacity use ordinary notification storage; this is not a bound on all notification memory or on execution time.

Existing persistence diagnostics expose publication_queue_depth, publication_queue_capacity, publication_queued_fixed_writes, publication_fixed_write_capacity, and publication_overwritten_total. The overwrite counter counts lost notifications, each of which may contain multiple writes. The engine reports publication backpressure through ARGUS. These counters are separate from recording_dropped_rows_total.

The live values preserved by notification overflow are local to this RAPID instance. Share also consumes these asynchronous notifications, so a local commit does not guarantee delivery of every update to a remote node. Remote command workflows need their own acknowledgement, freshness, and fault-response contract; notification loss is not only a display concern.

Project configuration and local state

Track workspace/<project>/rapid/channel_snapshot.json in Git. It stores channel configuration, metadata, and explicit startup values. Changing live values and timestamps are saved separately in engine/rapid/<project>/channel_state.json; RapidLog samples and CSV exports also stay under that engine-side RAPID directory. ARGUS uses its own engine/argus/<project>/ directory.

A fresh project clone restores configured startup values without needing another engine’s runtime state or recordings. See Tracking a Project in GitHub for the full layout and cloning workflow.