salvus.flow.ng.telemetry
salvus.flow.ng.telemetry salvus flow ng telemetry Run telemetry for in-process runs: a TelemetrySnapshot captured at
solve_forward_through_sessions’s progress_callback chunk boundaries, a
TelemetryCollector holding the latest snapshot plus a bounded in-memory
history, and sinks that publish it. FileSink writes
<output_folder>/progress.json — a strict superset of command-line solver
Salvus’ own format (status/last_written/tasks keep their exact meaning),
so salvus.flow.executors.salvus_job.SalvusJob. update_status() parses a
solver run’s progress through the exact same code path a remote command-line
solver job’s status already goes through.
T2 increment of the design in notes/ng-run-telemetry-design.md
(control-project repo): adds the live, fileless ZmqSink — a local
REP-socket server thread a separate Python process can query
(snapshot/history/cancel/ping) via the one handshake file it
writes, <output_folder>/telemetry_endpoint.json. NGInProcessJob. telemetry()/telemetry_stream()/cancel() (in in_process_job.py)
use it too, though a same-process caller reads the TelemetryCollector
it already holds directly rather than round-tripping through the
socket.
Threading model: this module is written to be correct under BOTH the
GIL (Python 3.11) and free-threaded Python (3.14t, no GIL) — nothing
here relies on the GIL for thread-safety. TelemetryCollector guards
its latest-snapshot slot and its bounded history with one explicit
threading.Lock; every reader (latest, history) copies the
reference/tuple out while holding that lock rather than assuming a
plain attribute read or deque append is atomic. The cancel flag is a
threading.Event, not a bare bool. This is now exercised by more than
one thread for real: ZmqSink runs its request loop on its own daemon
thread, reading collector.latest/collector.history and setting
collector.cancel_requested concurrently with the solve loop’s own
thread writing them through the same lock-guarded paths. On the native
side, the session bindings release the GIL for the duration of
advance/run_segment (see session_binding.hpp) so this thread can
actually run while a chunk is in flight; the session’s own
steps_completed counter is NOT read from here or anywhere else
mid-advance (it is a plain, non-atomic C++ counter) — this module’s
snapshots are built entirely from the Python-side progress_callback
arguments and psutil, never from native session state read
cross-thread.
Functions
current_process_resources()
current_process_resources()def current_process_resources(peak_rss_bytes: int) -> TelemetryResources: ...Read the current process’ resource usage via psutil.
peak_rss_bytesint — The running peak RSS the caller has tracked so far — folded into the returned peak with the freshly-read current RSS, since this function has no memory of its own across calls.
live_timer_snapshot()
live_timer_snapshot()def live_timer_snapshot(timer: typing.Any) -> dict[str, typing.Any] | None: ...Serialize a Timer tree that may still be running, tolerating
unstopped nodes — unlike in_process_job._serialize_timer, which
requires the whole tree to already be stopped (Timer.duration
raises ValueError on a running timer with no accumulated
durations, per its own docstring).
A still-running node’s duration is estimated as elapsed wall time
so far (time.perf_counter_ns() - t._start_time_ns); a stopped
node uses its own real duration unchanged. Never raises into the
solve loop: if reading a node’s timing fails for any reason, that
node’s duration/percentage are omitted rather than propagating
the failure (a telemetry snapshot is best-effort, and must never be
able to take the actual solve down with it).
timertyping.Any — The rootTimer, running or stopped.
_serialize_timer produces, or None if even the root’s own timing could not be read.make_progress_callback()
make_progress_callback()def make_progress_callback(
collector: TelemetryCollector,
sink: TelemetrySink | None,
timer: typing.Any,
estimator: RunningDurationEstimator,
job_id: str,
backend: str,
dtype: str,
solution_order: int | None,
num_ranks: int,
num_threads: int,
start_time_wall: float,
start_time_str: str,
dt: float,
) -> typing.Callable[[int, int], None]: ...Build a solve_forward_through_sessions-shaped progress_callback
that builds a fresh TelemetrySnapshot on every chunk boundary and
feeds it to collector.update() and, if given, sink.publish().
Every snapshot this callback builds reports status="running" —
the mid-loop callback has no way to know whether whatever comes
after the time-stepping loop returns (output copying, etc.) will
itself succeed, so "finished"/"failed" are always the caller’s
own explicit final collector.update()/sink.close() call, made
after (or instead of) this callback’s own last firing.
collectorTelemetryCollector — Updated with a new snapshot on every call.sinkTelemetrySink | None — Published to on every call, if given.timertyping.Any — The run’s rootTimer, still running — serialized vialive_timer_snapshot.Noneif no timer exists yet.estimatorRunningDurationEstimator — Supplieseta_seconds/steps_per_second;Nonevalues are reported for both until it has timed at least one post-warm-up step.job_idstr —TelemetryRun.job_id.backendstr —TelemetryRun.backend.dtypestr —TelemetryRun.dtype.solution_orderint | None —TelemetryRun.solution_order.num_ranksint —TelemetryRun.num_ranks.num_threadsint —TelemetryRun.num_threads.start_time_wallfloat —time.time()at run start, forTelemetryProgress.wall_elapsed.start_time_strstr —TelemetryRun.start_time.dtfloat — The time axis’ own step size, forTelemetryProgress. sim_time = step * dt.
Classes
CompositeSink
CompositeSinkclass CompositeSink(builtins.object):
def __init__(self, sinks: typing.Sequence[TelemetrySink]) -> None: ...Fan publish/close out to several sinks (e.g. a FileSink and a
ZmqSink together), each independently — one sink’s own exception
(already supposed to be impossible per TelemetrySink’s contract,
but never trusted blindly here) can never stop the others from
being called.
sinkstyping.Sequence[TelemetrySink] — Published to, in order, on everypublish/closecall.
close()
close()def close(self, final_snapshot: TelemetrySnapshot) -> None: ...Close every sink in sinks with final_snapshot.
final_snapshotTelemetrySnapshot — The run’s final (terminal-status) snapshot.
publish()
publish()def publish(self, snapshot: TelemetrySnapshot) -> None: ...Publish snapshot to every sink in sinks.
snapshotTelemetrySnapshot — The snapshot to publish.
FileSink
FileSinkclass FileSink(builtins.object):
def __init__(
self,
output_folder: pathlib.Path,
interval: float = 2.0,
history_path: pathlib.Path | None = None,
) -> None: ...Writes each published snapshot to <output_folder>/progress.json,
at most every interval seconds — except a run’s very first
publish and any publish whose snapshot.run.status is terminal
(finished/failed/cancelled), both of which always write
regardless of interval.
The written JSON is a strict SUPERSET of SalvusCompute’s own
progress.json shape: status (one of TelemetryRun.status’s values),
last_written (formatted exactly like command-line solver’s own,
_LAST_WRITTEN_FORMAT — SalvusJob.update_status() parses this with a
fixed arrow format list and does not accept an otherwise-valid ISO-8601
string in the wrong shape), and tasks (one task_type: "progress" task
tracking the time-stepping loop, empty while n_steps is not yet known
during setup) — so SalvusJob.update_status() and the tqdm-based
progress-callback renderer (salvus.flow.utils.progress_callback) both
work on an The solver run’s progress.json unchanged. Everything
solver-specific lives under one additional top-level telemetry key
(TelemetrySnapshot.to_dict()), so it can never collide with or be
mistaken for a command-line solver field.
This is a purely local-filesystem sink: NGInProcessJob (the only consumer
in T1) reads its own collector directly, in-process, so the write cadence
here has no correctness requirement of its own yet. A future remote the
solver site (T3) that relies on SalvusJob. update_status()’s own
staleness check will need a cadence at least as tight as
2 * site.update_interval_in_seconds (0.5s for a local site, 2.0s for
ssh, 60s for slurm, per BaseExecutor.update_interval_in_seconds) or
that check discards the snapshot as stale — not a concern this increment’s
own consumer hits, since it never goes through that remote-IO path.
output_folderpathlib.Path — Directoryprogress.jsonis written into. Created if missing.intervalfloat — Minimum seconds between two non-terminal publishes.history_pathpathlib.Path | None — If given, one JSON line per publish is appended here (history.jsonl-shaped) for post-mortem plots. Off (None) by default.
progress_json_path pathlib.Path
progress_json_path pathlib.PathWhere the progress file is written.
close()
close()def close(self, final_snapshot: TelemetrySnapshot) -> None: ...Force-write final_snapshot, bypassing interval.
final_snapshotTelemetrySnapshot — The run’s final (terminal-status) snapshot.
publish()
publish()def publish(self, snapshot: TelemetrySnapshot) -> None: ...Write snapshot, subject to interval throttling.
snapshotTelemetrySnapshot — The snapshot to write.
TelemetryCollector
TelemetryCollectorclass TelemetryCollector(builtins.object):
def __init__(self, history_size: int = 1000) -> None: ...Owns the latest TelemetrySnapshot and a bounded in-memory
history; updated from the solve loop’s own progress callback.
See this module’s own docstring for the threading model: every
access below is under one threading.Lock, correct with or
without the GIL.
history_sizeint — Maximum number of past snapshots kept in memory (oldest dropped first). ~1000 snapshots is a few hundred KB at this dataclass’ size.
history tuple[TelemetrySnapshot, ...]
history tuple[TelemetrySnapshot, ...]A point-in-time copy of the bounded history, oldest first.
latest TelemetrySnapshot | None
latest TelemetrySnapshot | NoneThe most recently update()-d snapshot, or None.
update()
update()def update(self, snapshot: TelemetrySnapshot) -> None: ...Swap in a new latest snapshot and append it to the history.
snapshotTelemetrySnapshot — The new latest snapshot.
TelemetryOutput
TelemetryOutputclass TelemetryOutput(builtins.object):
def __init__(self, count: int, bytes_written: int) -> None: ...Generic, physics-blind bookkeeping for one output kind.
countint — Number of items of this kind produced so far (e.g.1for a receivers file, however many test masses for Newtonian-noise — whatever the registering caller counts as one item of this kind).bytes_writtenint — Bytes written to disk for this kind so far,0until the corresponding file actually exists.
to_dict()
to_dict()def to_dict(self) -> dict[str, int]: ...JSON-safe representation.
TelemetryProgress
TelemetryProgressclass TelemetryProgress(builtins.object):
def __init__(
self,
step: int,
n_steps: int,
sim_time: float,
wall_elapsed: float,
eta_seconds: float | None,
steps_per_second: float | None,
) -> None: ...Where the run is in its time-stepping loop.
stepint — Steps completed so far.n_stepsint — Total steps this run will take.0before the time axis is known (thesetupphase).sim_timefloat — Simulated time elapsed, in seconds.wall_elapsedfloat — Wall-clock time elapsed since the run started, in seconds.eta_secondsfloat | None — Predicted wall-clock time to completion, fromRunningDurationEstimator.Nonebefore any post-warm-up step has been timed.steps_per_secondfloat | None — Recent mean throughput (steps timed so far divided by the wall-clock timeRunningDurationEstimatorattributes to them).Noneunder the same condition aseta_seconds.
to_dict()
to_dict()def to_dict(self) -> dict[str, typing.Any]: ...JSON-safe representation.
TelemetryResources
TelemetryResourcesclass TelemetryResources(builtins.object):
def __init__(
self,
rss_bytes: int,
peak_rss_bytes: int,
cpu_percent: float,
num_threads: int,
) -> None: ...Process resource usage at the moment a snapshot was built.
rss_bytesint — Current resident set size.peak_rss_bytesint — Largest RSS observed by this collector so far (not a true OS-level high-water mark, whichpsutildoes not expose portably — a running max over every snapshot taken).cpu_percentfloat —psutil.Process.cpu_percent(), measured since the previous call on the same process handle (0.0 on the very first call in a process, which is apsutilconvention, not a real “no CPU used” reading).num_threadsint — OS-level thread count of the current process (psutil.Process.num_threads()) — independent ofTelemetryRun.num_threads, which is the solver’s own requestedexecution_policythread count.
to_dict()
to_dict()def to_dict(self) -> dict[str, typing.Any]: ...JSON-safe representation.
TelemetryRun
TelemetryRunclass TelemetryRun(builtins.object):
def __init__(
self,
job_id: str,
backend: str,
dtype: str,
solution_order: int | None,
num_ranks: int,
num_threads: int,
start_time: str,
status: RunStatus,
error: str | None = None,
) -> None: ...Static-ish facts about the run itself.
job_idstr — Stable identifier for this run, shared withNGInProcessJob.job_namewhen one exists.backendstr —solver.backend("python","cpu", or"cuda").dtypestr —solver.dtype("float32"or"float64").solution_orderint | None —solver.solution_order, if resolved yet;Noneduringsetup.num_ranksint —solver.execution_policy.num_ranks.num_threadsint —solver.execution_policy.num_threads_per_rank.start_timestr — When the run started, formatted likelast_written(_LAST_WRITTEN_FORMAT).statusRunStatus — One ofsetup,running,finished,failed,cancelled.errorstr | None — The failing exception’s type name (e.g."ValueError"), whenstatus == "failed";Noneotherwise.
to_dict()
to_dict()def to_dict(self) -> dict[str, typing.Any]: ...JSON-safe representation.
TelemetrySink
TelemetrySinkclass TelemetrySink(typing.Protocol):
def __init__(self):
...Something a TelemetrySnapshot can be published to.
publish must never raise into the solve loop — implementations
are responsible for catching and swallowing their own IO failures,
since a telemetry sink must never be able to take a real solve down
with it.
close()
close()def close(self, final_snapshot: TelemetrySnapshot) -> None: ...Publish final_snapshot unconditionally and release resources.
final_snapshotTelemetrySnapshot — The run’s final (terminal-status) snapshot.
publish()
publish()def publish(self, snapshot: TelemetrySnapshot) -> None: ...Publish one snapshot, subject to the sink’s own cadence.
snapshotTelemetrySnapshot — The snapshot to publish.
TelemetrySnapshot
TelemetrySnapshotclass TelemetrySnapshot(builtins.object):
def __init__(
self,
progress: TelemetryProgress,
run: TelemetryRun,
timings: dict[str, typing.Any] | None,
resources: TelemetryResources,
outputs: dict[str, TelemetryOutput],
source: str = "in_process",
version: int = 2,
) -> None: ...One immutable point-in-time view of a running (or finished) the solver job.
progressTelemetryProgress — Time-stepping progress.runTelemetryRun — Static-ish run facts, includingstatus.timingsdict[str, typing.Any] | None — The liveTimertree, in the same shape assalvus.flow.ng.in_process_job._serialize_timerproduces for the final, stopped tree (label/duration/duration_unit/percentage/optionalchildren) — seelive_timer_snapshotbelow for how a still-running node’sdurationis estimated.Nonebefore any timer exists (immediately atsetup).resourcesTelemetryResources — Process resource usage.outputsdict[str, TelemetryOutput] — Per output kind (e.g."receivers","newtonian_noise"), keyed by a short lowercase name.sourcestr — Where this snapshot came from; always"in_process"for T1 (the only transport that exists yet).versionint —TELEMETRY_SNAPSHOT_VERSIONat construction time.
to_dict()
to_dict()def to_dict(self) -> dict[str, typing.Any]: ...JSON-safe representation, nested plain dicts throughout.
ZmqSink
ZmqSinkclass ZmqSink(builtins.object):
def __init__(
self,
collector: TelemetryCollector,
output_folder: pathlib.Path,
bind_address: str = "127.0.0.1",
poll_timeout_ms: int = 200,
) -> None: ...The live, fileless telemetry channel: one daemon thread owning a
zmq.REP socket, bound to a random local port, answering
snapshot/history/cancel/ping requests — JSON in, JSON out,
authenticated by a random per-run token no request can guess.
Unlike FileSink, this sink never actively writes a snapshot
anywhere on publish() — the socket answers on demand, reading
straight from the TelemetryCollector it was constructed with
(already updated by the solve loop’s own collector.update() call,
made right alongside sink.publish()). publish() is therefore a
no-op; only close() does real work (stopping the thread and
tearing down the endpoint file).
The one handshake a separate process needs is
<output_folder>/telemetry_endpoint.json ({host, port, token, pid, started_at, schema_version}), written once at bind time and
removed at close().
collectorTelemetryCollector — Read directly by every request; never mutated here exceptcancel_requested.set()for a"cancel"request.output_folderpathlib.Path — Where the endpoint descriptor is written.bind_addressstr — The interface to bind to. Left at its default ("127.0.0.1", localhost-only) unless a caller explicitly opts into"0.0.0.0"— this socket has no encryption, only a shared-secret token, so exposing it beyond localhost is a deliberate choice, not this class’ own default.poll_timeout_msint — How often the server thread’szmq.Pollerwakes up to checkclose()’s stop signal even with no request pending — bounds shutdown latency, not a correctness knob.
port int
port intThe bound TCP port (useful for tests; not part of the
stable API a consumer should rely on — read telemetry_endpoint.json
instead).
close()
close()def close(self, final_snapshot: TelemetrySnapshot) -> None: ...Stop the server thread, close the socket, and remove the
endpoint file — final_snapshot is unused (it is already the
collector’s own latest by the time close() is ever called,
matching every other TelemetrySink.close() in this module),
kept only to satisfy the shared TelemetrySink protocol.
final_snapshotTelemetrySnapshot — Unused.
publish()
publish()def publish(self, snapshot: TelemetrySnapshot) -> None: ...No-op: requests are answered on demand from the collector.
snapshotTelemetrySnapshot — Unused.