# Graph parameters, calibration and system identification The compiled step of a graph is ``` step_fn(state, external_inputs, params) ``` `state` is what the integrators advance, `external_inputs` is what the outside world supplies each step, and `params` is every time-invariant constant a {term}`node ` reads that someone might want to differentiate or change at runtime — stiffness, mass, diffusivity, gravity. `GraphManager.params` holds the compile-time snapshot: ```python import jax import jax.numpy as jnp from maddening import GraphManager from maddening.nodes import BallNode, SpringDamperNode gm = GraphManager() gm.add_node(SpringDamperNode("spring", 0.01, stiffness=30.0, damping=2.0, initial_position=0.5)) gm.add_node(BallNode("ball", 0.01, elasticity=0.7)) gm.compile() gm.params # {"nodes": {"spring": {"stiffness": Array(30.), "damping": Array(2.), ...}, # "ball": {"gravity": Array(-9.81), "elasticity": Array(0.7), ...}}, # "mappings": {}} ``` Every run method takes an optional `params=`; `None` means the snapshot. An explicit {term}`pytree ` is a *traced input*: a new value takes effect without recompiling, and `jax.grad` / `jax.jvp` / `jax.jacfwd` reach it — through {term}`coupling groups ` too, because the {term}`implicit-function-theorem ` rule carries the parameter dependence. ```python p = jax.tree.map(lambda x: x, gm.params) p["nodes"]["spring"]["stiffness"] = jnp.asarray(45.0) gm.run_scan(1000, params=p) # no recompile def loss(p): final = gm._compiled_step(gm._state, gm._default_external_inputs(), p) return final["spring"]["position"] ** 2 jax.grad(loss)(gm.params)["nodes"]["spring"]["stiffness"] ``` ## Which nodes take part A node opts in by declaring a keyword-only `params` on `update` and reading its constants from it: ```python def update(self, state, boundary_inputs, dt, *, params=None): p = self.params if params is None else {**self.params, **params} k = p["stiffness"] ... ``` `SimulationNode.params_pytree()` decides which entries of `self.params` appear in the pytree: every float-valued one by default (Python floats, float arrays, lists of numbers); ints, bools, strings and dicts are structural and stay on the recompile path. Nodes on the 3-argument contract keep working, but their constants are baked into the trace and are **not** in `gm.params`. `gm.nodes_without_params()` lists them, and passing an entry for such a node (or a misspelled key) is a `ValueError`, not a silently ignored leaf. The same holds for a leaf that *is* in `gm.params` but that the compiled step cannot read: an `initial_*` entry (only `initial_state()` reads it, from the node), a parameter a node bakes into a static when it is constructed (`HeatNode`'s `grid_points` on a non-uniform grid, `WaveletAdaptiveNode`'s `mass` -- declared by `static_data_deps`), or geometry a node consumed in `__init__`. (A leaf only *this* graph does not exercise -- a ball's `elasticity` with no table {term}`edge ` -- is not one of them: the node reads it as soon as the input arrives, and the value carried in `gm.params` is then the one used, so it is kept and serialised.) Changing one in `gm.params` is a `ValueError` naming the leaf and why, raised by every run method, `check_params`, `to_dict` and `save_state`; before 0.4.0 the edit was ignored by the step and then written out by `to_dict`, so the saved graph reloaded as a different model. To change such a value, rebuild the node with it; `gm.reset_params()` drops the edit. Whether the step reads a leaf is decided from the node's `static_data_deps` declaration and, failing that, from one trace of the compiled step and one of the node's own hooks with every declared {term}`boundary input ` supplied (both taken only when a leaf differs from its node's value, once per compile); a value that also reaches the node is not refused here, because the node's own value is the reference -- `PUT /graph/params` writes both, so it takes the decision itself (see "Writing parameters over REST" below) -- and a traced leaf (inside a fit or an {term}`FIM`) is never compared. An explicit `params=` pytree is not refused either: it is never serialised, and a leaf the step ignores may be one your own code consumes (a residual that seeds the initial state from `initial_velocity`); the live leaves a partial pytree is completed from are checked as `gm.params`. A misspelt key written to the node itself is not caught. `gm.get_node("spring").params["stiffnes"] = 45.0` is accepted with no warning and changes nothing in the step: five steps gave a state bit for bit equal to an untouched graph's. The key then appears as a new leaf of `gm.params["nodes"]["spring"]`, and `to_dict()` saves it. The same key written to `gm.params`, or passed in `step(params=)`, is refused when the graph next runs: `ValueError: params['nodes']['spring'] has unknown key(s) ['stiffnes']; SpringDamperNode.params_pytree() exposes [...]`. So write constants through `gm.params` when a misspelling must not pass silently. A value that carries a floating dtype of its own (an array, a numpy scalar) keeps it; a value that carries none (a Python float, a list of them) is placed at {term}`JAX`'s canonical float precision — float32, or float64 under `jax_enable_x64`. Nothing is narrowed below that, so a graph run under x64 has float64 constants as well as float64 arithmetic. A node that exposes boundary fluxes takes `params` there too and reads the same constants from it: ```python def compute_boundary_fluxes(self, state, boundary_inputs, dt, *, params=None): p = self.params if params is None else {**self.params, **params} ... ``` The graph passes the node's entry on every flux evaluation, so a calibrated stiffness changes the force a flux edge *delivers*, not only the node's own integration. The same rule applies to `compute_interface_correction(..., *, params=None)` for nodes that recompute coupled interface cells (`HeatNode`). A flux producer whose `update` takes `params` but whose `compute_boundary_fluxes` does not is exactly the trap the 2026-09 audit found; `verify_node` now fails it. `verify_node` checks the contract for you: `params_consistent` (injected params reproduce the baked step *and* fluxes), `params_gradient_finite`, and `params_effective` (every trainable leaf actually influences the outputs, fluxes included, the way a constructed value does — the check that catches a constant still read from `self.params`, or split between the injected value and a copy made in `__init__`, when the node rebuilds from `to_dict()`). See [verification](../developer_guide/verification.md). ### Writing parameters over REST `PUT /graph/params/{node}` writes a value to the node as well as to `gm.params`, and writing `node.params` rebuilds nothing the node derived from the value when it was constructed. So before anything is written the server asks the graph whether the running node would use the new value: * a leaf the compiled step (or the node's own hooks, with every declared boundary input supplied) reads from the injected params takes effect on the next step, without a recompile; * a structural value (an int, a bool, a string, or any constant of a node on the 3-argument contract) is accepted when the node's hooks trace differently with it; the graph is marked dirty and the recompile uses it; * otherwise, a value `initial_state()` reads -- an `initial_*` condition -- is accepted and takes effect at the next `POST /sim/reset`; * anything else is a **400** naming the parameter and why: a `static_data_deps` entry (`WaveletAdaptiveNode.mass`), or a value nothing the running node computes reads (`LBMPipeNode.pipe_radius`, baked into the wall mask; `propeller_x`; `initial_rho_liquid`, which `initial_state` reads from a copy). Nothing in the request is written. To change such a value, rebuild the node: `DELETE /graph/nodes/{node}`, then `POST /graph/nodes` with the new value. A 200 also promises that the graph a save would reload runs what the running graph runs, so two more questions are asked, of the whole request at once and with the params `to_dict()` would carry (every other key's live value included, which a fit may have moved). The node's constructor must take them: a `HeatNode` `thermal_diffusivity` past its Fourier limit, or a pipe's `rho_gas` above its `rho_liquid` (alone or written together with a new `rho_liquid`), is a 400. And where the rebuilt node holds anything its constructor derives differently from the running one, both are traced on the same state and params and both initial states are built; a write they compute differently with is a 400 -- `LBMPipeNode`'s `G` crossing zero, which picks the single- or multiphase branch at construction. A non-finite number anywhere in the request is a 400 before anything else, integers are bounded as in `POST /graph/nodes` (a 422), and a value that would change the state's layout, or take it past the API's state cap, is refused before a node of that size is built wherever its `initial_state()` can be evaluated abstractly. Two more rules close what that leaves. A value the node's step cannot run with is a 400 naming the error: the node's hooks are traced, on a copy, with the request's values, and a trace that raises where the current values' trace does not is refused (`RigidBodyNode` `constraints: {"w": 0}` used to answer 200 and make every later step a 500). And a structural value is stored in its parameter's own numeric type: a float with no fractional part written for an integer parameter is stored as that integer (`stencil_order: 4.0` is `4`), any other float for one is a 400, and an integer written for a float parameter is stored as a float. Before 0.4.0 the raw value was stored, and `stencil_order: 4.0` became a trainable leaf of `gm.params`. The size of what a write, or a new node, would build is checked before anything is built. `HeatNode`, `LBMNode`, `LBMPipeNode` and `WaveletAdaptiveNode` estimate, from their parameters alone, the state they build and the memory their constructor takes; `POST /graph/nodes` and `PUT /graph/params` refuse a node over `MAX_NODE_STATE_ELEMENTS` state elements or `MAX_NODE_BUILD_BYTES` of build memory before calling its constructor, and `GraphManager.from_dict` refuses (`ValueError`) one that would not fit in the machine's memory. A node class without an estimate is checked on the state it builds, as before. `python -m maddening.examples.servers.rest_params_demo` walks through these answers in-process (FastAPI's `TestClient`, no port, presenting the server's token as an in-process client must): a write honoured on the next step without a recompile, a bound and a constructor refusal, and an initial condition applied at reset. Before 0.4.0 such a write answered 200, was served by `GET`, was ignored by every step (even after `POST /graph/compile`) and was saved by `to_dict()` and `save_state()`, so the reloaded graph ran a different model. The check runs the node's code on a shallow copy that reads the new value, never on the node itself. When no faithful copy can be made (a node holding a method bound to itself) the write is refused, with a 400 saying so. When the copy's code raises with the new value where it does not with the current one, the write is refused as one the step cannot run with; when it raises with both, or needs a concrete value, that question decides nothing. It detects "no path at all": a value a node consumes at construction *and* reads again later passes, so a node that bakes a parameter should declare it in `static_data_deps`, which refuses it on every surface. Likewise `POST /checkpoint/load` refuses, and undoes, a {term}`checkpoint ` whose parameters include another value of one the node consumed at construction (a checkpoint of a pipe built with another radius); before, the load succeeded and every later `/sim/step` failed. More generally the load asks of every parameter it changes what `PUT /graph/params` asks, by the same code: a non-finite value, one outside its {codeterm}`ParamSpec` bounds, one the node's constructor refuses with the graph's other values (a rod past its Fourier limit at this graph's timestep), one that moves a mapped edge's points -- each is a 400 and nothing is loaded. Finiteness and the bounds are asked, per element, of a value that is neither the leaf's now nor the node's own, so a graph built outside its bounds reloads its own checkpoint. `GraphManager.load_state` in Python does not ask these: like a `gm.params` write, it may hold a value outside a spec's bounds on purpose, and a graph whose parameters Python moved outside them must resume its own checkpoint into a freshly built graph. It refuses only text and booleans for a numeric leaf, which no save writes. `POST /graph/nodes` applies the same `ParamSpec` bounds to the values it is given, and refuses a boolean or text for a numeric parameter. ### Live values, recompiles and partial pytrees `gm.params` is populated by `compile()` and **survives a recompile**: a calibrated leaf whose node, key, shape and dtype still exist is carried over when you add an edge or an {term}`external input `, replace a node, or the profiler recompiles behind your back. Leaves that no longer fit are dropped with a `RuntimeWarning`. `gm.reset_params()` is the explicit way back to the constructor snapshot. A checkpoint loaded before the first compile compiles the graph so its params are not lost. **The later of a `node.params` write and a `gm.params` write wins.** Writing `gm.params` takes effect on the next step with no recompile, and is the way to change a constant mid-run. A `node.params` write reaches the graph the next time anything reads `gm.params` -- your code, `save_state`, `to_dict`, the {term}`FMU` export, `GET /graph/params`, a `jax.jit` of a {term}`sysid` loss handed `gm.params` -- or runs it (`step`, `run`, every `run_scan*`, the fitters, `windowed_loss`), or compiles it, whichever comes first: one sync point takes it in. A constant `gm.params` carries is copied into it (no recompile); anything else -- a structural value, any constant of a node on the three-argument contract -- marks the graph dirty, and the next run recompiles, so every entry point runs the same model. An export cannot recompile the step it hands out, so the FMU export refuses a graph left dirty this way, asking for `compile()` first; until 0.4.0's fix it ran the old model. `node.params` counts its writes, so writing a constructor value back is a write (it reverts a calibration). Assigning it -- `node.params = {**node.params, "k": v}` -- stores the items in a fresh counting mapping, so writes made through `node.params` afterwards count as well (a reference to the dict you assigned is not `node.params`: write through `node.params`); and an element written in place into a list, a NumPy array or a nested dict (`node.params["rates"][0] = 5.0`) is found by comparing each such value with a copy taken at the last sync, which costs a copy and a comparison of those values per sync. Until 0.4.0's fix both were lost to `gm.params`, every run, `compile()` and `to_dict()`. **The assignment is a write of the keys whose values it changes** -- a key added, dropped, or given another value, bit for bit -- **and of no other**: the keys it hands back unchanged are the node's own values, not writes, so a constant of that node you calibrated in `gm.params` stays calibrated. It is the rule the in-place write follows, and the one way the two spellings of a write differ: `node.params = {**node.params, "k": v}` with the `v` the node already held writes nothing, while `node.params["k"] = v` is a write whatever the value. Write the key to put a node's own value back over a calibration. (For a time during 0.4.0 development the assignment was read as a write of every key, and the one-key idiom put the node's own value over every other calibrated constant of the node, without a word.) A `gm.params` write reads `gm.params` first, which takes in any pending node write, so the two are always ordered and the later wins; the one exception is a `gm.params` leaf written through a reference held across a node write, with no read in between, which the node write replaces. `gm.params = tree` is later than every node write before it, and `load_state()` restores `gm.params` later than them too. (Until 0.4.0 a recompile kept every live leaf, so a `node.params` write after the first compile was dropped without a word -- a `HeartPumpNode` set to 144 bpm and recompiled went on at 72 -- and then for a time it reached the entry points that ran the graph but not the code that read `gm.params`.) All of this holds for a node inside a wrapper as for a node on its own. A wrapper (`ShardedStencilNode`, `ShardedPointwiseNode`, `ShardedUnstructuredNode`, `HybridNode`, and one of them around another) has no `params` of its own: `gm.get_node(name).params` *is* the wrapped node's mapping, so a keyed write, an assignment and an element written in place are the same write whether you make them on the wrapper or on the node you wrapped. (Until 0.4.0's fix the two only shared a mapping from construction on, and an assignment on the wrapper replaced the wrapper's alone: the step went on with the old value, `node.params` read the new one back, and every later keyed write through the wrapper was lost: MADD-ANO-278.) **A constant belongs in `params`.** One a node keeps as an attribute (`self.stiffness = stiffness`) is outside everything above. Writing it after `compile()` (`node.stiffness = 40.0`) is not a counted write, so nothing sees it until the next compile, and each entry point differently: `step()` and a `run_scan` length already traced go on with the old value, while a `run_scan` at a length not traced before runs the new one. Call `gm.compile()` after such a write, or move the constant into `params`; no save carries it either (see [where a constant belongs](../developer_guide/node_authoring.md#where-a-constant-belongs-and-what-a-saved-graph-carries)). And `node.params` takes any key: a misspelt one (`node.params["stifness"] = 40.0`) is accepted, changes nothing the step reads, and after the recompile it causes is a new leaf of `gm.params` and an entry of every config saved from then on, where the same key written into `gm.params` or passed to `step(params=)` is a `ValueError`. Such a config no longer loads: `from_dict` hands the key to the node's constructor, which raises `TypeError` naming it (`SpringDamperNode.__init__() got an unexpected keyword argument 'stifness'`) unless the class takes `**kwargs`. Delete the key (`del node.params["stifness"]`) before saving; the recompile that follows says, with a `RuntimeWarning`, that it drops the leaf. `gm.params_table()` lists the keys there are. ```python gm.get_node("ball").params["elasticity"] = 0.6 # on the node, after compile assert float(gm.params["nodes"]["ball"]["elasticity"]) == float(jnp.float32(0.6)) gm.params["nodes"]["ball"]["elasticity"] = jnp.float32(0.7) # later: it wins gm.compile() assert float(gm.params["nodes"]["ball"]["elasticity"]) == float(jnp.float32(0.7)) ``` The graph's *state* survives the same recompile, and so does the internal bookkeeping that goes with it: a {term}`multi-rate ` graph keeps its sub-step phase and a coupling group keeps its predictor history and IQN warm start, so a mid-run edit changes no number. Only a change that moves a node's rate divider restarts the phase, because the sub-step it counts then means something else. `gm.reset_state()` is the explicit way to zero all of it. A *partial* pytree passed to `gm.step(params=...)` / `gm.run_scan` / `gm.run` is completed from the **live** `gm.params` (a missing node or key keeps its calibrated value, not its constructor constant). The raw compiled step (`gm._compiled_step`, what the {term}`FMI` {term}`sidecar ` calls) refuses an incomplete pytree instead of guessing. ### Warnings when several threads run graphs Much of what this page promises is said as a Python warning: a compile-time advisory, a leaf dropped at a recompile, a rank decided at the {term}`precision floor `. Python keeps one list of warning filters for the whole process, so what one thread does to that list, every thread lives with. To check a write, MADDENING runs the node's own code a second time -- a trace of the step to see which leaves it reads, the constructor with the new value, `initial_state()` for the state's layout -- and silences what that code warns, because the real run says it. The same is done when a {term}`sharded ` wrapper checks its inner node's state and when the profiler compiles its one-iteration variant. That silence covers **the thread that is probing and no other**, and nothing is saved or put back: graphs, servers and profilers running in other threads keep every warning of their own, and the process's filters afterwards are the ones before. (During 0.4.0's development the probes used `warnings.catch_warnings()`, which does neither; no release carried it. MADD-ANO-204.) **What MADDENING cannot do** is make `warnings.catch_warnings()` safe for *other* code that uses it while your threads run. That block saves the process's filter list on the way in and puts it back on the way out. Two of them that overlap in two threads can put back each other's list, and a filter one of them set inside -- often `ignore` -- then stays for good, with no sign. (Python 3.14 changes this only when started with `-X context_aware_warnings`, the default of its free-threaded build; MADDENING is tested on neither.) Where such a block overlaps one of MADDENING's probes the damage to MADDENING is bounded: at worst that probe is not silenced for the rest of its run, so what the node's code warns there is shown -- or, where warnings are errors, raised, which the probe reads as "cannot tell". Nothing of MADDENING's that ignores a warning is left in the filters. Two foreign blocks can still trade filters with each other. Libraries on MADDENING's own paths that use the block, found by recording every `catch_warnings` entered (JAX 0.11.0, NumPy 2.4.6, lineax 0.0.7, FastAPI 0.136.1, pydantic 2.13.4): | path | who opens a block | what it sets inside | |---|---|---| | `compile()`, `step()`, `run()`, `run_scan()`, a `gm.params` write, `save_state()` / `load_state()` | nobody (JAX and NumPy use the block only in their test utilities and at import) | -- | | tracing a gradient through a coupling group (the implicit solve's backward pass: `jax.grad` of a run, a `sysid` fit) | lineax, twice per trace | `ignore`, for every warning | | `SimulationServer.create_app()` | FastAPI, 48 times for a server with the built-in routes | `ignore` for `UserWarning`, and for one pydantic category | | the first `GET /openapi.json` of each app | FastAPI, 44 times | `ignore` for one pydantic category | `create_app()` builds one app at a time since 0.4.0 (MADD-ANO-191). The lineax blocks are a few microseconds long: four threads each tracing such a gradient at once changed the filters in 0 rounds of 30, and left an `ignore` for every warning in 1 round of 10 once every block was held open a millisecond longer. So it is rare, and it is not excluded. What to do in a process with more than one thread: - **Set your filters once, at start-up**, before any thread starts: `warnings.simplefilter(...)`, `-W`, or `PYTHONWARNINGS`. A filter set that way is never saved or put back. - **Do not open `warnings.catch_warnings()` -- or `pytest.warns`, which is one -- in a thread while other threads run graphs.** Record warnings in the main thread, with the others idle. - **Take a gradient through a coupling group in one thread at a time**, unless it is compiled with `jax.jit` and was traced before the threads started: only the trace opens lineax's blocks. - To see whether a process has lost its warnings, look for `('ignore', None, Warning, None, 0)` at the front of `warnings.filters`. ## `ParamSpec`: what an optimiser may do Each leaf carries a `ParamSpec` (`maddening.core.params`): | field | meaning | |-------|---------| | `trainable` | may an optimiser move it (default `True`; `initial_*` entries default to `False`) | | `bounds` | physical range, `(lo, hi)` with `None` for open | | `transform` | `None` (clip to bounds; a leaf exactly on a bound has derivative 1, the one-sided one into its range, where `jnp.clip` alone gives 0.5), `"log"` (`p = lo + exp(u)`, strictly above `lo`, or above 0 when `lo` is `None` -- `check` refuses anything else), `"logit"` (`lo < p < hi`) | Nodes declare specs for their own constants in `param_specs()` (`SpringDamperNode`: stiffness and mass are `log`-positive, damping is `>= 0`). A graph overrides any of them: ```python from maddening.core.params import ParamSpec gm.set_param_spec("spring", "mass", ParamSpec(trainable=False)) gm.trainable_mask() # params-shaped pytree of bools u = gm.unconstrain() # optimiser coordinates (log k, log m, ...) gm.constrain(u) # back to physical values, always inside bounds gm.check_params(p) # ValueError naming the first leaf out of range ``` `gm.constrain(gm.unconstrain(p))` returns `p` to the transform's own resolution, which its bounds set and not the value: within 16 of its spacings -- `eps * max(|lo|, |hi|, hi - lo)` for `logit`, `eps * |lo|` for `log` -- plus 16 `eps` of the value, because the maps rebuild the value as `lo + ...` in the leaf's dtype. Under bounds far wider than the value that is coarse -- a float32 `2.0` under `logit` bounds `(-1e6, 1e6)` comes back up to 0.12 away, and not at the same float from eager and jitted code -- so give a transformed parameter bounds near its own size, or `transform=None`. The fitters make that map once per iterate, run the model on the arrays it returns and hand those arrays back, so what they report is always the loss of what they return; and they warn (`PrecisionLimitWarning`, naming the leaf) about a parameter they are asked to fit whose value its transform resolves to worse than `sqrt(eps)` of itself. A leaf a fit does not move is never passed through its transform at all. ## System identification: `maddening.sysid` ```python from maddening.sysid import (fim, fim_core, fit, observations_from_history, windowed_loss) gm.reset_state() # the spring from its initial position, not at rest init = {n: gm.get_node_state(n) for n in gm.node_names} _, hist = gm.run_scan_with_history(1000) obs = observations_from_history(init, hist) # T = 1001 samples loss = lambda p: windowed_loss( gm, p, obs, obs_fn=lambda h: h["spring"]["position"], window=50) ``` `python -m maddening.examples.advanced.sysid_demo` runs this section end to end on a spring recorded with noise: the FIM's rank-2-of-3 verdict for `(k, c, m)`, `fit` holding the undetermined scale (`excited_rank`, `hold_declined`), `fit_lm` recovering `k` and `c` after a `ParamSpec` freeze and under a `mask=`, `params_table()` before and after, and the {term}`Cramér–Rao bounds ` the fit lands within. Its residual is a `run_sweep` rollout, which does not advance the graph. Record data that can tell the parameters apart. `run_scan` leaves the graph at its final state, and the 1000 steps run above left the spring at rest, where its position moves by about `3e-5` and says almost nothing about `k` or `c`. `gm.reset_state()` starts it again from `initial_position=0.5`, so the record holds the whole transient. `windowed_loss` is teacher-forced: every window restarts from the measured state, so gradients cannot compound over a long stiff rollout (`mask_unconverged=True` drops, from the loss and from its gradient, windows in which a coupling group exited at `max_iterations` unconverged, including a window that diverged). On a multi-rate graph each window also restarts on the sub-step its first sample was recorded at, which needs to know where the record began: pass `start_step=` (0 for a record taken from `compile()` or `reset_state()`, `n` after `gm.run(n)`; {term}`experimental `). A multi-rate graph without it warns that 0 is assumed; a record that began elsewhere would be replayed on the wrong phase in every window. A coupling group's predictor history and IQN-IMVJ warm starts are not in the observations either, so they are replayed: each window starts from the ones the previous window ended with (gradient stopped), the first from the cold ones `compile()` and `reset_state()` leave, and the loss is exactly zero at the parameters that generated a record taken from either. A record that began after the graph had stepped (`start_step > 0`) cannot be replayed exactly, and says so. Before fitting, ask what the data can identify: ```python report = fim(lambda p: residual(p), gm.params, mask=gm.trainable_mask()) report.rank # < len(param_names) => directions the data misses report.crb # +inf for every parameter those directions spoil report.least_identifiable() # ("['nodes']['spring']['mass']", 0.58) report.eigvecs[:, 0] # the weakest direction, in relative coordinates ``` For a spring observed through position only, `k`, `c` and `m` enter as `k/m` and `c/m`: scaling all three together is invisible, and the FIM's weakest eigenvector is `(1, 1, 1)/√3`. `rank` is 2 of 3, and all three bounds are `+inf` — each parameter lies partly along the invisible direction, so none of them is separately determined. Read `rank` rather than `cond` for that verdict: `cond` is `eigvals[-1] / eigvals[0]`, which for a direction that is not in the data is `inf` or merely enormous (of order `1e14` for this spring in float32) as the rounding of the sensitivities falls, whereas `rank` is held to a stated cutoff -- an eigenvalue counts when it is above `max(n, sqrt(m)) * eps` of the largest, for `n` parameters and `m` residual rows -- and that threshold scales with the matrix (until the matrix itself leaves float32's normal range -- or flushes to exactly zero from a Jacobian that is not, at `|J|` below about `1e-19` -- which `fim` reports with a `PrecisionLimitWarning`, and `fim_core` with `precision_limited`). The eigenvalues are those of `JᵀJ` read from the Jacobian itself (its squared singular values), not from the matrix in `report.fim`. Each entry of that matrix is a sum over the `m` rows, rounded at the working precision, and a record whose sensitivities are constant or piecewise constant -- a steady state, a step -- makes every rounding fall the same way: the entries are then off by hundreds of `eps`, enough to make two parameters that enter the model only through their sum look separately identifiable. So take identifiability from `rank` and `crb`, and do not decide it by inverting or decomposing `report.fim` yourself. With the default cutoff a direction that is exactly unidentifiable is never counted while the sensitivities are right to about `sqrt(eps)` relative (`3e-4` in float32, `1e-8` in float64), whatever the record's length, and a direction is counted only when it is at least `sqrt(cutoff)` of the strongest in the Jacobian (in float32, `5e-4` for a short record and `3e-3` at 4,000 rows). An eigenvalue the float64 decomposition cannot tell from zero (a singular value at or under `max(m, n) * n * 2.2e-16` of the largest) is reported as exactly `0.0`, so two exactly proportional columns give `cond = inf`; that happens only below the cutoff in force, so it never changes `rank` or `crb`, and `rank_rtol=0.0` still counts every positive eigenvalue. A verdict within a factor of two of the cutoff comes with a `PrecisionLimitWarning`; re-running under `jax_enable_x64` with the parameters and the state in float64 settles it, since the sensitivities are then computed to float64's precision and the cutoff moves nine decades down. Freeze one of them, then fit: `scale="relative"` — the default, and the coordinates the eigenvectors above are in — multiplies each Jacobian column by the parameter's value, so a parameter sitting at exactly `0.0` (`SpringDamperNode`'s `initial_velocity` default) has no column at all and reads as unidentifiable however well the data determine it; `report.zero_scaled` names those. `scale="nominal"` removes the problem by taking the column scale from the parameter's `ParamSpec` instead — the width `hi - lo` of a finite `bounds`, a scale and not a location, so a symmetric range around zero is no longer a zero: ```python gm.set_param_spec("spring", "initial_velocity", ParamSpec(trainable=False, bounds=(-1.0, 1.0))) report = fim(residual, gm.params, scale="nominal", specs=gm.param_specs()) report.value_scaled # columns whose spec had no finite width ``` A spec with no finite width — `(0.0, None)`, `(None, None)`, a `"log"` constant — has no nominal scale, so that column keeps the value scaling (`p`, or `p - lo` under `"log"`) and is named in `report.value_scaled`: a nominal report says which columns were answered from the spec and which from the value, and a value-scaled zero still appears in `zero_scaled`. Nothing falls back to an absolute `1.0`, which would put units back into `cond` unannounced. The `fim` docstring tabulates the policy per spec. `specs` must mirror `params`: a nested dict for every dict level — and for every namedtuple or dataclass level, keyed by field name, since that is how JAX addresses a field — a `ParamSpec` at each leaf, and for a list/tuple level either a list of specs by position or one `ParamSpec` covering every position. A dict where a leaf needs a `ParamSpec` (`ParamSpec.to_dict()` output), a `ParamSpec` above a dict or record level, a list of specs for a namedtuple, or a `specs` that is not a dict is a `ValueError` naming the key path, and not only in `fim`: `trainable_mask`, `unconstrain`, `constrain` and `check_bounds` read `specs` through the same walk and refuse the same entries, where they used to hand the leaf the default (trainable, unbounded) spec. A key that matches *no* parameter is where the two differ. The tree maps ignore it, because `gm.param_specs()` declares specs for constants that are not leaves of `gm.params` — a uniform `HeatNode`'s `grid_points=None`, any constant spelled as a Python `int` — and `gm.check_params` hands them exactly that tree; a misspelt key is therefore not caught there. `fim(scale="nominal")` refuses one whenever it could have changed the report: a stray spec carrying a finite width or a `"log"` offset from a non-zero lower bound is a `ValueError`, while a stray spec whose column record is the default's (unbounded, one-sided, `(0, None)` under `"log"`) is accepted, because the report is bit-identical with or without it. That is what lets `specs=gm.param_specs()` through for the graph it came from, and it is also why slicing `params` to a sub-tree usually needs no matching slice of the specs: the node's whole spec dict works unless an entry the sub-tree does not reach carries a width. A leaf *without* an entry still gets the default spec and is named in `value_scaled`; `{}` remains the explicit "no leaf has a declared width". ### Asking the same question inside a loop `fim` is the reporting path: it reads the answer back to the host so it can raise on a non-finite matrix, warn when the verdict rests on rounding, and name the parameters `scale="relative"` found at zero. For a control loop, `fim_core` is the same computation with none of that — it returns device arrays, reads nothing back, and traces: ```python core_fn = jax.jit(functools.partial(fim_core, residual_fn)) core = core_fn(params) # no host sync at all ok = core.finite & ~core.precision_limited & (core.crb[i] < tol) ``` **Tracing freezes what the residual reads.** A residual usually takes a few leaves as its argument and reads the rest of the model from outside it — `gm.params` for the leaves it does not fit, an attribute of `self`, a window of data. Tracing reads those values once and compiles them in as constants. So `fim` re-traces on every call by default, and every call answers for the residual as it stands then; a 200-step spring-damper rollout measured ~230 ms per call that way on four pinned CPU cores. `fim(..., reuse_trace=True)` keeps the compiled Jacobian across calls with an equal residual function (plus the same scale, column set and nominal record) and costs ~1 ms warm — but it is only correct for a *pure* residual, one that depends on its argument and on nothing that can change between calls. With it on, a residual that reads `gm.params` reports the first call's matrix after those leaves change, and a bound method (equal across attribute accesses) reports the first call's matrix after its object changes. Both are silent. The same is true of `fim_core` under your own `jax.jit`: `core_fn` holds what `residual_fn` read when it was first traced. Pass anything that changes through the argument, or build a new `core_fn` when it changes. `FIMCore` carries the same verdicts as `FIMReport` — `rank`, `cond`, `crb`, `zero_scaled`, `value_scaled` — plus `finite`, which is what `fim` raises on, and `precision_limited`, which is what it warns on. Read them as a third outcome rather than a refusal: a `precision_limited` core means *verdict unavailable*, not *unidentifiable*. `crb` keeps `fim`'s polarity, `+inf` unless finiteness was positively established, so `crb < tol` is False for an unidentifiable parameter and for a `NaN` matrix alike. The two are not bit-identical and are not meant to be: both read the singular values of the Jacobian, but `fim_core` decomposes it at its own precision on the device and lets the compiler schedule `J.T @ J`, where `fim` decomposes it in float64 on the host and keeps the Gram product eager. Measured over 1,740 synthetic Jacobians (2 to 25 parameters, 20 to 4,000 rows, the deciding eigenvalue from 0.2 to 50 times the rank cutoff), they reached the same `(rank, precision_limited)` on every one. Past about 16,300 rows in float32 `fim_core`'s default cutoff is the larger of the two: it adds the conventional bound on a decomposition at that precision, `max(m, n) * eps` of the largest singular value, to the allowance for the sensitivities' own error, which makes it 6.1 times `fim`'s at 100,000 rows and 0.020 at a million (and resolves nothing from 4.1 million float32 rows). The bound is loose on CPU -- a rank-one float32 Jacobian of a million constant rows decomposes to a ratio of `2e-12` -- but it is not tied to one backend's arithmetic. Every direction `fim_core` reads between half of `fim`'s cutoff and twice its own is unresolved and `precision_limited` there: the verdict is unavailable, never silently withdrawn, and `fim` settles it. `fim` decomposes on the host, so it copies `J` back from the device and holds a float64 copy of it while it works: peak memory rose by about 4.5 times `J`'s size (690 MiB for a 4-million-row, 10-parameter float32 `J`). For a `J` that does not fit that comfortably, `fim_core` keeps the decomposition on the device. ```python gm.set_param_spec("spring", "mass", ParamSpec(trainable=False)) start = jax.tree.map(lambda x: x, gm.params) start["nodes"]["spring"]["stiffness"] = jnp.asarray(45.0) # the data's: 30 start["nodes"]["spring"]["damping"] = jnp.asarray(3.0) # the data's: 2 res = fit(gm, loss, params=start, n_iter=300, lr=0.1) res.params["nodes"]["spring"] # stiffness ~30, damping ~2: physical, inside bounds res.losses # per-iteration loss: 0.58 at the start, below 1e-10 by the end res.best_iteration, res.best_loss # which iterate `params` is, and its loss ``` `fit` returns the lowest-loss iterate it evaluated, which is not always its last. Adam's step is about `lr` in size however small the gradient, so near a minimum it can step past it and end the run higher than it was, and a fit started right next to the minimum can walk away from it. `best_iteration` counts the updates that produced `params` (0 is the start) and indexes `losses`: `res.losses[res.best_iteration] == res.best_loss`, except when it equals `len(res.losses)`. Then `params` is the iterate the last update produced, which the loop never evaluated, and `fit` evaluated it once more to compare it with the rest. A run whose loss never rose returns its last iterate. `fit_multiple_shooting` chooses the same way, taking the parameters and the window states from the same iterate, and `fit_lm` needs no choice: it accepts only a step that lowers the loss, so its last iterate is its lowest. `fit_lm` is the Gauss–Newton alternative: it reuses the `jacfwd` sensitivities `fim` computes, so with a handful of parameters it converges in a few iterations where Adam needs hundreds; give it a residual function rather than a scalar loss, and `noise_std` (a scalar or a per-leaf σ, also accepted by `fim`) to weight the residual so the Cramér–Rao bound comes out in the parameters' own units. Its `converged` is `True` when the loss reached `tol`, or when the step an iteration proposed moved every trainable parameter by no more than `step_tol` of its own magnitude -- whether or not that step lowered the loss, since at the float floor a fit's proposal rounds to nothing and cannot. The default `step_tol` is `2**4` ulps of each parameter's dtype, so a float32 fit that reaches its optimum reports it; and once the fit has lowered the loss, an iteration that rejects every candidate down to one within `step_tol` converges too: no step the tolerance resolves lowers the loss, which for a differentiable residual is its rounding floor (see "The residual must be differentiable" below for one that is not). Neither fires while a parameter on its bound could lower the loss by moving into its range. (`fit` and `fit_multiple_shooting` have only the `tol` test, so with the default `tol=0.0` their `converged` is always `False`; read `best_loss`.) The solve's floor is `eps` times each column's own curvature, so the step depends neither on the residual's units nor on any parameter's (a floor shared across columns once crushed the step of a parameter measured in small units, which then read as converged), and with the identifiability guard below making its tests in coordinates no change of units moves, neither does the answer `fit_lm` returns with its defaults. That holds across the whole float range, for any residual and any parameter whose residual entries (its own rounding at the optimum included) and Jacobian entries are normal numbers of the working precision -- in float32 a residual of `1e-30` or `1e30`, a parameter whose natural scale is `1e-23` (a 10 nm particle's volume in cubic metres) or `1e23` -- because the loss is computed on the residual framed by a power of two and the solve on Jacobian columns framed the same way, exactly; until 0.4.0's fix `r * r`, `JᵀJ` and `Jᵀr` flushed or overflowed there and `fit_lm` reported `converged=True` at a wrong point or at its start. It never reports `converged` on a solve whose columns it could not represent, or on a loss of `0.0` from a residual that is not zero; it warns instead. And neither test fires unless the undamped Gauss-Newton step from the iterate -- least squares on the equilibrated Jacobian, with no damping and no floor -- would also move every parameter by no more than `step_tol`, or, at the floor, would not lower the loss: a shrunken step cannot read as stationary. When that step would still lower the loss there it is taken as the next iterate, so the verdict at the floor does not depend on which units the damped candidates' rounding happened to favour. An on-bound gradient whose own Newton step is within `step_tol` is the residual's rounding and counts as zero, so a truth exactly on a bound converges on either bound. A float32 constant in an x64 graph is optimised on float32's grid, as the model sees it, so it converges in as few iterations as a float64 one. **The residual must be differentiable in the trained parameters along the fitted trajectory.** Every fitter here, and `fim`, reads derivatives, and a derivative of a step is the derivative of the branch that step took. A node with an event, a contact or a valve whose timing depends on a trained parameter breaks the condition: `BallNode`'s bounce and `HeartPumpNode`'s valve switch one time step earlier or later as a parameter moves, and the residual *jumps* there (MADD-ANO-021). On the bouncing ball of the first snippet, at the very parameters that generated a record, lowering the elasticity by one part in a million raises the loss from `1e-11` to `0.03`. What you see on such a residual: - `fit_lm` follows a smooth piece of the loss to its edge, where every step the tolerance resolves crosses the jump and is rejected. Its floor rule would read that as the rounding floor, so it first compares each small rejected step with the same step taken the other way: rounding is alike on both sides, a jump is not. Where it finds one the run ends `converged=False` with a `RuntimeWarning` that names a residual that is not differentiable; `res.params` is the lowest loss the run found, which may be far above the minimum (on that ball, 11 of 24 starts: losses of 0.02 to 3.7 against 1e-13). Start again from elsewhere, or smooth the event. A *kink* -- a continuous residual whose slope jumps -- is not detected: a run that ends in the crease of one reports `converged=True` at a point no step lowers the loss from. - `fit` and `fit_multiple_shooting` descend each smooth piece and cross jumps by chance; they return the lowest-loss iterate they evaluated and say nothing about whether it is a minimum (their `converged` is the `tol` test alone). - `fim` at such a point reports the rank and the bounds of the smooth piece the point is on, finite and with no warning. They bound nothing a jump can do: on that ball the bound on the elasticity is `1.3e-3` relative where a change of `1e-6` moves the weighted loss by 311. Every fitter keeps each optimiser coordinate where its transform can be stepped on: a `transform=None` leaf inside its bounds (clipped, with the one-sided derivative on a bound), and a `log` / `logit` leaf at least `sqrt(eps)` of its bounds' own size inside each bound (3.5e-4 of the range in float32, 1.5e-8 in float64), and short of where `exp` floors or overflows. Nearer a bound than that the transform is flat to the working precision: the distance to the bound has lost half its digits, and a unit step of the coordinate moves the value by a few float spacings. So **a `log` / `logit` parameter cannot be fitted onto its bound**: a fit that ends on that margin -- its data pull the parameter to the bound, or it started out there and could not come back within its budget -- says so in a `RuntimeWarning` naming the parameter. Declare a parameter that belongs on its bound with `transform=None`, which clips and can sit there. `fit_lm` reads each step of a `log` / `logit` coordinate along the transform's curve or along its tangent, whichever moves the value less. The two agree to first order. Where the transform is flat, the tangent's reading is the step the same parameter would take under `transform=None`, so a coordinate at the edge of its range comes back in one step; where a step heads for a bound, the curve's closes the distance by a factor of *e* at most and never lands on it. The test suite holds `fit_lm` to the consequence over a grid of this spring (each transform, the truth and the start anywhere in the range and a few float spacings inside each edge, both precisions): wherever the same fit with the parameter clipped instead of transformed recovers its optimum, the transformed one does too. `fit` and `fit_multiple_shooting` step a coordinate by about `lr` whatever the gradient, so they leave such an edge only over about `8 / lr` updates in float32 and `18 / lr` in float64 (more while the coordinate's gradient is below Adam's `eps` of the largest); they name a parameter started there before they run. Until 0.4.0's fix a coordinate one step carried to such an edge stayed there. A `transform=None` damping started at 4 against a truth of 0.05 landed on 0 and `fit_lm` called it converged. A `logit` coordinate's range ended where its derivative was a few `eps` of the bounds, from where every Levenberg-Marquardt candidate, `1/eps` long, landed on the opposite edge: under x64 on the spring above, a damping bounded to `(0.5, 2)` came back 2.0 for a truth of 1.9 from 20 of 60 starts, not converged and without a word, where the same fit with the bounds clipped recovered the truth from every one. For noisy data, `fit_multiple_shooting` replaces teacher forcing with free per-window initial states and a continuity penalty (`windowed_loss(..., window_states=, continuity_weight=)`), so the optimum is one continuous trajectory rather than windows each seeded with measurement error. It returns the fit and the window states, both from its lowest-loss iterate: ```python from maddening.sysid import fit_multiple_shooting mask = jax.tree.map(lambda _: False, gm.params) mask["nodes"]["spring"]["stiffness"] = mask["nodes"]["spring"]["damping"] = True ms, window_states = fit_multiple_shooting( gm, obs, obs_fn=lambda h: h["spring"]["position"], window=50, params=start, mask=mask, n_iter=300, lr=0.1) ms.params["nodes"]["spring"] # stiffness ~30, damping ~2 window_states["spring"]["position"].shape # (20,): one free start per window ``` The record above has no noise, so this lands where `fit` did. `python -m maddening.examples.advanced.multiple_shooting_fit` runs both fitters on a noisy record of a coupled pair (two masses joined by one spring, in a coupling group): teacher forcing misses the far mass by 9% on that record, and multiple shooting by 0.4%. All three fitters emit a `"fit_progress"` event to `gm.add_observer` callbacks every `notify_every` iterations. `fit` runs Adam in the unconstrained coordinates under the trainable mask, so positive constants stay positive. Every leaf *outside* the mask — frozen, or trainable but not selected — comes back bit-identical to the value passed in, so comparing a fit's input and output leaf by leaf says exactly which constants it touched. Gradients through the whole graph come from a single `jax.value_and_grad`; a non-finite gradient raises rather than continuing. A loss written in units so small that its gradient's products flush (`0.5 * ||s * r||²` with `s = 1e-19`) has its gradients taken with a power-of-two cotangent that lifts them out of the flush, exactly, so `fit` and `fit_multiple_shooting` move as they would at scale one; the loss's own value is yours and can still read `0.0` there, which they say once with a `RuntimeWarning` and which then does not meet `tol`. `fit` also keeps out of the directions the data cannot determine. Every gradient of a least-squares loss is `Jᵀr`, so a direction `v` with `Jv = 0` has `g·v = 0` at every iterate — but Adam's step is `−lr·D g` for a diagonal `D`, and `(D g)·v` is not zero, so plain Adam wanders inside the flat manifold. The loss does not notice; the parameters do. On a spring like the one above, observed with σ = 0.02 noise and fitted unguarded, the scale of `(k, c, m)` lands from −7.5% to −5.4% of the value it started at depending only on `lr`, with the loss unchanged in its first six digits, and Adam's iterate keeps drifting: +46% after 10,000 iterations at `lr=0.2` on a shorter record. Returning the lowest-loss iterate does not help here, because along a flat direction which iterate is lowest is decided by rounding, not by the data. `fit` therefore holds such directions at the values you supplied — the data has not contradicted them. A direction is held only if it passes two tests. First, none of the run's gradients pointed along it: `fit` accumulates them, and every gradient of a least-squares loss lies in the span the data can see. That is necessary but not sufficient, because a short or fast-converging run's gradients need not span everything the data does determine. So, second, the loss must have no curvature along it at the iterate `fit` returns, measured with Hessian-vector products there (`fit_lm` reads `JᵀJ` instead, with `fim`'s rank rule). The net displacement along the directions that pass both is removed. Both tests, the hold and the loss check below compare parameters with each other, so they are made with each identity-transform parameter measured relative to its own size (a `log` or `logit` coordinate is relative already) — the coordinates `fim(scale="relative")` uses — and the guard decides the same in any units. Made in the optimiser's own coordinates, a damping written in units of `1e-5` looked undetermined beside the others and was held at its start, raising `fit_lm`'s loss from `1.6e-11` to `13.9`. Then the loss gets the last word. The held point's loss is evaluated, and if it is above the selected iterate's by more than rounding — `2¹⁰·eps` relative (1.2e-4 in float32) plus what storing the held point in the working precision can cost — nothing is held, `res.hold_declined` is `True`, and a `RuntimeWarning` gives both losses. So the guard never trades loss for reproducibility, and a fit with no direction passing both tests gets its iterate back bit for bit. Earlier 0.4.0 development builds applied the first test alone: `fit_lm` on a well-posed four-parameter bowl reached a loss of 0.0 and came back at 0.22. The degeneracy has to be a fixed direction in the unconstrained coordinates, though. `SpringDamperNode` gives `damping` the identity transform so that zero damping stays representable, which makes the scale direction `(c, 1, 1)` and rotates it as `c` moves; then no direction is null for the whole run, and the guard holds nothing. `HeartPumpNode` does the same with its default specs: its data fix `R·C` and `SV/C` but not the scale of `(R, C, SV)`, and `stroke_volume` has the identity transform beside a `log` resistance and compliance. `excited_rank` then reads the full count -- it is the count minus the directions *the guard found* undetermined, and the guard found none -- on a fit the data do not determine, where `fim` at the returned point reads one direction fewer. `fit_lm`, which holds the Jacobian at the point it returns, says so in a `RuntimeWarning`; `fit` and `fit_multiple_shooting` hold no such matrix and do not. Declare `transform="log"` on every parameter a scale degeneracy mixes — the coordinates `fim(scale="relative")` already assumes — and it becomes constant (`damping` on the spring, `stroke_volume` on the heart pump). Then fit all three of `(k, c, m)`, and nothing else: ```python gm.set_param_spec("spring", "damping", ParamSpec(bounds=(0.0, None), transform="log")) gm.set_param_spec("spring", "mass", # trainable again ParamSpec(bounds=(0.0, None), transform="log")) mask = jax.tree.map(lambda _: False, gm.params) # see mask= below for key in ("stiffness", "damping", "mass"): mask["nodes"]["spring"][key] = True res = fit(gm, loss, params=start, mask=mask, n_iter=300, lr=0.1) res.excited_rank # 2 of 3: the data left one direction undetermined res.undetermined_drift # how far the raw iterate had drifted along it (~1e-2) res.hold_declined # False: holding it cost no loss, so it was held ``` Without the `damping` line, the same fit reports an `excited_rank` of 3 and holds nothing. `excited_rank is None` means the question was not answered, not that the answer was "full rank". That happens when: - `hold_undetermined=False`; - there are more than 512 trainable coordinates (array elements, not leaves); - the run took fewer iterations than there are trainable coordinates, so a direction no gradient has pointed along yet cannot be told from one that none ever will; or - every gradient was zero, a degenerate spectrum with nothing to measure a rank against: a fit started at an exact optimum, or a loss that reads none of the trainable parameters. The cutoff is numerical — a direction the data resolves *weakly* is kept, not held; for "well enough to use", read `crb` against a tolerance you declare. **`fit_lm` and `fit_multiple_shooting` run the same guard**, on the same `hold_undetermined` keyword, and fill the same three fields. Neither is immune for the reason it might look immune: the gradient is orthogonal to the null space, but no step rule here *is* the gradient. Levenberg–Marquardt solves `(A + λ·diag(A))⁻¹g`, which is orthogonal to `null(A)` only where `diag(A)` is isotropic there — take `A = [[1, 2], [2, 4]]` and `g = (1, 2)`, whose step is `∝ (2, 1)` for every `λ` against a null space spanned by `(2, −1)`. They differ in how badly. `fit_lm`'s step vanishes with the gradient, so its drift *converges*: on the spring above it settles 1.78% (noiseless) or 0.66% (σ = 0.02) from the scale you supplied and stays there, bit for bit, from iteration 10 to 200. `fit_multiple_shooting` is Adam, so it does not settle: 2.0% to 4.8% across `lr` 0.01–0.2, a 3.0% spread that the schedule picks and the data has no opinion about. Only the parameters are held there — the returned `window_states` are decision variables of that fit and come back from the selected iterate as the optimiser left them. All three fitters take a `mask=` that *narrows* `gm.trainable_mask()` — fit two of the three trainable constants, say. It cannot widen it: a mask naming a leaf whose spec says `trainable=False` is a `ValueError`, because `unconstrain` / `constrain` apply a leaf's transform and bounds only when its spec is trainable, so such a leaf would be stepped in physical coordinates with nothing clipping it. To fit a frozen parameter, make it trainable in its `ParamSpec` — that is what turns its bounds and transform on. (`fim`'s `mask=` is unrestricted: it linearises, it never steps a parameter.) ## Persistence and FMI * **Checkpoints** (`save_state` / `load_state`) store `gm.params` alongside the state, so a calibrated graph resumes calibrated (`python -m maddening.examples.advanced.checkpoint_resume_demo`). * **Graph serialisation** (`gm.to_dict()`, `maddening.serialization.config`, `save_graph_to_usd`) writes each node's *effective* params — the constructor arguments with the live `gm.params` values written over them — plus any `set_param_spec` overrides (`param_specs` in the dict, `maddening:paramSpecOverridesJson` on the USD node prim). Reloading gives a node constructed with the calibrated constants and the same trainable mask. A config and a stage are both **untrusted input**: each names the Python class of every node, so each is read against an explicit registry — `from_dict(config, node_registry)` and `load_graph_from_usd(stage, node_registry=...)`, which also accepts whatever `register_node_class` registered and the built-ins. Before 0.4.0 the USD reader imported the module a stage named, which ran that module; pass `allow_import=True` to get that back, and only for a stage you trust as much as a script. * **FMI**: `build_model_description` exposes every leaf of `gm.params` that the compiled step reads (see below for the rest) as a `causality="parameter"`, `variability="tunable"` variable named `.params.` (its own namespace, mirroring the pytree path, so it never collides with a `.` output), with `ParamSpec.description` / `units` as its metadata and `ParamSpec.bounds` as the XML `min` / `max` attributes (`include_parameters=False` to opt out). A sidecar built with `SidecarConfig(params=gm.params, param_specs=gm.param_specs(), step_fn=gm._compiled_step)` serves them through `get_params` / `set_params` (wire kinds `get_params` / `set_params`); a set value takes effect on the next step without recompiling, a value outside the declared bounds is rejected before anything is written (the same rule as `PUT /graph/params`), and `GetFMUState` / `SetFMUState` snapshots carry the parameters. The sidecar's Python API computes directional derivatives with respect to a parameter with `jax.jvp` (`FmuSidecar.get_directional_derivative`); the FMU binary does not provide them: the description does not declare `providesDirectionalDerivatives`, and the C wrapper's `fmi3GetDirectionalDerivative` returns `fmi3Error`. A leaf the step cannot read -- an `initial_*` condition, a value a node consumed at construction or declares in `static_data_deps`, one only an unconnected input would read -- is not exported (an FMU's graph is frozen, so it is a knob that does nothing) and is listed with the reason in `md.fixed_parameters`; pass that as `SidecarConfig(fixed_params=...)` (`FmuTcpBridge` applies it to its sidecar either way), and `set_params`, `set_fmu_state` and the bridge's `set_state` refuse a new value for one of them before anything is written. The sidecar holds only the compiled step, so a standalone sidecar built without `fixed_params` cannot tell. The description, the sidecar and the bridge each refuse a graph that has changed since its `compile()`: a structural `node.params` write is taken into the graph only as a pending recompile, which the compiled step the FMU runs would never see. Compile first. ## Mapping weights Edges with an interface `mapping` keep their weights under `gm.params["mappings"][".->."]`, with the same traced-input semantics as node constants. A second mapped edge on the same field pair (two additive contributions) gets its own slot, `"...#1"`, `"...#2"`, so the two never share weights. They are `trainable=False` by default (an interface operator is geometry, not a physical constant); a learned edge opts in with `gm.set_param_spec(edge.key, "H", ParamSpec())`. `H` is the one weight of the built-in dense mappings; a mapping of a [registered kind](../algorithm_guide/coupling/interface_mapping.md#registering-your-own-mapping-kind) names its own weights, each a floating-point array under an identifier, and what it may put here is checked when its edge is added. See the [interface mapping guide](../algorithm_guide/coupling/interface_mapping.md). A config (`to_dict`, USD) stores the mapping's *recipe* (`MappingSpec`: kind, hyper-parameters, point references) and rebuilds the weights on load; a checkpoint stores the weights themselves (`_params_mappings/`), and when both are loaded the checkpoint's — possibly trained — weights win. A [sparse mapping](../algorithm_guide/coupling/interface_mapping.md#sparse-mappings) keeps one weight `W` per slot of an index that is not a parameter; its checkpoint also records the pattern's digest, and a load refuses weights saved for another pattern. ### Calibrating a parameter that a mapped edge's grid derives from **The weights do not follow a calibrated geometry parameter, and nothing tells you (MADD-ANO-022).** A mapping built from a node's coordinates (the point reference `{"node": "rod", "field": "grid_x"}`, or the same array passed by hand) computes its weights once, from the grid as it was constructed. On the default, uniform `HeatNode`, `grid_x` is derived from `length`, and `length` is trainable. Calibrating `length` -- a fit, or any `params=` pytree you pass -- therefore moves the rod and leaves the mapped edge interpolating from the old grid. `compile()` accepts the graph, no warning is raised, and the reference's recorded hash still matches, because the static array itself never changed. *Writing* the new value into the running graph is refused (MADD-ANO-063): a `gm.params` write is a `ValueError` at the next run, `check_params`, `to_dict()` and `save_state()`; `PUT /graph/params` is a 400 naming the mapped edge and the field; `POST /checkpoint/load` refuses and undoes a checkpoint carrying one; and an exported FMU leaves the parameter out of its tunable set. The graph asks whether the node rebuilt with the value reads a referenced field differently, because the node would use the value while the mapping kept the old points, and the saved config would not load. Until 0.4.0's fix such a write was taken, the graph ran on the old weights, and `to_dict()` wrote a config whose mapping `from_dict()` refused to rebuild. So the result of a fit through the mapped edge cannot be written back; the fit itself is not refused. `python -m maddening.examples.coupling.interface_mapping_demo` shows the refusal at the next run and at `to_dict()`, the state left untouched, and the supported alternative: rebuilding the node and the mapped edge from the new points. What you would see, measured on an 8-cell rod calibrated from `length` 1.0 to 1.25 and mapped onto a 16-cell rod: - the target rod moves by about `4e-4`, where the same graph constructed at 1.25 moves it by about `1.2e-2`; - the gradient of the target with respect to `length` has the wrong sign; - `fit_lm` on the target's data stops at `length ≈ 0.22` with a small loss (about `1e-5`), so the fit looks successful. The truth is 1.25. The same fit on the source rod's own data recovers 1.25. Until a fix lands (being scoped for 0.5.0), use one of these: - **Freeze the parameter** when a mapped edge references a grid it derives: `gm.set_param_spec("rod", "length", ParamSpec(trainable=False))`. The default mask then leaves it alone, and `fit` refuses a mask that tries to widen back onto it. - **Give the mapping explicit coordinates**, as an `{"asset": "points.npy"}` or `{"inline": [...]}` reference. The config then states that the mapping's geometry is fixed, and does not imply that it follows the node. - **If the geometry has to be calibrated**, fit it from observations of the node itself rather than through the mapped edge. Then rebuild the graph at the fitted value, so the mapping is rebuilt from the new grid. The same applies to any node whose static data is derived from a trainable parameter while its step reads the parameter directly. ## What is not a parameter * Initial conditions (`initial_*`): they are state, not dynamics, and are `trainable=False` by default. * Structural constants (`n_cells`, {term}`stencil ` order, shapes, grid geometry): changing one changes the trace; they stay Python-side and require a recompile. * `static_data` (meshes, precomputed operators): constants baked into the compiled step by design. * A random seed: an `int` constructor parameter declared `ParamSpec(trainable=False)`, so it is rebuilt from a config and is no leaf a fit can move. See [A node that draws random numbers](../developer_guide/node_authoring.md#a-node-that-draws-random-numbers). * Anything in `state`: the adaptive error norm, history logging and {term}`coupling ` residuals would see it.