What’s new in v0.4.0#

Added in version v0.4.0.

Summary#

MADDENING 0.4.0 makes a graph’s constants data. Node constants are parameters that jax.grad reaches and that change without a recompile, and maddening.sysid fits them to recorded data. Interface mappings couple nodes whose grids do not match. A graph exports as a working FMI 3.0 FMU. Coupling groups now stop at a tolerance and report how far from the fixed point they may be.

The upgrade cost is the platform: 0.4.0 needs Python 3.12 or later and JAX 0.10 or 0.11. Neither range overlaps 0.3.1’s, so plan the JAX migration before you install. Read “What changes for existing code” and the “Upgrade checklist”.

The changelog entry is the short list. How the release was built and checked, with every audit round and every defect found and fixed before the release, is in the development record.

What is new#

Each item gives the stability label the code carries; “API stability” says what each label promises.

Parameters and system identification#

Node constants are parameters. gm.params holds every node’s constants as one nested dictionary of arrays. The compiled step takes it as an argument, so jax.grad reaches the constants and changing one does not recompile. The run methods take params=. All eleven shipped node classes and both sharded wrappers take part; gm.nodes_without_params() lists your own nodes that do not yet. STABLE: gm.params and the params= argument. Guide: Graph parameters. Example: python -m maddening.examples.advanced.calibrated_node makes a node of your own calibratable, checks it and fits it.
ParamSpec: what an optimiser may do with each parameter. Mark a parameter trainable or fixed, give it bounds and a transform that keeps it inside them. gm.set_param_spec(), gm.param_specs(), gm.trainable_mask(), gm.unconstrain(), gm.constrain() and gm.check_params() are the maps a fitter uses. EVOLVING. Guide: ParamSpec.
System identification: maddening.sysid. fit (gradient descent), fit_lm (Levenberg-Marquardt) and fit_multiple_shooting fit a graph’s parameters to recorded data; windowed_loss and observations_from_history build the objective. A fit returns the lowest-loss iterate it evaluated. EVOLVING. Guide: System identification. Example: python -m maddening.examples.advanced.multiple_shooting_fit fits a coupled graph by multiple shooting.
fim: can the data determine the parameters? sysid.fim returns the Fisher information, its rank, Cramér-Rao bounds and the least identifiable direction. The fitters hold the directions the data cannot determine at their starting values. When the rank was decided at the float32 noise floor, fim says so with a PrecisionLimitWarning. EVOLVING (fim, FIMReport, FitResult); fim_core and FIMCore, jittable and with no host round trip, are EXPERIMENTAL.
Parameters follow the graph. Configs, USD stages and checkpoints store the parameter values and the ParamSpec overrides. PUT /graph/params/{node} writes any parameter of a running graph without a recompile, after checking its type, shape, finiteness and bounds. An exported FMU exposes each parameter as an FMI parameter. Guide: Persistence and FMI.

Coupling between nodes#

Coupling groups stop at a tolerance and say how close they are. solver="ift" is the default: the loop exits when the group has converged, and converged=True means the estimated distance to the fixed point is within tolerance. gm.coupling_diagnostics() reports, for each group, an error estimate, a bound from the spectrum (spectral_error_bound, with rho_spectral) and a bound on the error of the IFT gradient (gradient_relative_error_bound). Each bound has a flag (spectral_usable, gradient_bound_usable) that says whether to rely on it. strict_convergence=True makes an unconverged step raise. EVOLVING (CouplingGroup); gm.print_coupling_report(), which prints the report, is EXPERIMENTAL. Guide: Reading coupling_diagnostics().
- The bounds need solver="ift" and diagnostics=True. A group whose pass depends on the previous one through more than eight independent interface scalars has no usable flag: the spectral estimate takes eight Krylov steps.
Why a flag is False: reason codes. Each report has reason_codes, a list of codes for each flag that is False (the constants of maddening.core.coupling.reason_codes), a not_usable_reason in words, and the float floor as residual_precision_floor. Branch on the codes, not on the sentence. EXPERIMENTAL. Example: python -m maddening.examples.coupling.acting_on_diagnostics_demo says what to change for each code.
Coupling groups survive save and load. to_dict / from_dict and USD stages carry each group with all of its options, so a reloaded graph solves the way the saved one did. EVOLVING (CouplingGroup).
Measured advice on choosing a solver and an acceleration. The coupling guide gives it from a sweep over eight graph fixtures; profile_graph takes n_stat_steps= to fix the window its coupling statistics come from. Guide: Coupling algorithm guide.

Interface mappings#

Mappings on edges. add_edge(..., mapping=) transfers a field between two nodes whose grids do not match, with rbf_mapping, nearest_neighbor_mapping, projection_1d_mapping or matrix_mapping. The weights live in gm.params["mappings"]: jax.grad reaches them, and they are fixed until you mark them trainable. A mapped edge is saved as a MappingSpec in configs, USD stages and checkpoints. EVOLVING. Guide: Interface mapping.
Sparse mappings. StaticSparseMapping and the kinds sparse_nearest_neighbor, sparse_projection_1d and sparse_matrix store only the non-zero weights, for interfaces too large for a dense matrix. A checkpoint refuses weights saved for another sparsity pattern. EXPERIMENTAL. Guide: Sparse mappings. Example: python -m maddening.examples.coupling.sparse_mapping_round_trip_demo saves and reloads a coupling group with one.
Mappings that follow moving geometry. add_edge(..., mapping=m, geometry=(anchor, field)) with the kind multilinear_grid interpolates between a uniform grid of one to three axes and points that are part of a node’s state, such as markers that move each step. A single-rate coupling group whose geometry mappings are all multilinear_grid reports its bounds under convergence_norm="l2" or "mixed", and checks its derivative along the positions against a finite difference. EXPERIMENTAL. Guide: Geometry-dependent mappings.
- Limits in 0.4.0: a group that solves positions (its pass reads one from the iterate) has both flags False on every step; the adaptive steppers and sharded ends are refused; under convergence_norm="interface" the report gives its float floor but no bounds, and that norm is refused for other geometry-dependent kinds and for sub-cycled groups; the geometry must be held by the edge’s source or target node. Float32 positions far from the grid’s first point put the report at its float floor: hold them in float64.
Mapping kinds of your own. @register_mapping adds a kind that is saved and rebuilt like the built-in ones. A file can only name a kind already registered in the process; nothing is imported because a file asks for it. EXPERIMENTAL. Guide: Registering your own mapping kind.
A worked example: moving a node’s own sampling onto an edge. Every snippet runs. Guide: Moving a sampling onto an edge.

FMU export#

A graph exports as a working FMU. build_fmu_binary and write_fmu produce an FMI 3.0 co-simulation .fmu: a small C wrapper that talks over a local TCP connection to a Python process running the graph. 0.3.x could write the model description but no loadable binary. EVOLVING (build_fmu_binary, write_fmu, FmuTcpBridge, SidecarConfig). Guide: Exporting a graph as an FMU.
- The bridge authenticates no caller (MADD-ANO-023): keep its default loopback bind, on a host whose other users you trust.
Binary frames and multi-clock export. Large arrays cross the FMU connection as raw bytes when both ends support it. build_model_description( multi_clock=True) declares one clock per node timestep, so an importer knows which outputs change on which ticks. EVOLVING. Guide: Wire protocol and Multi-rate graphs: clocks. Stages 5 and 6 of python -m maddening.examples.advanced.fmu_export_demo write a multi-clock description and run the FMUs in FMPy.

Adaptive nodes#

AdaptiveNode: a base class for nodes that solve on a changing subset of unknowns. Its gradient is exact while the active subset does not change, and gradient_capture_ratio says how much of the true gradient that covers. Read MADD-ANO-003 before optimising through one. EVOLVING. Guides: algorithm, developer.
WaveletAdaptiveNode: the first concrete adaptive node. It solves (-Δ + m) u = f in one to three dimensions in a wavelet basis and keeps only the coefficients that matter. EXPERIMENTAL. Guides: algorithm, developer.

Multi-GPU and sharding#

Per-neighbour halo exchange for unstructured meshes. ShardedUnstructuredNode(..., exchange="ppermute") sends each device only what its neighbours need, with results identical to the default. exchange_traffic(layout) reports what each method would move before you use a GPU. EVOLVING. Guide: Sharding topology. Example: python -m maddening.examples.advanced.unstructured_sharding_demo runs it, with a differentiable sharded_cg, on four emulated CPU devices.
Differentiable sharded linear solvers. sharded_cg and sharded_gmres take differentiable=True for exact gradients through the solve, with jacobi_preconditioner and block_jacobi_preconditioner. The solvers are STABLE; the preconditioners are EVOLVING. Differentiating a call without the flag raises (see “The sharded linear solvers”).
The sharded wrappers take part in parameters. A sharded node reads gm.params like the node it wraps, so a sharded graph can be calibrated. Grid-shaped boundary inputs are split across devices, and SimulationNode.domain_integral_axes() lets a node sum an integral over some mesh axes only. The wrappers are STABLE. Guide: Sharded static data.
The sharded wrappers, checked on four GPUs. On 2026-10-12 the multi-GPU checklist ran on four NVIDIA RTX PRO 6000 Blackwell cards (96 GB each, PCIe, no NVLink) with jax and jaxlib 0.11.2 and JAX_DEFAULT_MATMUL_PRECISION=highest, at commit c0c6add8. All six items closed, with 825 of 825 checks passed: sharded against unsharded results and gradients for both wrappers, halos at the shard and grid edges, a HybridNode over a sharded node, the refusal of an indivisible grid, and a coupling group with a sharded and a replicated member. A body-fitted tetrahedral mesh of a vessel, 1,001,664 cells, passed 14 of 14.
- Larger than one card. A sharded D3Q19 LBMNode of 472 million cells ran and stayed within 1.2e-6 of its reference, using about 200 GB of the 4x96 GB. That test built the state one device’s block at a time itself. The wrapper’s own initial_state() still builds the whole field on one device (MADD-ANO-262), so in 0.4.0 a run started that way is still bounded by one card.
- Memory. The sharded lattice step peaks at about 4.3 times its state, per device. Size a run on that.
- Exchange. all_to_all stays the default for ShardedUnstructuredNode. ppermute took 4 to 5% less time at 10⁵ and 3×10⁵ cells, all_to_all 17% less at 10⁶, and at 10⁷ cells ppermute took half the time. The two gave identical results at every size.
- Streaming. The WebRTC stream went from the four-GPU host to a laptop over the internet, through a TURN relay over TCP (an SSH tunnel), and was watched in a real browser (the shipped client page, set to use that relay). To try the stream on one machine, with no cloud and no GPU: python -m maddening.examples.cloud.streaming.09_local_webrtc_viewer (it needs GStreamer).
The files and how they were taken: benchmarks/results/multigpu/2026-10-12_rtxpro6000/. The full account is in the development record.

The HTTP API and network transports#

The HTTP API asks for a token unless it is bound to loopback. See “Security”. A new /healthz route reports the version. EVOLVING (APIAuth). Guide: Serving the REST API on a network. Example: python -m maddening.examples.servers.serve_with_token runs the token flow on a loopback port.
A REST write that fails leaves the graph as it was. Every route that can change the graph either completes or changes nothing, and simultaneous requests no longer step the graph at the same time. Request numbers are strict, and counts are held to their declared range.
ZeroMQ transports are private by default. Every socket binds to loopback unless told otherwise, and a socket bound to a reachable address is encrypted and authenticated (CURVE). EVOLVING (TransportAuth). Guide: ZeroMQ transports.
The surrogate-training routes and the state streams are labelled. /surrogate/train, /surrogate/status, /surrogate/activate, /surrogate/deactivate, the streams /ws/state, /ws/state/binary and /ws/render, and StateRelay are EXPERIMENTAL in 0.4.0, to be hardened in 0.5.0. /openapi.json gives each of these routes its label as x-maddening-stability.

Inspecting, profiling and caching#

Read-only inspection of a graph. gm.print_graph(), to_mermaid(), to_dot() and tables of state, parameters, memory and coupling show what a graph contains without running it. gm.print_graph_diagram() draws it in the terminal (the terminal extra). EXPERIMENTAL. Guide: Inspection.
A version report for bug reports. python -m maddening info (or maddening.show_versions()) prints the MADDENING, Python and JAX versions, the devices and the installed extras, and never prints a secret. EXPERIMENTAL.
A compilation cache that persists between runs, and a profiler that measures coupling cost. compile_cache.enable(dir) (or MADDENING_COMPILATION_CACHE_DIR) and warm_cache() avoid recompiling a graph in each new process. profile_graph measures the coupling overhead of each group. EVOLVING. Guide: Profiling. Example: python -m maddening.examples.advanced.compile_cache_demo runs a cold and a warm process against one cache directory and prints both compile times.
gm.reset_state() and gm.get_node(). Reset a graph to its initial state without recompiling, and look a node up by name. STABLE, as members of GraphManager.

For node authors#

maddening.testing: check a node you wrote. verify_node runs a battery of checks (outputs, determinism, compiled against uncompiled, gradients, parameters). maddening.testing.mms measures a node’s order of accuracy by manufactured solutions or by grid refinement. verify_mapping checks an interface mapping, and verify_graph_order measures a coupled graph’s order. EXPERIMENTAL. Guide: Verification.
SimulationNode.static_data_deps(). A node declares which parameters its precomputed data depends on, so that data is rebuilt when one of them changes; compile() refuses a static derived from a trainable parameter. STABLE.
A node that draws random numbers. The node authoring guide gives the convention (keep the key’s raw data in the state) and what holds through each entry point. Guide: Node authoring.

Packaging and documentation#

Type information ships with the package. MADDENING includes py.typed, so your type checker reads its annotations. Guide: Typing.
New examples. Examples of profiling, system identification, checkpoint and resume, sharding on four emulated CPU devices, FMU export, interface mapping, REST parameters and convergence diagnostics, each asserting what it prints; the items above link the examples for their feature. The remote-simulation examples run locally with --local, with no cloud account.
A software bill of materials. docs/validation/sbom/ holds CycloneDX SBOMs for the base install and the server, surrogates and usd extras, regenerated from the release commit.
Anomaly registries of your own project validate. python -m maddening.compliance check-anomalies accepts a registry with exactly one <project>_version field (--version-field NAME names it), and ResolutionStatus has PARTIALLY_RESOLVED.

What changes for existing code#

Only differences from 0.3.1 are listed. A fix to a defect that 0.3.1 had, where the fix also changes what your code sees, is here with its registry number and not under “Fixed”.

Dependencies#

Requirement	0.3.1	0.4.0
Python	`>=3.10`	`>=3.12`
`jax` / `jaxlib`	`>=0.4,<0.6`	`>=0.10,<0.13`
`lineax`	the `[ift]` extra	base, `>=0.0.7`
`scipy`	not declared	base, `>=1.14`

The JAX ranges do not overlap: move your project to JAX 0.10 or 0.11, and to Python 3.12, before you install 0.4.0. There is no JAX 0.12, so the range has two reachable minors, and CI tests both.
JAX 0.11 needs Python 3.12. A 3.11 floor would have locked out half of the declared range.
lineax is in the base install because the default coupling solver’s gradient needs it. The [ift] extra still resolves and installs nothing. scipy was already installed by JAX.
The [verify] extra installs only hypothesis; the external formal-verification suite and its dependency are removed.
The terminal extra adds termaid>=0.9,<1, for gm.print_graph_diagram().

Behaviour changes#

Change	What to do
Coupling groups default to `solver="ift"`. The loop exits when the group has converged instead of after a fixed number of passes.	Nothing, unless you relied on a fixed iteration count. `solver="fori"` still works, warns, and goes in the next minor release.
`converged=True` means “within tolerance of the fixed point” (`MADD-ANO-005`, partly resolved; `MADD-ANO-114`). The threshold applies to an error estimate, `residual / (1 - rho)`, and `tolerance` and `rtol` are relative to each field’s magnitude. `atol` is a dead band, defaults to `0.0`, and `compile()` warns for every group that sets it above zero.	Expect more passes and a different `residual`. If a group now reaches `max_iterations`, raise the cap or loosen the tolerance on purpose. Leave `atol` at `0.0` (`MADD-ANO-254`); under `-W error` the new warning raises. Recorded Aitken and IQN trajectories move.
`run_sweep` and `run_scan` may differ by about one ulp. Constants are no longer compiled in, so XLA may fuse a product differently in two compiled shapes.	Loosen any test that asserts bitwise equality between the two to a few ulps.
`HeatNode` applies its boundary temperatures at the rod ends, refuses an unstable timestep, and reports the flux at the end (`MADD-ANO-007`, `008`, `009`, `062`). The scheme is second order; `T[0]` no longer equals `left_temperature`; with no boundary data the end cells evolve under a zero-flux closure. `compute_boundary_fluxes` changed by up to 10% at `n_cells=10`.	Read the boundary value you supplied, not the end cell. Keep `timestep * thermal_diffusivity / dx**2` at or below 1/2 (`stencil_order=2`) or 5/16 (`stencil_order=4`). Re-calibrate a coupled interface fitted to the old flux.
*`HeatNode.boundary_flux_spec()` declares `Km/s`, not `W/m^2`.** Metadata only; the node reports `-alpha dT/dx`.	If you multiplied by `rho*c_p` to convert, stop.
`gm.timestep` is the time one `step()` advances (`MADD-ANO-096`). On a graph with a sub-cycling coupling group it is the group’s largest member timestep, larger than before.	Compute a step count as `duration / gm.timestep` again. The live runner’s clock, USD `baseDt` and the FMU’s default step follow.
A multi-rate graph must keep every clock (`MADD-ANO-227`, `046`). Timesteps with no usable common step no longer compile, and a sub-cycled node’s timestep must divide its group’s largest.	Choose timesteps that divide each other.
On a multi-rate graph a slow node’s flux is the flux of the state it holds (`MADD-ANO-235`).	Results of such graphs change; nothing to do.
A step that changes the shape of the state raises `ValueError` (`MADD-ANO-220`). It used to be stored.	Keep each state leaf’s shape and dtype from step to step.
An edge and a declared external input on one field no longer compile (`MADD-ANO-265`). The input replaced what the edge delivered on every step, with zeros when it was not fed.	Keep one: leave out the `add_external_input`, or remove the edge.
`add_external_input` refuses a second declaration of a field that differs from the first. `from_dict` and `load_graph_from_usd` refuse a file that holds two.	Declare each external input once.
`add_node` refuses a timestep that is not a finite number above zero, the names `_meta`, `_params` and `_params_mappings`, and a name with a control character (`MADD-ANO-181`, `182`, `208`).	Give every node a finite positive timestep and another name.
`remove_node` takes the node out of its coupling group (`MADD-ANO-214`). The group is removed when one member would remain.	Nothing.
`copy.copy(gm)` raises `TypeError`. The shallow copy shared the graph’s nodes and parameters but not its compiled step.	Use `copy.deepcopy(gm)`.
`to_dict()` returns a copy (`MADD-ANO-205`). Editing the config no longer changes the graph.	Nothing, unless you relied on the shared containers.
`to_dict()` warns of a constructor argument it does not save (`MADD-ANO-277`, partly resolved).	Pass every constructor argument to `super().__init__` and read it from `self.params`.
`add_coupling_group` refuses out-of-range options (`MADD-ANO-145`): `waveform_iterations <= 0`, a boolean option that is not a boolean, and `rtol=0` under the norms that divide by it.	Pass values in range.
`compile()` refuses a group-internal flux edge the coupling loop reads from the state (`MADD-ANO-060`, `071`), where 0.3.1 failed inside the step with a bare `KeyError`.	Carry the quantity in a state field, or change the group’s options as the message says.
What a group returns under `convergence_norm="interface"` (`MADD-ANO-240`, `241`). Every member is computed from readings that met the criterion.	Results of such groups change; nothing to do.
`load_graph_from_usd` builds only built-in and registered node classes (`MADD-ANO-075`).	Pass `node_registry={"MyNode": MyNode}`, call `register_node_class`, or pass `allow_import=True` for a stage you trust.
`POST /graph/nodes` refuses at the endpoint what 0.3.x accepted. Five input classes go 201 → 400: a name containing `/`, `#` or `->`; an empty name; a name spelling `NaN`, `Infinity` or `-Infinity`; a `HeatNode` timestep above its stencil’s Fourier limit; and a non-finite constructor constant. A repeat POST of any of them goes 409 → 400. An array dimension over the new cap goes 201 → 422.	Read 400 as “refused, see `detail`”.
REST request bodies are strict (`MADD-ANO-228`, `253`). `true`, `"0.5"` or `" 0.25 "` given for a number is refused with 422, and `POST /graph/edges` and `POST /graph/nodes` refuse a key they do not apply.	Send numbers as JSON numbers and only the documented keys.
The sharded wrappers refuse settings that gave wrong answers (`MADD-ANO-025`, `030`, `037`, `039`, `042`, `057`, `064`). `ShardedStencilNode` takes only `boundary="periodic"` around an `LBMNode`, refuses `"zero"` and `"periodic"` around a `HeatNode`, and refuses a per-axis `boundary` dictionary, an empty `axis_map`, and a `heat_source` or `body_force` of a shape the unsharded node refuses. `ShardedUnstructuredNode` refuses a Cartesian stencil node (one with a halo), a per-cell state that is not one row per layout cell, and a per-cell input the length of one shard’s slab.	Wrap an `LBMNode` with `boundary="periodic"`. Give a sharded `HeatNode` cold ends by feeding its `left_temperature` and `right_temperature` boundary inputs (`gm.add_external_input(<node>, "left_temperature")`, then a value of 0.0), as unsharded. Pass one boundary mode as a string.
`LBMPipeNode` refuses a propeller outside its grid, and the LBM constructors a non-finite viscosity or relaxation time (`MADD-ANO-058`). The default `propeller_x=10` lies outside a pipe with `nx <= 10`, and a `propeller_radius` above 1 is refused.	Pass an in-grid `propeller_x` for a short pipe.
`LBMNode` and `LBMPipeNode` compute their lattice products at `Precision.HIGHEST` (`MADD-ANO-290`). On an NVIDIA GPU at JAX’s default matmul precision, 0.3.1 ran `velocity @ e.T` and `f @ e` at reduced precision: 3.4e-4 relative per product, and the LBM gradients of the multi-GPU stencil check off by 1.6e-3 relative. On CPU nothing changes, to the bit.	Nothing; GPU results change by up to that much. On 0.3.1, set `JAX_DEFAULT_MATMUL_PRECISION=highest`, and keep it for your own float32 matrix products.
Graphs trace their float32 matrix products at full precision (`MADD-ANO-291`). The compiled step, every scan, sweep and adaptive program, the coupling group’s implicit-function gradient, the sysid fits and the Fisher information run under `jax.default_matmul_precision("highest")`; 0.3.1 took JAX’s default, which on an NVIDIA GPU runs a float32 matrix-matrix product at reduced precision (2.5e-4 relative for an interface mapping on a 3-component field, against 6.5e-7). Affected in 0.3.1: interface mappings on multi-component fields, IQN-ILS and the FNO surrogate. On CPU nothing changes, to the bit.	Nothing; GPU results in those paths move by up to about 1e-4 relative. For tensor-core speed in an MLP or FNO surrogate node, set `gm.matmul_precision = None` (experimental) to use JAX’s own setting.
`LBMNode.meta.algorithm_id` is `MADD-NODE-011`. `LBMNode` and `RigidBodyNode` shared `MADD-NODE-007`; `RigidBodyNode` keeps it.	Move any key on the old value.
`FMIVariable` is keyword-only. 0.4.0 inserted `node` and `field` before `shape`.	Build it with keywords.
`build_model_description` refuses what it used to round or skip (`MADD-ANO-201`, `219`): a `selected_outputs` pair that names no state field, and a `default_step_size` that is not a whole number of graph steps.	Pass exportable pairs and a step that divides into graph steps.
`validate_anomaly_registry()` requires and enum-checks `resolution_status`, rejects unknown keys and resolves `affected_components`.	Add the field to every entry of your own registry, or pass `resolve_references=False` for a schema-only check.

The sharded linear solvers#

sharded_cg and sharded_gmres changed in four ways that a caller can see.

The default call answers where lineax’s CG breaks down (MADD-ANO-264, partly resolved). In float32, lineax’s CG returns NaN above about 1e5 unknowns, and for a smooth right-hand side from about a thousand. The default backend="auto" now solves the same system again with the loop backend when lineax’s value is not finite. 0.3.0 and 0.3.1 returned the NaN. backend="lineax", asked for by name, still does.
converged and residual_norm describe the value returned (MADD-ANO-263, 269, 281). On every route, converged is ||b - A x|| <= max(atol, rtol * ||b||) on the returned x, with no allowance for rounding. A float32 solve that asked for a tolerance it cannot reach now answers converged=False with the true residual. At the default rtol=1e-6 that is a condition number above about 17. If your code branches on converged, ask for an rtol a few times above 1e-7 times the condition number in float32, solve in float64, or read residual_norm.
The default atol is relative (MADD-ANO-123). It was an absolute 1e-8, which returned a wrong answer for a right-hand side near 1e-9. A concrete negative or NaN rtol or atol raises ValueError (MADD-ANO-271).
Differentiating a solve needs differentiable=True (MADD-ANO-266). jax.grad, jax.jvp, jax.jacfwd and jax.jacrev through a call without it raise ValueError. 0.3.x differentiated the iteration, which was wrong when the tangent was not of b’s size. If the solve is a constant, put jax.lax.stop_gradient on its inputs.

Deprecated#

calibrate and tune_coupling_params: use sysid.fit. Removed in 0.5.0.
solver="fori" on a coupling group: removed in the next minor release.
maddening.core.simulation.checkpoint.download_and_load_state: import maddening.cloud.download_and_load_state. If you pass a presigned URL, also pass manifest_url= (or set RESUME_MANIFEST_URL).

Removed#

The external formal-verification suite, its CI job and its dependency. No public API was removed.

Names that only 0.4.0 development builds had (blindness_ratio, is_trapped_at, the bound_valid and gradient_error_bound keys, positional FitResult and FIMReport) are not changes from 0.3.1. The old spellings still warn, for code written against a development build; the development record has the details.

Security#

These changes close the network exposure that 0.3.1 had. Each is a fix to a released version and a change you may have to act on.

The HTTP API asks for a token unless it is bound to loopback (MADD-ANO-051). 0.3.1 served every route to any caller. A server reachable from another machine requires Authorization: Bearer <token>: set MADDENING_API_TOKEN, or read the token the server logs once at start-up. Pass SimulationServer(bind_host=...) (or set MADDENING_HOST) so the server knows which rule applies. Browsers use the maddening.bearer.* WebSocket subprotocol, and the bundled pages do it for you.
The API checks the request’s origin (MADD-ANO-076). A web page in your browser can no longer drive a loopback-bound server. A loopback-bound server also asks the token of a peer that is not a loopback address and of a forwarded request: an in-process TestClient(app) presents server.auth.token, or is built with base_url="http://127.0.0.1" and client=("127.0.0.1", 50000). Serve it with uvicorn’s --no-proxy-headers.
Checkpoint paths are confined to one directory (MADD-ANO-052). Set SimulationServer(checkpoint_root=); paths in requests are relative to it.
A cloud launch opens no API port by default. JobConfig.ports defaults to []. Reach the API over ssh -L 8000:127.0.0.1:8000, or set ports=[8000] and MADDENING_API_TOKEN.
ZeroMQ sockets bind to loopback, and encrypt off it (MADD-ANO-015). NetworkRelay, CommandPublisher and Coordinator bind to 127.0.0.1 by default, and a remote subscriber stops receiving until you give an address. A non-loopback bind demands CURVE and refuses to start without the secret. Set MADDENING_TRANSPORT_TOKEN on both ends, to a value different from MADDENING_API_TOKEN.
The WebRTC signaling server checks the client’s token (MADD-ANO-053). It used to check its own, so it admitted every client.
A USD stage cannot run code. load_graph_from_usd no longer imports the module a stage names; see “Behaviour changes”.
The FMU bridge no longer unpickles (MADD-ANO-055). FMU-state blobs are npz: regenerate any stored blob.

The REST guide, src/maddening/api/README.md, says who the server is for (a trusted client on loopback, or a token holder) and what it does not defend against.

Fixed#

Defects that a released version had and that 0.4.0 fixes. A fix that also changes what your code sees is under “What changes for existing code” or “Security” instead. Defects introduced and fixed during the 0.4.0 cycle are not listed; the development record has them. The registry, docs/validation/known_anomalies.yaml, has each entry with its reproducer and the test that pins the fix.

Parameters and system identification#

Entry	The defect, as the registry titles it	Released versions that had it
`MADD-ANO-105`	A node.params write with no recompile reached whichever program was traced next: gm.step kept its cached trace while run_scan at a new length, and later the sysid losses, ran the written value	0.1.0 to 0.3.1
`MADD-ANO-278`	An assignment to node.params on a sharded wrapper or a HybridNode replaced the wrapper’s mapping alone: the step ran the old value while node.params read the new one back	0.1.0 to 0.3.1

Coupling between nodes#

Entry	The defect, as the registry titles it	Released versions that had it
`MADD-ANO-026`	With waveform_iterations > 1, coupling_diagnostics() reported only the last sweep’s pass count: an earlier sweep stopped at max_iterations read as iterations=1	0.1.0 to 0.3.1
`MADD-ANO-045`	solver=”fori” with acceleration=”iqn-imvj” carried zero secant columns to the next step, so jacobian_reuse did nothing	0.1.0 to 0.3.1
`MADD-ANO-059`	Accelerating a coupling group that holds an integer, boolean or PRNG-key state leaf failed at the first step	0.1.0 to 0.3.1
`MADD-ANO-113`	The IFT adjoint (and in 0.4.0’s development the tangent) of a coupled step was exactly zero, reported successful, whenever its right-hand side was below about 1e-8	0.3.0 and 0.3.1
`MADD-ANO-116`	solver=”fori” with Aitken or IQN acceleration on a bfloat16 or float16 coupling group raised a carry TypeError at trace time	0.1.0 to 0.3.1
`MADD-ANO-144`	An outside node added between two members of a coupling group that is part of a larger feedback loop read the group one step late	0.1.0 to 0.3.1
`MADD-ANO-158`	A typed PRNG key in a coupling group member fails to trace under solver=’ift’ with acceleration=’aitken’ or ‘fixed’	0.1.0 to 0.3.1
`MADD-ANO-159`	A coupling group whose members are joined only through an outside node, on no cycle, reads that node one step late with no warning	0.1.0 to 0.3.1
`MADD-ANO-161`	jax.grad through a 16-bit coupling group under solver=”ift” raises NotImplementedError	0.3.0 and 0.3.1
`MADD-ANO-186`	A bfloat16 or float16 coupling group’s norms and floor were computed in its own dtype: residual 0.0 above 65,504 active entries, inf on a finite state at the default rtol	0.1.0 to 0.3.1
`MADD-ANO-193`	The conservation diagnostic and the surrogate dataset generator rebuilt boundary inputs by a rule of their own, not the step’s	0.1.0 to 0.3.1
`MADD-ANO-233`	A coupling group under IQN whose iterate left float range before its cap never returned from step()	0.3.0 and 0.3.1
`MADD-ANO-283`	ift_linear_solve with solver=”cg” or “gmres” returned a value that was not the solution, with no error, where the floor of lineax’s stopping test was zero, underflowed, or was negative or NaN	0.3.1
—	`validate()`’s line for a loop that a coupling group iterates ended `(Gauss-Seidel)` for a `"jacobi"` group too; it now ends `(Jacobi)` there.	0.1.0 to 0.3.1

Interface mappings#

Entry	The defect, as the registry titles it	Released versions that had it
`MADD-ANO-192`	Interface-mapping factories returned an operator for coordinates their formula does not cover	0.1.0 to 0.3.1

Time stepping and multi-rate graphs#

Entry	The defect, as the registry titles it	Released versions that had it
`MADD-ANO-043`	run_adaptive / run_adaptive_scan advanced a sub-cycled node by its rate divider times dt on every adaptive step	0.1.0 to 0.3.1
`MADD-ANO-044`	On a multi-rate graph a coupling group’s diagnostics, predictor history and IQN-IMVJ warm start came from solves the step discarded	0.1.0 to 0.3.1
`MADD-ANO-061`	run_adaptive advanced its clock by dt_min when it accepted an attempt at dt_min, while the state advanced by the attempted step	0.1.0 to 0.3.1
`MADD-ANO-074`	The adaptive steppers’ error norm read integer and boolean state leaves: a bool raised, a uint32 wrapped and a counter moved the step sequence	0.1.0 to 0.3.1
`MADD-ANO-079`	compile() restarted a multi-rate graph’s sub-step counter, so a recompile in the middle of a run re-phased the sub-steps each slower node fired on	0.1.0 to 0.3.1
`MADD-ANO-095`	After run_adaptive or run_adaptive_scan the coupling report described only the second kept half step: a first half step stopped at max_iterations was hidden, and converged read True where strict_convergence refused the step	0.1.0 to 0.3.1
`MADD-ANO-118`	run_adaptive accepted a rejected attempt larger than dt_min whenever shrinking it would reach dt_min, where run_adaptive_scan retried at dt_min	0.1.0 to 0.3.1
`MADD-ANO-125`	The multi-rate base timestep’s GCD stopped on an absolute 1e-9: nodes at 1e-9 and 2e-9 got a base step of 2e-9 and their clocks drifted apart	0.1.0 to 0.3.1
`MADD-ANO-160`	jax.grad through run_adaptive_scan was 0.0 or NaN where an attempt’s step-doubling estimates agreed exactly, as for an accelerated coupling group whose iterate starts at its fixed point	0.1.0 to 0.3.1
`MADD-ANO-189`	The adaptive steppers’ error norm was computed in a bfloat16 or float16 state’s own dtype: run_adaptive did not return on a float16 state above 65,504 elements, run_adaptive_scan refused any 16-bit state	0.1.0 to 0.3.1

Graph construction and stepping#

Entry	The defect, as the registry titles it	Released versions that had it
`MADD-ANO-099`	GraphManager.step did not zero-fill the inputs a partial external_inputs left out, and dropped an undeclared node or field name in silence: the node took its own “input missing” branch	0.1.0 to 0.3.1
`MADD-ANO-120`	A node downstream of a cycle that was added to the graph before the cycle’s nodes was scheduled ahead of them and read their previous-step output: an edge on no cycle, staggered silently	0.1.0 to 0.3.1
`MADD-ANO-250`	A state field written with another shape after compile() was broadcast into the node that reads it: the shape compile() holds an edge to was asked once	0.1.0 to 0.3.1

Nodes#

Entry	The defect, as the registry titles it	Released versions that had it
`MADD-ANO-001`	LBM GPU segfault on CUDA 12.2 + jaxlib 0.5.1	0.1.0 to 0.3.1
`MADD-ANO-020`	LBMNode’s Zou-He pressure boundary imposed rho_p + S_K instead of the prescribed density	0.1.0 to 0.3.1
`MADD-ANO-034`	LBMNode’s wall_mask was dropped by every save/reload: the reloaded graph ran with no walls	0.1.0 to 0.3.1

Multi-GPU and sharding#

Entry	The defect, as the registry titles it	Released versions that had it
`MADD-ANO-004`	The sharded wrappers ignored a REST parameter write; PUT /graph/params answered 200 without changing the physics	0.1.0 to 0.3.1
`MADD-ANO-028`	A sharded stencil node on a mesh axis of one device ran with periodic global halos whatever `boundary` said	0.2.0 to 0.3.1
`MADD-ANO-029`	The default `"edge"` halo fill copied a halo two or more cells wide from the shard’s first cells instead of repeating its edge cell	0.2.0 to 0.3.1
`MADD-ANO-033`	ShardedStencilNode stepped with dt rounded to float32 under jax_enable_x64	0.2.0 to 0.3.1
`MADD-ANO-038`	ShardedStencilNode edge-filled a sharded static’s global halos under a periodic wrapper	0.2.1 to 0.3.1
`MADD-ANO-040`	The sharded wrappers placed a domain integral carried in the state like a grid field: an integral-emitting node could not run in a graph	0.2.1 to 0.3.1
`MADD-ANO-041`	halo_exchange ignored a per-axis boundary key naming no exchanged mesh axis, and that axis took the edge fill	0.2.0 to 0.3.1
`MADD-ANO-056`	ShardedStencilNode read the block size it hands update_padded in shard_info off a domain integral carried in the state, so a sharded HeatNode carrying a vector or per-shard energy integral never closed its right rod end	0.2.1 to 0.3.1
`MADD-ANO-065`	ShardedUnstructuredNode.gather_global reshaped a domain integral listed in state_fields() as a per-cell field: a scalar raised, and values one per shard came back reordered	0.3.0 and 0.3.1
`MADD-ANO-067`	ShardedStencilNode summed a domain integral over a mesh axis its axis_map leaves unused, counting every block once per device along it	0.2.1 to 0.3.1
`MADD-ANO-124`	sharded_gmres’s loop backend returned NaN with converged=False whenever restart exceeded the system size, the default 50 against any system of fewer than 50 unknowns	0.3.0 and 0.3.1
`MADD-ANO-155`	sharded_cg returned zeros with converged=True for a right-hand side with a NaN or infinite entry, and ift_linear_solve returned zeros with no error	0.3.0 and 0.3.1
`MADD-ANO-269`	sharded_cg’s default call and backend=’lineax’ answered an operator that returns NaN or infinity with zeros, converged=True and iters=0	0.3.0 and 0.3.1
`MADD-ANO-272`	At the top of the dtype’s range sharded_cg and sharded_gmres vouch for a value that is not the solution: zeros from the CG loop until 0.4.0 for a right-hand side whose norm overflows, and in 0.4.0 infinities on every route where the solution itself overflows	0.3.0 and 0.3.1
`MADD-ANO-281`	With a floor of zero for lineax’s entrywise test (an explicit atol=0.0, or rtol=0) sharded_cg and sharded_gmres read converged=True on their lineax routes after a step or a cycle, at a residual of 0.4 \|\|b\|\|	0.3.0 and 0.3.1

Saving and loading#

Entry	The defect, as the registry titles it	Released versions that had it
`MADD-ANO-006`	Non-finite numbers are written as non-standard JSON tokens	0.1.0 to 0.3.1
`MADD-ANO-097`	USD baseDt and the FMU default step read a _base_dt attribute that nothing set: USDWriter wrote 0.01 for every graph, and save_graph_to_usd gave the smallest node timestep rather than the graph’s step	0.1.0 to 0.3.1
`MADD-ANO-138`	load_state cast a checkpoint value to the graph’s dtype with no check: a float64 1e39 loaded into a float32 field as inf, and -1e-50 as -0.0	0.1.0 to 0.3.1
`MADD-ANO-251`	A node’s delta_t written after add_node was neither followed nor reported: the graph kept its timestep and to_dict() wrote the new one	0.1.0 to 0.3.1

FMU export#

Entry	The defect, as the registry titles it	Released versions that had it
`MADD-ANO-078`	build_model_description of a real graph declared no FMU inputs and a default step of 1e-3, whatever the graph’s inputs and timestep	0.3.0 and 0.3.1
`MADD-ANO-273`	ModelDescription.to_xml wrote the unit attribute on integer and Boolean variables: a modelDescription.xml that FMI 3.0’s schema refuses, with nothing said at export	0.3.0 and 0.3.1

REST API and streams#

Entry	The defect, as the registry titles it	Released versions that had it
`MADD-ANO-024`	PUT /graph/params accepted a value a node consumes at construction: reported and saved, never used by the running graph	0.1.0 to 0.3.1
`MADD-ANO-049`	A non-finite value written through PUT /graph/params was stored, then the reply failed with a 500, and every later GET /graph/params for the node failed too	0.1.0 to 0.3.1
`MADD-ANO-087`	PUT /graph/params answered 200 to a value the running node could not step with, and every POST /sim/step after it was a 500	0.1.0 to 0.3.1
`MADD-ANO-088`	POST /sim/stop answered “stopped” while the runner’s thread kept stepping, and a reset after it was overwritten	0.1.0 to 0.3.1
`MADD-ANO-090`	PUT /sim/stride echoed values it did not apply, and DELETE /graph/edges answered 200 for an edge the graph did not have	0.1.0 to 0.3.1
`MADD-ANO-091`	A state or render WebSocket stream never noticed a client that left after the simulation stopped, so the server could not shut down; an ordinary disconnect was logged as an error	0.1.0 to 0.3.1
`MADD-ANO-106`	Concurrent REST requests stepped the graph at once and lost steps silently: 200 concurrent POST /sim/step took about 60, every one answered 200, and the streams counted all 200	0.1.0 to 0.3.1
`MADD-ANO-107`	POST /surrogate/train reset the live simulation to its initial state, and its reply did not say so	0.1.0 to 0.3.1
`MADD-ANO-108`	After POST /checkpoint/load the state streams served the state from before the load, at the old step’s time, until the next step	0.1.0 to 0.3.1
`MADD-ANO-109`	/ws/state/binary kept the frame layout of a node that had been replaced over REST: values cut off, or phantom zeros, in frames of the advertised length	0.1.0 to 0.3.1
`MADD-ANO-110`	A runner whose thread had died was reported running: /sim/pause and /sim/resume answered 200, /sim/start 409 “already started”, and /sim/reset was_running true	0.1.0 to 0.3.1
`MADD-ANO-111`	The real-time runner burst through the time it had been paused: about 1.5 s simulated in the 0.4 s after a 1 s pause of a real-time run	0.1.0 to 0.3.1
`MADD-ANO-112`	/ws/state encoded each frame on the event loop, once per client, for any number of clients: a large state stalled every other request	0.1.0 to 0.3.1
`MADD-ANO-132`	POST /sim/profile reset the live simulation and left it at the profiler’s last step, and the streams counted the profiler’s steps	0.2.0 to 0.3.1
`MADD-ANO-133`	A reset, a state write or a node removed over REST was not shown by the streams: they served the state from before it until the next step	0.1.0 to 0.3.1
`MADD-ANO-139`	PUT /sim/stride reset a value left out of the query to 1: a call naming only steps_per_frame reset the relay’s stride, and the reverse	0.1.0 to 0.3.1
`MADD-ANO-164`	A parameter write to a HybridNode was answered and then lost: the hybrid held a copy of its physics node’s params, so PUT /graph/params echoed the new value and the step kept the old one	0.1.0 to 0.3.1
`MADD-ANO-166`	POST /sim/profile answered 500 for a graph that could not compile with any error but a RuntimeError	0.2.0 to 0.3.1
`MADD-ANO-180`	PUT /graph/state wrote every field as float32 whatever its dtype, and parsed text and booleans as numbers	0.1.0 to 0.3.1
`MADD-ANO-183`	POST /sim/start on a graph with no nodes answered ‘started’ and its runner thread died on its first frame	0.1.0 to 0.3.1
`MADD-ANO-184`	A request refused by its model for a NaN or infinite value answered 500	0.1.0 to 0.3.1
`MADD-ANO-191`	Apps built at once in several threads left the process’s warnings filters changed: a silenced warning was raised, or every UserWarning was ignored from then on	0.1.0 to 0.3.1
`MADD-ANO-209`	A refusal that echoed a surrogate from the request answered 500	0.1.0 to 0.3.1

Network transports and cloud#

Entry	The defect, as the registry titles it	Released versions that had it
`MADD-ANO-286`	WebRTC streaming never negotiated: SelkiesSession sent a viewer no answer and no ICE, so no video ever reached a viewer	0.1.0 to 0.3.1
`MADD-ANO-287`	The cloud container streamed nothing: its entry point never started the WebRTC session, and its API could add no node to build a graph	0.1.0 to 0.3.1

Surrogates#

Entry	The defect, as the registry titles it	Released versions that had it
`MADD-ANO-077`	Swapping a surrogate in with replace_node, or out with POST /surrogate/deactivate, re-added each edge without its additive flag and units: an additive coupling became an overwriting one, silently	0.1.0 to 0.3.1

Tooling and compliance#

Entry	The defect, as the registry titles it	Released versions that had it
`MADD-ANO-117`	profile_graph’s per-node timing called each node with no boundary inputs, so a node that reads an input raised KeyError	0.1.0 to 0.3.1
`MADD-ANO-130`	The anomaly-registry validator raised TypeError on a list-valued severity, safety_relevance or resolution_status instead of reporting the entry	0.1.0 to 0.3.1

Security fixes#

Fixes to released versions that bound what a request can cost or keep a secret out of a log. The changes to network exposure are under “Security”.

Entry	The defect, as the registry titles it	Released versions that had it
`MADD-ANO-086`	A REST request could make the server allocate without bound: before 0.4.0 POST /graph/nodes built any size, and 0.4.0’s size checks ran only after the node was built	0.1.0 to 0.3.1
`MADD-ANO-092`	POST /sim/run and POST /surrogate/train took any size: one request could hold a worker indefinitely or name more memory than the host has	0.1.0 to 0.3.1
`MADD-ANO-134`	A JAX trace started over REST had no bound: left running, it grew the server’s memory by kilobytes per step until stopped	0.2.0 to 0.3.1
`MADD-ANO-185`	A checkpoint load decompressed every archive member before it checked any name or shape: a small archive cost as much memory as its members declared	0.1.0 to 0.3.1
`MADD-ANO-274`	A 422 sent the refused value back whole: a request body the server refused was encoded and returned at its full size	0.1.0 to 0.3.1
`MADD-ANO-288`	The cloud entry point logged MADDENING_CLOUD_CONFIG verbatim at INFO, so a TURN username and credential in stream_config.ice_servers were written to the container’s log	0.1.0 to 0.3.1

Known anomalies#

These are the registry entries a user can reach in 0.4.0: those still open and those resolved in part. Each says what to do. The registry has the full description of each, and the development record indexes every entry.

Parameters and system identification#

MADD-ANO-129: the deprecated calibrate and tune_coupling_params default to absolute thresholds, so a loss already below 1e-6 is converged before a single step. Use sysid.fit, or pass tolerance and accuracy_threshold in the loss’s own units.
MADD-ANO-217: fit_lm started from a stiffness far too high can stop short, unconverged. Read converged; restart from the returned parameters, or start within a factor of 30 of a plausible value.
MADD-ANO-047 (partly resolved): a write that crosses a branch a node fixed at construction, such as an LBMPipeNode’s G crossing zero, is refused over REST but not in gm.params or a fit. Bound such a parameter to one sign with a ParamSpec, or rebuild the node.
MADD-ANO-048 (partly resolved): a value written into gm.params or moved by a fit can lie outside what the node’s constructor accepts, so the saved graph does not reload. Bound the parameter’s ParamSpec inside the constructor’s range, and check that a saved config reloads.

Coupling between nodes#

MADD-ANO-005 (partly resolved): converged=True rests on an estimated contraction rate. Read ratio_usable; where it is False, treat the verdict as a residual test, and read spectral_error_bound with diagnostics=True.
MADD-ANO-027: with sub-cycling, waveform_iterations > 1 re-solves the same fixed point. Use waveform_iterations=1, and raise max_iterations instead of adding sweeps.
MADD-ANO-170: the Aitken two-pass guard can add two passes, where the guide says at most one. Budget from the reported iterations.
MADD-ANO-275: IQN acceleration under the Jacobi schedule can stall on a group the plain Jacobi loop solves. Read converged; use Gauss-Seidel with IQN, or Jacobi with "none" or "aitken".
MADD-ANO-285: under Aitken and IQN acceleration a run depends on the units of one field. Write the group’s quantities in comparable units, or use "none" or "fixed".
MADD-ANO-254: with atol > 0, a change that must cross a field inside the dead band is not seen, and a group can report converged=True far from its fixed point. Leave atol at 0.0.
MADD-ANO-212, MADD-ANO-247: where a member reads a difference of entries of one field (a gradient or jump operator), the float floor is too small and spectral_error_bound can read under the true distance with its flag set. Run such a group under Jacobi with the "l2" or "mixed" norm, or have the producer hold the difference as a field of its own. A multilinear_grid gather that samples a field near its zero needs float64 positions.
MADD-ANO-257, MADD-ANO-260: a long mapped row, or a sum of many terms written in an edge transform, is not counted in the float floor. Hold the group in a wider dtype, or declare the operator as a static sparse mapping.
MADD-ANO-258: under the interface norm, a multilinear_grid mapping read at its source and anchored at its target counts no position in the floor. Hold the positions in float64, or anchor the scatter at its source.
MADD-ANO-259: a narrowing transform on a flux edge is not counted in the float floor. Carry the value as a state field and put the transform there.
MADD-ANO-230, MADD-ANO-239: in float32, a field below a hundredth of what it drives, or two multiplied fields held in units far apart, can make rho_spectral read wrong with its flag set. Run the group in float64, or rescale the fields.
MADD-ANO-232: a gradient_relative_error_bound whose flag is False can read NaN on a GPU. Read the flag before the value.
MADD-ANO-246: the geometry self-check can withhold a float32 Gauss-Seidel report when a position sits within about 2e-5 of a spacing of a lattice plane. Read the next step’s report, or run under Jacobi or in float64.

Interface mappings#

MADD-ANO-022: a mapped edge whose points come from a grid derived from a trainable parameter keeps the constructor’s geometry when that parameter is calibrated. Mark the parameter ParamSpec(trainable=False), or fit it from the node’s own data and rebuild the graph.

Time stepping and graph construction#

MADD-ANO-014 (partly resolved): an explicit integrator converges at first order when a time-varying boundary input is given once per step. Carry time as a state field with derivative 1 and read the forcing from it; the recipe is in the explicit integrators guide.
MADD-ANO-017 (partly resolved): under jax_enable_x64 a scan refuses a float32 carry on a freshly compiled graph. Promote the state to float64 after compile(), or leave jax_enable_x64 off.
MADD-ANO-128: the adaptive steppers’ default atol=1e-6 and dt_min=1e-8 are absolute. Pass atol in the state’s own units and a smaller dt_min.
MADD-ANO-173: run_adaptive never returns once its error norm is NaN. Use run_adaptive_scan with max_steps, or check the state in a callback.
MADD-ANO-190: run_adaptive_scan raises on a float32 or 16-bit state under jax_enable_x64. Use run_adaptive, or hold the state in float64.
MADD-ANO-156, MADD-ANO-157: a flux edge that crosses a coupling group’s boundary, or that an ungrouped cycle reads from the previous step, fails with a bare KeyError. Carry the quantity in a state field, or put the producer in the group.
MADD-ANO-172: reset_state() inside a differentiated loss can raise UnexpectedTracerError. Reset outside the differentiated function.
MADD-ANO-277 (partly resolved): a constructor argument a node keeps outside params is not saved; to_dict() now warns. Pass it to super().__init__ and read it from self.params.

Nodes#

MADD-ANO-002: HeatNode checks its stability limit at construction only. Apply the same limit to any dt you pass to update(), and bound a calibrated thermal_diffusivity or length with a ParamSpec.
MADD-ANO-031: a HeatNode at stencil_order=4 with no boundary input is not insulated. Use stencil_order=2, or give the end a temperature.
MADD-ANO-050: two HeatNode rods coupled end to end are unstable above Fourier number 3/8 (stencil_order=2) or 0.226 (stencil_order=4). Stay below those on both rods; compile() warns for the pairs it recognises.
MADD-ANO-098: two SpringDamperNodes anchored on each other in a coupling group gain energy when stiffness*dt > damping. Keep damping >= stiffness*dt; compile() warns for the pairs it recognises.
MADD-ANO-021: a node that branches on its own state returns the gradient of the branch it took; BallNode’s gradient through a bounce omits the contact time. Smooth the switch, or fit over intervals with no switch.
MADD-ANO-003: AdaptiveNode’s gradient omits a first-order term where the active set switches. Use the largest active-set budget you can afford, and add hysteresis to the selection.
MADD-ANO-011: BallNode declares forward Euler and implements semi-implicit Euler. Treat it as semi-implicit.
MADD-ANO-012: HeartPumpNode samples its inflow at the end of the step. Keep dt at or below 1e-3 s.
MADD-ANO-013: several nodes hold a float64 quantity at float32. Run them in float32, or expect a float32 floor in float64 studies.

Multi-GPU and sharding#

MADD-ANO-262: ShardedStencilNode builds the wrapped node’s whole state on one device, so a grid that fits the mesh but not one card cannot be started. Build the state one device’s block at a time, as benchmarks/multigpu/run_capacity.py does.
MADD-ANO-068 (partly resolved): XLA miscompiles a ShardedStencilNode step inside a loop in one layout of a sharded static; the loop entry points refuse it. Build the mesh from the sharded axes only, or advance with step().
MADD-ANO-032 (partly resolved): after writing a constant of a wrapped node, a direct call to the wrapper’s update() uses what it built from the old value. Call invalidate_static_cache() first; the graph path needs nothing.
MADD-ANO-035: ShardedUnstructuredNode reads a global-order state as partition layout on a partition not in global order. Renumber the cells, or convert the state with partition_value.
MADD-ANO-264 (partly resolved): backend="lineax" by name and ift_linear_solve(solver="cg") still return NaN where lineax’s CG breaks down. Use the default backend, backend="loop" or float64; for ift_linear_solve, solver="gmres".
MADD-ANO-270: with differentiable=True, converged says nothing of the tangent and cotangent solves. Give max_iters what a solve from zero needs.
MADD-ANO-280: sharded_gmres on its default route counts max_iters in restart cycles. Pass backend="loop" to bound the work.
MADD-ANO-282: with differentiable=True, a solution near the dtype’s largest number is reported as not converged. Rescale the system, or solve in float64.
MADD-ANO-294: a node whose static_data builds new arrays at each access makes a sharded wrapper place them on the devices again at every eager update(). The results are the same; the step was about 50 times slower at 10⁷ cells on four GPUs. Return the same objects at every access, or step the node in a compiled graph.

Saving, loading and the HTTP API#

MADD-ANO-036 (partly resolved): a config does not record a node’s sharding; from_dict warns and rebuilds the node unsharded. Wrap it again with replace_node before compile(). A USD stage of a sharded graph cannot be loaded: save it with to_dict().
MADD-ANO-171: save_state cannot write a typed PRNG key leaf. Keep the key’s raw data (jax.random.key_data) in the state.
MADD-ANO-206: a bfloat16 leaf cannot be restored from a checkpoint. Hold such a state in float16 or float32.
MADD-ANO-010: a string spelling a non-finite token ("NaN", "Infinity") is refused by the JSON serialisers. Spell it differently.
MADD-ANO-023: the FMU TCP bridge authenticates no caller. Keep its loopback bind, on a single-user host or in its own network namespace.
MADD-ANO-194: DatasetGenerator pairs a forward edge’s boundary input with the source’s state before the step. Shift the inputs one sample yourself.

Cloud#

MADD-ANO-016 (partly resolved): the SkyPilot calls of CloudSession have never been run against a live provider. Use maddening.cloud.launcher.CloudLauncher, and confirm teardown with sky status.
MADD-ANO-292: SelkiesSession.update_framebuffer_gpu drops every frame, with a warning, unless cupy is installed, and no extra or image installs it. Install cupy, or read the frame to the host in your renderer and push it with update_framebuffer_cpu.
MADD-ANO-293: update_framebuffer_gpu ignores stride_bytes, so a frame with padded rows raises ValueError on every frame. Pass unpadded rows, or push host frames with update_framebuffer_cpu.

API stability#

Label	What it promises	Changed by
`STABLE`	The signature and the documented behaviour hold for the life of the major version.	A major release, after two minor releases of `DeprecationWarning`.
`EVOLVING`	The signature and wire format are settled; additions are expected.	A minor release, after one minor release of `DeprecationWarning`.
`EXPERIMENTAL`	Opt-in. May change or disappear in any minor release.	Any minor release, with a changelog entry.
`DEPRECATED`	Scheduled for removal.	The release its warning names.

gm.params and the params= argument of the run methods are STABLE. The methods that take, return or are defined by a ParamSpec are EVOLVING, as ParamSpec is: GraphManager.param_specs, set_param_spec, param_spec_overrides, trainable_mask, constrain, unconstrain, check_params, and SimulationNode.param_specs.
Nothing was promoted to STABLE in this release. Everything 0.4.0 added is new, and lowering a label again is itself a breaking change.
A STABLE signature cannot change unannounced: scripts/check_stable_signatures.py compares the tree with docs/developer_guide/stable_api.json in CI.
Not labelled yet: SimulationServer (but for its experimental routes), CloudSession, CloudConfig, maddening.viz (but for StateRelay) and the concrete surrogate architectures.
The stability report, generated from the tree, lists every surface with its label. The deprecation policy says how a label is withdrawn.

Upgrade checklist#

Upgrading from 0.3.1 walks through what a 0.3.1 graph runs into on 0.4.0, in the order you meet it, with each refusal and its fix.

Move to Python 3.12 and JAX 0.10 or 0.11, then pin maddening>=0.4.0,<0.5.
Run your test suite. Loosen any bitwise run_sweep against run_scan assertion to a few ulps.
If a coupling group now reaches max_iterations, raise the cap or loosen its tolerance on purpose. Leave atol at 0.0.
If you subclass SimulationNode and want its constants calibratable, declare update(..., *, params=None), read them from params, and declare param_specs(). gm.nodes_without_params() lists the nodes that still bake constants. A flux producer gives compute_boundary_fluxes the same keyword-only params.
Replace maddening.core.simulation.checkpoint.download_and_load_state with maddening.cloud.download_and_load_state, and pass manifest_url= for a presigned URL.
If a graph declares an external input on a field an edge also delivers to, keep one of the two.
If you differentiate through sharded_cg or sharded_gmres, pass differentiable=True. If you branch on converged, read “The sharded linear solvers”.
If you run the FMU bridge, regenerate any stored FMU-state blob: it is npz now.
If you serve the REST API, set SimulationServer(checkpoint_root=). Off loopback, set MADDENING_API_TOKEN, pass bind_host= and send Authorization: Bearer <token>. In-process, give TestClient the token or a loopback client.
If you bind a ZeroMQ transport off loopback, set MADDENING_TRANSPORT_TOKEN on both ends, to a value different from MADDENING_API_TOKEN.
If you launch cloud jobs, add ports=[8000] only if you want the API port open; otherwise tunnel to it.
If you build an LBMPipeNode with nx <= 10, pass an in-grid propeller_x.
If you load USD stages naming your own node classes, pass node_registry=.
On an NVIDIA GPU, a graph now traces its float32 matrix products at "highest" (MADD-ANO-291; gm.matmul_precision), and the LBM nodes pin their own (MADD-ANO-290). For float32 matrix products of your own outside a graph (a surrogate’s training loop, code around the graph), set JAX_DEFAULT_MATMUL_PRECISION=highest: JAX computes them at reduced precision by default there. See Installation.

What is not in v0.4.0#

Sharding a stencil node buys speed, not capacity beyond one card (MADD-ANO-262). ShardedStencilNode builds the wrapped node’s whole state on one device, so a grid larger than one card’s memory cannot be started through it. 0.5.0 addresses this early.
Sharding checked beyond four PCIe cards in one machine. The 2026-10-12 session ran four GPUs in one host, joined by PCIe without NVLink. More devices, NVLink and several hosts have not been run.
A GPU video encoder. The WebRTC stream encodes in software (x264enc) on every host.
Bounds for a geometry group under the interface norm. rho_spectral, spectral_error_bound and the gradient bound of such a group are planned for 0.5.0; run it under "l2" or "mixed" to read them.
A stability label for SimulationServer (beyond its experimental routes), CloudSession, CloudConfig, maddening.viz (beyond StateRelay) and the concrete surrogate architectures.
The fluid-dynamics side of the wavelet work: the lid-driven cavity, the biharmonic operator and variable coefficients. These are 0.5.0 numerics.
A ZeroMQ transport for the FMU wire protocol. The payloads are transport-neutral; only TCP is wired up.