Repository navigation
Conversation
|
|
![github-advanced-security[bot]](https://avatars.githubusercontent.com/in/57789?s=60&v=4){kind=small}
Performance Comparison
|
🏗️ Architecture Diff
No architecture changes detected. ✅ Legend: ⚪ No change · 🔵 Minor (attrs/inits) · 🟡 Moderate (nodes added/removed) · 🔴 Major (interface changed) |
There was a problem hiding this comment.
🔵 Needs a closer look
Unresolved correctness and serialization issues accompany failed decoder numerical gates and pending corrected-package qualification.
3 open findings
What changed in this PR
Adds pinned Nemotron 3.5 Bolt Omni support to Mobius’s ONNX export pipeline. Full-model numerical qualification remains incomplete.
Changes:
- Adds heterogeneous Puzzle decoding, RADIOv4 vision, and image/video prompt processing.
- Preserves native mixed-format weights through transactional loading and prepared-feature GenAI metadata.
- Adds precision corrections, regression tests, and qualification documentation.
| File | Description |
|---|---|
| tests/model_coverage_test.py | Documents standalone Puzzle coverage exception. |
| tests/_test_configs.py | Adds tiny Puzzle and Omni configurations. |
| testdata/cases/vision-language/nemotron-3.5-bolt-74b.yaml | Declares pinned qualification case. |
| src/mobius/tasks/_nemotron_h_omni.py | Builds three-component Omni package. |
| src/mobius/tasks/_cache_utils.py | Preserves configured SSM state dtype. |
| src/mobius/tasks/__init__.py | Registers Omni task. |
| src/mobius/models/nemotron3_diarization.py | Reuses parameter-realization helper. |
| src/mobius/models/nemotron_h.py | Adds Puzzle topology and precision handling. |
| src/mobius/models/nemotron_h_puzzle_test.py | Tests MoE numerical precision. |
| src/mobius/models/nemotron_h_omni.py | Composes native Omni components. |
| src/mobius/models/nemotron_h_omni_test.py | Tests component validation. |
| src/mobius/models/kimi_k3.py | Preserves FP32 residual-scoring weights. |
| src/mobius/models/kimi_k3_test.py | Checks reduced-precision arithmetic consistency. |
| src/mobius/models/_nemotron_h_omni_vision.py | Implements projection and video compaction. |
| src/mobius/models/__init__.py | Exports new model classes. |
| src/mobius/integrations/transformers/_nemotron_h_omni_weights.py | Defines native streaming weight bindings. |
| src/mobius/integrations/transformers/_nemotron_h_omni_weights_test.py | Tests storage fidelity and atomic failure. |
| src/mobius/integrations/transformers/_nemotron_h_omni_processing.py | Validates media and compacts prompts. |
| src/mobius/integrations/transformers/_builder.py | Pins sources and selects native loading. |
| src/mobius/integrations/transformers/_builder_test.py | Tests pinning and unsupported-mode rejection. |
| src/mobius/integrations/ort_genai/genai_config.py | Adds Omni runtime metadata. |
| src/mobius/integrations/ort_genai/auto_export.py | Maps hybrid caches and pinned assets. |
| src/mobius/integrations/ort_genai/_nemotron_h_omni_test.py | Tests prepared-feature metadata. |
| src/mobius/integrations/gguf/_runtime_evidence_test.py | Separates current topology from historical counts. |
| src/mobius/integrations/_weight_loading.py | Adds transactional whole-package streaming. |
| src/mobius/components/_rms_norm.py | Adds FP32 gated normalization. |
| src/mobius/components/_radio_v4_vision.py | Implements RADIOv4 backbone. |
| src/mobius/components/_radio_v4_vision_test.py | Tests backbone parity and shapes. |
| src/mobius/components/_quantized_linear.py | Adds native packed projection operators. |
| src/mobius/components/_quantized_linear_test.py | Updates activation-dtype validation. |
| src/mobius/components/_native_packed_linear_test.py | Tests native operator contracts and rounding. |
| src/mobius/components/_multimodal.py | Adds flat media-feature mixing. |
| src/mobius/components/_mamba_block.py | Adds dtype annotations and masking hook. |
| src/mobius/components/_float32_linear.py | Preserves FP32 accumulation through bias. |
| src/mobius/components/_float32_linear_test.py | Tests dense rounding boundaries. |
| src/mobius/components/_float32_gated_rms_norm_test.py | Tests grouped normalization precision. |
| src/mobius/components/_flat_input_mixer_test.py | Tests feature accounting and absent media. |
| src/mobius/components/_common.py | Shares parameter realization and propagates dtype. |
| src/mobius/components/__init__.py | Exports reusable components. |
| src/mobius/_testing/ort_inference.py | Corrects scalar, empty, and strided transport. |
| src/mobius/_testing/ort_inference_test.py | Tests tensor transport round trips. |
| src/mobius/_registry.py | Registers pinned Omni and Puzzle architectures. |
| src/mobius/_configs/per_model/_nemotron_h_puzzle.py | Resolves per-layer topology and state settings. |
| src/mobius/_configs/per_model/_nemotron_h_puzzle_test.py | Tests Puzzle extraction and validation. |
| src/mobius/_configs/per_model/_nemotron_h_omni.py | Parses nested architecture and storage contracts. |
| src/mobius/_configs/per_model/_nemotron_h_omni_test.py | Tests Omni configuration contracts. |
| src/mobius/_configs/_base.py | Adds per-layer configuration hook. |
| src/mobius/_configs/__init__.py | Exports new configuration classes. |
| docs/nemotron-h-omni.md | Documents export and qualification limits. |
| docs/nemotron-h-omni-processing.md | Documents media preparation and decoding handoff. |
🧠 Review effort: Balanced
💡 Configure MCP servers for context-aware, tailored reviews. Learn more in the docs.
|
Fixed the L1 smoke failure in 2c9857a. The component-quantization path leaves a LayerNormalization activation dtype unresolved during graph construction; Float32Linear incorrectly treated that as an unsupported dtype. Known unsupported dtypes still raise; graph-wide inference now resolves unknown activations, retaining FP32 MatMul+bias and CastLike. Added six F32/F16/BF16 inference regressions. Full unchanged L1 suite: 1,745 passed, 52 existing skips; dense/vision CPU+CUDA selection: 176 passed, zero skips. No smoke-test exclusions or tolerance changes. Fresh remote CI is starting. |
Full reviewRecommendation: request changes before merge. Reviewed head: Five reviewers covered implementation, readability, failure modes, numerical/reference contracts, and integration. The author updated the branch during review, so a second targeted five-reviewer pass inspected the complete delta from Confirmed findingsAll paths below are relative to F1 — Major: cached decoding rejects a valid full attention maskLocation:
Current-head tiny repro: The Minimal fix: retain exact-shape validation for prefill compaction. For cached no-media calls, accept a rank-two mask with matching batch and sufficient past-plus-current length, and return it unchanged. Add the full-mask regression, including ordinary generated token 18. F2 — Major: direct configuration defaults change recurrence and cache semanticsLocation: Direct Both families reproduced this difference on current-head tiny BF16 graphs:
The ports were Minimal fix: override these dataclass defaults on Puzzle so Omni inherits them, preserve explicit source overrides, and test direct BF16 construction. This finding does not claim that source-reader exports use the wrong defaults. F3 — Major: native lazy loading bypasses the existing bounded-streaming save policyLocation: The plan reports Actual tiny package saves on the reviewed head showed:
Package-local Minimal fix: set F4 — Minor: the default revision pin misses early CLI probesLocation: The first CLI Hub/Diffusers probe happens before the builder applies the default Bolt revision. A current-head mocked CLI probe receives A mutable default-branch pipeline index could change routing before the pin, but a present live misroute was not established. The documented explicit-revision command is unaffected. Minimal fix: share checkpoint source/revision resolution and apply it before the first Hub probe, rather than duplicating more model-specific pin branches across entry points. F5 — Minor: new FP16 CUDA tests assume an unsupported source/oracle equivalenceLocation: All three biased FLOAT16 CUDA cases, ranks 1/2/3, fail at: torch.testing.assert_close(source, reference, rtol=0, atol=0)Observed on the supplied A100/PyTorch backend: 2/9 mismatched elements, maximum absolute difference An earlier-head diagnostic explicitly recorded that source assertion failure and continued the unchanged rank-2 ORT-versus-oracle array assertions; those completed. The current-head batch still reports the three original failures separately. No ORT output defect was established by these failures. Minimal fix: separate the component's defined FP32-through-bias arithmetic oracle from native PyTorch backend qualification. Keep cancellation sentinels and exact arithmetic checks; do not globally loosen tolerances or hide the source-backend limitation. Reuse, scalability, and Lightning relationshipThe whole-package loader usefully reuses shared plans, source accounting, and staged binding. The new One nonblocking consolidation opportunity is the new public This PR does not require Lightning #761 to merge first. The reviewed heads are parallel branches from the same base, neither containing the other. They reuse the preexisting Nemotron-H foundation rather than depending on #761's MTP/BF16-reconstruction modules. Shared-file overlap still requires integration and regression testing after either merges. The separate runtime deliverables are microsoft/onnxruntime#33400 for native operators and microsoft/onnxruntime-genai#2741 for prepared-feature/hybrid-cache generation. Their existence is not a claim of stock released-runtime support. Findings ledger and dissent
The second-round delta review established no additional material correctness findings. It did not clear F1-F5. Readability suggestions about hook and precision-contract documentation remain nonblocking. ValidationThe final-head offline batch produced:
Imports were verified against the reviewed worktree. No checkpoint downloads or dependency installations were performed. The continuation diagnostic is not counted as a passing substitute for a failed pytest case. Attributed open questions and exclusions
Not checked: fresh full-checkpoint export, native IQ2 execution, companion runtime execution, full-model parity/generation, performance, 24 GB residency, exhaustive repository tests, or full lint/type qualification. Historical v5/v6 failures and later corrected variants are distinct evidence. This review neither asserts that the latest arithmetic necessarily retains identical historical errors nor clears its fresh full-package qualification gates. Threat-model coverage was bounded to external configuration/tensors/media provenance, lazy-source lifetime, and output policy; this was not a broad security audit. |
|
Full numerical qualification: FAIL. This candidate is not shipping-ready. The source-chunk candidate now has a complete eight-step comparison: all 1,496 required arrays are finite and aligned, but 281 fail the unchanged normalized-RMSE <=0.03 / cosine >=0.999 gates. A separate lead audit recomputed every metric/gate from the preserved FP64 sums and confirmed the verdict.
Exact expected/actual tokens: The first prefill common-output threshold crossing is convolution state at global layer27 (NRMSE about0.033628); the first failed prefill hidden output is The diagnostic-header issue is resolved in a separately versioned local validator: the exporter intentionally removes non-interface symbolic aliases, leaving legitimate partially anonymous axes. The validator now distinguishes raw anonymous Actual CUDA execution/provider/library capture passed, with no unexpected CPU floating-point computation. The original qualified29 runtime was used; the newer allocator55db runtime was not substituted. All28 original package files remained unchanged during this completed qualification run; its only package addition was the sibling diagnostic header Export/source provenance is unchanged: Mobius Receipt caveat remains: the unchanged exporter used Standalone1e-5 component failures remain open independently. A separate identical-prepared-input microcase localized its first exposed difference to five one-ULP BF16 weighted-B values; cumsum FP32, CB FP32 and intrachunk BF16 coefficients were bitwise equal. Source pre-cast FP32 arithmetic is still unobserved, so this is not a proven cast/exp cause or a demonstrated cause of the full-model failure. No thresholds, golds, precision, backend, masks or cache semantics were changed. Sealed comparison SHA256: No full-model readiness, public-Python GenAI validation, 24GB deployment, speculative acceptance or model-performance claim is made. |
CUDA numerical investigation update (not model acceptance)The original full-decoder result is unchanged: 1,215/1,496 numerical comparisons pass; 281 fail at the fixed NRMSE <= 0.03 / cosine >= 0.999 gates. All comparisons are finite; the eight greedy tokens match, but that does not establish decoder parity. The first same-input discrepancy is already visible in the layer-0 NVFP4 input projection, with M=27, N=18,560, K=4,096. On the same checkpoint bytes and BF16 input:
The original 85 differences are 80 one-step plus five five-step BF16 differences, not uniformly one-ULP differences. Disabling MMA is not a remedy and was not adopted. The last row uses the existing runtime, not a new operator or precision-policy patch: typed views of the original packed FLOAT4 and FLOAT8 local-scale bytes -> FLOAT casts and local multiplication -> FLOAT-operand For this one projection, all 501,120 final BF16 values match the frozen source bitwise. All 27 observed FP32 post-global products for output row 6239 and all 4,096 local-weight FLOAT bits in that row, including signed zero, also match. Actual CUDA profiling/logging records This narrows the role of operand/backend computation semantics. It does not prove a defect in the existing A16 operator, whose summation order is unspecified, or that this projection explains every full-model failure. It is not native packed-A16 qualification, official SM120 qualification, a performance result, or proof of a 24 GB deployment footprint. No new native precision API has been implemented. Reproduction pins: HF revision Full-model readiness remains unqualified. Media, speculative integration and deployment-memory claims remain held. |
Expanded-FP32 follow-up: single-projection equality is not a decoder remedyFollowing the single-projection result, the diagnostic replaced all 80 NVFP4 projections with the same explicitly expanded FLOAT-operand expression, using unchanged checkpoint payloads, baseline runtime and frozen reference. Two decoder forwards completed: prefill and the first cached step. The harness then stopped because its adapter requested The comparison is worse on the same 374 entries, not a comparison of partial counts against the full 1,496:
All 374 comparisons are finite and aligned; thresholds remain NRMSE <= 0.03 and cosine >= 0.999. The parent independently checked the actual arrays, pinned source NPZ/JSON checksums, declared native-dtype value hashes, state-layout alignment, FP64 metric sums and gates. Prefill profiling records 80 CUDA FLOAT Gemms and no substitution back to the native A16 operator. BLAS API/kernel symbols were not observed in this full-decoder run. The input embeddings and Only the generated prefix Partial-run artifact-manifest SHA256: |
Routing-boundary investigation (diagnostic replays, not full-model acceptance)All 40 per-layer source/native routing contracts agree: actual heterogeneous expert counts/top-k values, dimensions, scoring, selection bias, grouping, routing normalization/scaling and dtypes. No global-default or per-layer topology mismatch was found. The source-conditioned component controls also pass the original float gates:
The source controls reproduce the frozen source endpoints bitwise. The norm's internal source FP32 registers remain unobserved. The first MoE observer preserves endpoints and typed operators, but its complete native hardware tail audit is incomplete; it is not globally qualified. A separate, smaller routing-only sweep executed the pinned source norm/router callables and unchanged native routing prefixes on recorded prefill hidden states. Layer-1 routing fixtures reproduce the earlier captures bitwise, all executed native floating operators are on CUDA, and this sweep's CUPTI capture completed without overflow. These results are explicitly REPLAYED_ROUTING_ON_RECORDED_HIDDEN, not routes captured during a new complete decoder execution:
The sole identical-input mismatch is layer 34/token 3: source selects expert 372 where native selects 264. The normalization output already differs in one BF16 element, and the Kth/K+1 selection-score gap is On the original native hidden trajectory, the earliest replayed set change is layer 3/token 6 (source expert 211 versus native expert 210). On identical source inputs, layer 3 instead matches all 27 token sets. This distinguishes a conditional change following upstream hidden drift from evidence of a broken local TopK kernel. Selected sets and expert-associated weights were compared, not unsorted positional indices. The parent independently verified all 1,080 replayed rows, TopK validity, exchanged IDs, boundary gaps and the layer-34 normalization difference from retained arrays. These controls show sensitivity at discrete routing boundaries; they do not establish the cause of every full-decoder failure or justify a precision workaround. An additional static audit ruled out a missing IQ2/Q8 local-weight cast: both round the complete decoded weight to the activation type before FP32 multiplication. The historical correction The original full-model result remains 1,215 PASS / 281 FAIL of 1,496 under unchanged thresholds. No production arithmetic, reference, acceptance threshold or routing indices were changed. Media, speculative integration, deployment-memory/performance and shipping readiness remain unqualified. Routing report SHA256: Bounded BF16 cuBLAS projection diagnosticOne session/one forward compared two source-conditioned projections, each through its original packed operator and a standard BF16 CUDA Gemm. Complete weights were decoded with the unchanged pinned CPU helpers, cast to BF16 and stored inline in a separate diagnostic header. This is a dequantized BF16 single-projection diagnostic, not native packed execution or a 24 GB deployment result.
The IQ2 result is a new M1 source-input control, not the earlier masked M27 native expert branch. Packed Q8 NRMSE is All four typed compute nodes ran on CUDA. Actual Gemm calls were Report SHA256: Native packed BF16 cuBLAS candidate: first-MoE controlA local, unpublished ORT candidate ( The candidate passes 29 focused CUDA regressions without skips and matches the two frozen-source projection controls bitwise. In one source-conditioned forward of the unchanged full first-MoE graph, its residual endpoint now matches all 110,592 BF16 values bitwise; the original native endpoint differs at 1,782 values. The parent independently verified the artifact manifest, original input, raw endpoint arrays and 2,661 CUDA node-profile events. The bounded callback evidence records 484 BF16 cuBLAS calls with compute32F/default math. This is the actual masked-M27, 240-expert component, not the M1 projection proxy. Candidate CUDA provider SHA256: Scope remains component-only. No separate observer-invariance claim is made for this component control. One fresh full 1,496-array/eight-token candidate qualification has been authorized, including a new 776-output ordinary/instrumented bitwise-invariance check, unchanged numerical gates and exact generated tokens. It has no numerical result yet. The original full-decoder result remains 1,215 PASS / 281 FAIL; current-package media, GenAI pairing, speculation, performance and deployment-memory readiness are not established by this result. Full candidate qualification: FAIL, despite exact greedy tokensThe fresh full run of the local
All 776 ordinary/instrumented common outputs pass the fresh typed-byte invariance check. All eight greedy tokens exactly match the source: The candidate stays local/unpublished and is not selected as a qualified shipping fix. No thresholds, source values, model graph arithmetic or reference definitions were changed. Causal localization now uses saved prefill/cached arrays; a first failing entry in comparison-file order must not be confused with the first failing layer in execution order. No recurrence or precision change is justified yet. Runtime CUDA SHA256: Current-package multimodal generation, GenAI full-package pairing, speculation, performance and 24 GB deployment readiness remain unqualified. Current-package single-image component: independently audited PASSThe unchanged The parent independently rehashed the raw/source/profile archives, recomputed the full-tensor F64 metrics and verified exact source retention masks/indices:
Execution manifest SHA256: This is one current-package vision component case, not an embedding/decoder or multimodal-generation pass. Current-package video execution, merged-embedding parity, multimodal generation and full GenAI runtime pairing remain NOT_RUN. Full decoder numerical parity remains FAIL: 1,198 PASS / 298 FAIL; the separate declared ordinary eight-token sequence remains exact. Saved-array localization: chronology, not a new precision fixThe completed CPU-only comparison establishes the actual graph-topological order across all output families. The first candidate prefill threshold failure really is layer-36 V; the earliest cached failed layer is layer-17 output at step 5. Every step's first saved bit drift is the layer-0 raw pre-convolution projection tail, not a convolution output. Cached failures must be interpreted with each trajectory's own already-divergent prior states. Against the original runtime, 1,141 entries pass both, 224 fail both, 57 newly pass and 74 newly fail. Candidate/old layer-0 observed states and first hidden output are bitwise identical at all eight steps, but this does not expose candidate internal projection/normalization registers or prove a sole cause. The independently source-exact first-MoE component was conditioned on source input; the full candidate actually supplies a different preceding hidden state. No recurrence, precision, threshold or reference change is justified by the aggregate counts alone. The local candidate remains unpublished/unselected for shipping. Speculation, performance and 24 GB deployment readiness remain unqualified. Native GenAI tiny pairing: PASS, not full-package qualificationOne existing native test process built from GenAI Actual mapped libraries were core Receipt SHA256: Exact-head golden CI: real initialization failure, then job cancellationAt Mobius head That is a 180 MiB allocation request, before numerical comparison/generation, not a reported accuracy-gate failure. The requested provider order is CUDA then CPU, but the failed arena's device and actual RAM/VRAM availability are unobserved. These runs used serial pytest, not xdist workers. The Arcee failure returned; later Bamba cases timed out, so it was not simply Arcee hanging until cancellation. The workflows, Arcee case/class and session-constructor path are identical to the recorded base, which does not prove the failure is inherited or rule out changed shared graph behavior. No capacity, leak, PR-regression or surgical-fix verdict is established. Both complete official job streams were hashed; bounded traces, setup versions and head/base citations are preserved in Current-package compact video vision: both declared cases PASSThe unchanged current-package vision graph and local candidate runtime now pass both retained clips. Each ran once, in its own fresh one-session/one-forward/profile attempt. The parent independently rehashed raw/source/profile archives, recomputed F64 metrics, checked exact native EVS masks/indices, and inspected all 793 typed profile nodes.
Both preserve final-frame repetition and temporal-patch-2 semantics; these are ordered shifted-photo clips, not natural-video sampling. Native selection was computed by the graph, not injected from source indices. Direct unpruned dense-video output remains unobserved. Placement is not entirely GPU floating arithmetic: each video profile contains 755 CUDA and 38 CPU nodes, including two actual CPU FP64 shape-scalar operations ( The seven-frame final report SHA256 is Together with the previously reported single-image result, all three declared prepared-input vision-component cases pass on the current package. This does not qualify the raw-media processor, merged embedding, decoder/media integration or multimodal generation. Those integration gates remain NOT_RUN/HOLD; the unchanged native candidate's full decoder remains 1,198 PASS / 298 FAIL. One explicitly labeled expanded-FP32 NVFP4 graph plus native packed-IQ2/Q8-cuBLAS diagnostic has now been authorized, after independent 71-test CPU qualification and exact graph/header/source pin checks. It uses two sequential reaped workers, the same frozen 1,496 comparisons/eight tokens and a fresh 776-output invariance check. It has no numerical result yet. This is a separate non-self-contained diagnostic variant, not native FLOAT NVFP4 support, a qualified shipping package, speculative/performance evidence or a 24 GB deployment claim. Initialization update: the first combined diagnostic stopped before any forward: ORT rejected symlinked external data while checking an initializer in a nested If/Scan subgraph. Only embedding construction succeeded; ordinary decoder construction failed, hidden worker never started, and all 1,496 comparisons/776 invariance checks are NOT_RUN, not numerical failures. Worker cleanup succeeded and GPU4 was released. Failed-attempt report SHA256: Supported-layout follow-up: actual native CPU controls now verify that a regular single-link external payload loads, while symbolic links and multiple hard links both produce the expected nested If/Scan policy rejection. No checks were disabled and no payloads were copied or linked. A new, explicitly named unshipped diagnostic header was placed beside the existing expanded diagnostic and original payloads; canonical Independent image-merge reference is ready, not native integration: the unchanged pinned upstream whole Combined diagnostic completed: FAIL, not a shipping remedyThe supported-layout full-v2 attempt completed all three sessions and 24 forwards using two sequential, reaped workers. The parent independently rehashed the sealed evidence and recomputed every one of the 1,496 comparisons from lossless raw arrays, checking original source dtype/value hashes before the declared state-layout adaptations and applying unchanged F64 gates.
All outputs are finite. The actual ordinary greedy tokens remain exactly This explicitly labeled expanded-FP32 NVFP4 graph + native packed-IQ2/Q8 cuBLAS variant performs worse on full-state gates than both separate prior controls: original native 1,215/281, packed-cuBLAS native 1,198/298, combined diagnostic 1,059/437. Local source-conditioned component equality did not establish full-model parity. There is no new production correction or selected shipping variant. The first reported prefill threshold failure is layer22's raw pre-convolution projection-history tail, NRMSE approximately Full report SHA256: Full decoder parity remains FAIL. Exact eight-token identity and passing logits/SSM do not override hidden/KV/conv-history failures. Native FLOAT NVFP4 support, official SM120/121 W4A4 parity, a 24 GB footprint, public GenAI/media generation, speculative integration and performance remain unqualified. No thresholds, reference definitions, cache semantics or padding masks were changed to obtain a pass. Current embedding and prepared native image-feature wiring: scoped PASSOne bounded GPU7 attempt completed one worker, one original current-package embedding session and four acknowledged forwards/comparisons, with no retry. The source-feature image case, unchanged no-media 328-token case and ordinary no-media token18 case are each BF16 bitwise identical to the independently qualified upstream-method reference. The fourth case uses the retained, actual current V4 native vision output—not substituted source features—and passes against that same independent source merged-reference: F64 NRMSE 0.00739508357, cosine 0.99997309339, all finite. All 28 ordinary token positions remain source-bitwise exact; all 300 inserted rows are byte-identical to the actual native feature input. The parent independently recomputed all four comparisons from lossless BF16 outputs, verified the native-feature input lineage and insertion/ordinary-position invariants, rechecked all 21 accessed target embedding rows, and audited the complete four-run profile and actual mapped-library hashes. Placement is 76 CUDA events and four CPU scalar INT64 Greater-to-BOOL events, zero CPU floating events; this is not an all-CUDA-node claim. Actual embedded core/CUDA/shared hashes are Report SHA256: Saved-array localization: an earlier observable bracket, not a recurrence diagnosisThe combined prefill has exact The parent verified all localization artifact seals, independently checked the selected four raw prefill boundaries in both current native/combined variants, and cross-checked all 1,496 paired native/combined metric entries against their frozen full-run records. The combined variant fixes 82 previously failing comparisons but introduces 221 failures versus the native candidate; aggregate correctness therefore worsens despite the smaller first-layer BF16 difference count. Full input normalization/projection, actual convolution output, activation, recurrence readout, grouped normalization and output-projection internals remain UNOBSERVED in these saved full-model arrays. A CPU-only same-input first-layer stage-probe preflight is being prepared; no new GPU attempt or arithmetic change is authorized. Localization report SHA256: Layer-0 source reproducibility probe v3: qualification FAIL, native stages NOT_RUNThe one authorized same-input component attempt stopped at the first unhooked source forward, before source hooks or either native ONNX cut ran. The original source block reproduced its frozen BF16 Parent independently verified the report, manifest, every retained artifact hash/size, all three actual raw endpoints, and recomputed the source-source state metrics in FP64:
All three source outputs are finite. These small errors satisfy the numerical thresholds but do not override the separately declared bitwise reproducibility gate. This is a failed diagnostic qualification, not a new full-model numerical run. No first-stage arithmetic attribution is justified. A concrete invocation mismatch was subsequently found by reading the pinned source: the original full model calls Actual execution: one source model, one source forward, zero hooked forwards, zero native workers/sessions/forwards/profiles, no retry. GPU4 was released and package/source protections passed. Sampled peaks: host RSS 1,606,520,832 bytes; GPU 2,059 MiB. The original failed controller did not retain the source child's exact numeric exit/PID or post-forward maps/flags; those evidence limitations are explicit, not reconstructed. Evidence:
Unchanged readiness: all previously reported full decoder variants still FAIL their declared numerical gates. The eight matching greedy tokens do not establish full parity. Public GenAI/media generation, speculative integration, official SM120/121 W4A4, 24 GB deployment and performance claims remain NOT_RUN/HOLD. Qualified layer-0 probe v4: first observed BF16 activation drift is grouped normalizationThe corrected source caller now uses the original full-model Parent independently verified all artifact seals, actual raw arrays/types/shapes/hashes, original source and native endpoint identities, both native input feeds, source PID/exit/counters, ACK sequences, and one model run per typed profile. The graph variant remains the expanded-FP32 NVFP4 + packed-cuBLAS combined diagnostic, not proof of a native 24 GB deployment. On identical input values, source/native BF16 results are bitwise exact through input normalization, all 501,120 projection values, raw pre-convolution values, raw timestep, convolution, actual SiLU, and actual WITH-D readout. The first observed BF16 activation difference is:
All 12 compared stage/state boundaries satisfy the unchanged finite/NRMSE <= 0.03/cosine >= 0.999 gates. Raw K4-to-K3 convolution-cache comparison is exact. FP32 SSM cache differences are separately observed; "first BF16 activation difference" does not claim no earlier FP32 internal roundoff. Both native profiles contain 607 CUDA events and 26 CPU INT64/BOOL shape/control events, zero CPU floating events, and exactly one model run. One attempt, controller/source exit0, two source forwards followed by two sequential fully reaped native workers/sessions/forwards/profiles. GPU4 released. Sampled peaks: host about 1.62 GB, GPU 2,399 MiB; retained evidence about 56.44 MB; sealing completed in 194.95 seconds. Root cause is not established. The actual source kernel's sigmoid/gating, reduction and multiply-by-reciprocal ordering versus the native Swish/QuickGelu, reduction and division route are investigation candidates. The four BF16 differences are not yet demonstrated to cause the full-model failures. A separate CPU-only, same-input grouped-normalization diagnostic is being qualified; no arithmetic remedy has been applied. Evidence:
Unchanged readiness: full decoder FAIL for all previously measured variants; exact eight-token ordinary sequence PASS only in its stated scope. Public GenAI/media generation, speculation, official SM120/121 W4A4, 24 GB footprint and performance remain NOT_RUN/HOLD. Unexposed original kernel registers, processed timestep and internal recurrence are UNOBSERVED. |
Follow-up review at
|
| Entry point | Runtime timestep minimum | SSM graph input/output dtype |
|---|---|---|
| Direct constructor defaults | 0.001 |
BFLOAT16 |
| Source-converted control | 0.0 |
FLOAT |
The ordinary graphs therefore expose different cache ABIs, and the new source-chunk task rejects the direct controls with TypeError: Chunked prefill candidate requires FP32 cached SSM state.
Please make the Puzzle configuration defaults consistent with the family conversion policy, preserving explicitly supported overrides, and cover direct construction for both Puzzle and Omni. This is not a request to make the experimental source-chunk block accept non-FLOAT state.
2. Full-model numerical qualification is still an open merge gate.
The latest supplied full-model report still records 1,215/1,496 passing comparisons and 281 failures against the unchanged numerical gates. Exact generated-token identity does not clear hidden-state/cache failures. These are author-reported results, not a fresh full-model execution by this review.
The manual-only source-chunk candidate is clearly labelled experimental, which is good. However, the default Puzzle FP32 SiLU and all-mixer/final-normalization changes also affect the ordinary registered model. Please provide reproducible decoder/cache qualification for the exact production artifact/runtime intended for merge, distinguishing it from experimental candidates and unpublished local variants. Alternatively, a separately scoped experimental/component PR can be evaluated on its own acceptance criteria.
Additional findings
- Minor, new — missing shared batch/time validation:
src/mobius/components/_chunked_prefill.py:264-297. WithH=G=P=N=1,x=BF16[2,128,1,1],dt=FLOAT[1,256,1], and B/C matching x, construction andCheckerPassboth accept the inputs. The chunk reshape uses x's batch size and can reinterpret the mismatched operand's time axis as batch. Please reject conflicting known batch/time dimensions and initial-state batch sizes. The probe establishes missing rejection; it does not claim an executed numerical-output mismatch. - Minor, new — JUnit teardown accounting:
tests/ci_reporting_test.py:105. Both CI=true/false teardown cases fail under pytest 9.0.3 because the emitted JUnittestscount is4, not the hard-coded3. Please validate the assertion against supported pytest behavior rather than changing production logging merely to force the count. - Minor, carried — three FP16 CUDA oracle tests:
src/mobius/components/_float32_linear_test.py:96. Biased rank-1/2/3 cases still fail exact PyTorch source-versus-oracle equality before ORT comparison: 2/9 elements, maximum absolute difference2^-20. This is not evidence of an ORT production-output defect. - The earlier early-CLI revision-pin concern is unchanged; no new live misroute was established in this pass.
Verification and review limits
One 13-file offline batch: 480 passed, 5 failed, 110 skipped, with 13 integration cases deselected. The five failures are the three carried numerical-oracle cases plus the two new JUnit cases. Of the skips, 108 require external sealed qualification evidence and two require a pinned local source checkout. Passing graph/API tests do not replace those skipped numerical/placement gates.
Environment: Python 3.12.13, ORT 1.30.0, ONNX IR 1.0.0, PyTorch 2.13.0/CUDA 13.0, pytest 9.0.3, A100. Exact-head import origins were verified in an isolated worktree. No dependency installs, model downloads, full-model exports, performance measurements, or downstream GenAI execution were performed.
The component/block/model/task separation and reuse of existing weights, projections, vision/embedding graphs, and streaming machinery are positive changes. The absence of standard lowerings for the three native packed MatMuls is unchanged, but is not being presented here as a newly demonstrated correctness defect.
Review dispositions/open questions: the inherited ordinary default_task was raised as a Major by integration review; I do not retain it as a blocker because the documented candidate entry point explicitly requires its task instance. Explicit non-FLOAT candidate rejection is intentional, not a second independent Major. Text-candidate hybrid metadata completeness, the downstream BOOL-phase runtime contract, independent source-precision authority, and widened-multiply/add domain limits remain unqualified questions rather than confirmed failures.
No full-model, native packed-kernel, GenAI pairing, performance/24-GB deployment, or complete standard-only qualification is implied by this follow-up.
CUDA diagnostic update: source-order grouped normalization is not a full-model fixThe new full CUDA run completed all 1,496 comparisons. Independent recomputation from the lossless output/reference arrays confirms 1,034 PASS / 462 FAIL, all finite, under the unchanged normalized-RMSE <= 0.03 and cosine >= 0.999 gates.
There were 59 FAIL-to-PASS and 84 PASS-to-FAIL transitions: 25 more failures overall. This candidate must not be promoted as an accuracy improvement or an ORT correctness fix. Small backend rounding differences are not themselves evidence of an incorrect operator implementation. The candidate changes only 40 grouped Mamba norm Div sites per decoder header to Reciprocal followed by Mul. Weights, reference, runtime, masks, cache semantics and numerical thresholds are unchanged. Actual ordinary/hidden CUDA profiles each retain all 40 Reciprocal sites. The first failing comparison in the existing report order is prefill
Expected and actual token IDs both remain The first attempt stopped at the retention/project-allocation guard; its failure and all 57 sealed files remain preserved. This fresh attempt changed only evidence handling: after worker1 exited and was reaped, 99 input and 97 output arrays in its redundant RPC scratch were verified against durable canonical archives or inline actual small inputs before retiring those two scratch files. The original 2 GiB run / 200 GiB project limits were not raised. Both workers exited 0 and were reaped; GPU4 was released; no guard/cleanup error occurred. No production code or PR head changed. Evidence identity:
The reference remains an H100 BF16 reconstruction, not official SM120/121 W4A4 execution or proof of a 24 GB footprint. Shipping remains blocked by decoder parity. Next investigation is localization using retained evidence, not another blind precision variant. |
CUDA diagnosis update: layer-22 tail failure is reproduced by incoming trajectory driftShipping remains blocked. The latest full decoder diagnostic is still 1,034 PASS / 462 FAIL under the unchanged finite-output, normalized RMSE <=0.03 and cosine >=0.999 gates. Its eight greedy tokens match, but that does not establish decoder readiness. No production arithmetic change or new commit is justified by these probes; draft head remains The following bounded H100 diagnostics were independently audited from retained typed arrays, source-weight hashes, runtime provenance and controls:
Why layer 22 is not evidence of a local convolution bug: actual graph ancestry shows At layer 5, original-source expert sets differ between the frozen-source and baseline trajectories on token rows 9,17,21, and between frozen-source and reciprocal-candidate trajectories on rows 9,17. Source cutoff margins are approximately 3.53e-5,1.81e-5,3.72e-5. These are routing-sensitivity observations, not a confirmed ORT routing-kernel defect. A comparison of actual native layer-5 routing with full-layer endpoint controls is in preparation; it has not run. All model floating-point computation in the layer-22 native probe was CUDA-placed. Its profile also contains 18 CPU int64/bool metadata events and 3 CUDA MemcpyFromHost events; it must not be described as an entirely CUDA-only graph. No CPU floating-point model fallback was found. Owned workers exited successfully and were reaped; GPU 4 was released. Reference scope remains the pinned H100 BF16 reconstruction against the explicitly labeled expanded-FP32 NVFP4 diagnostic native graph. This does not establish official SM120/121 W4A4 execution, a 24 GB deployment footprint, performance, or speculative readiness. Preserved local evidence under
Exact launch scripts, separately hashed authorizations, typed raw arrays, commands and process/profile records are preserved in the corresponding preparation/result directories and session artifacts. These are local artifacts, not publicly downloadable attachments. No thresholds were relaxed, no failing evidence deleted, and no performance claim is being made. |
CUDA layer-5 MoE diagnostic: scoped PASS, shipping block unchangedThe corrected observer completed three identical-input source/native comparisons (frozen source, retained baseline, retained candidate). Independent verification covered all 51 sealed files, 45 raw typed source/native arrays, 487 loaded/post-run source tensor hashes, the frozen source endpoint, all 12 ordered source-router controls, and both native retained endpoint controls.
All 18 same-input numerical boundary gates pass the unchanged finite-output / FP64 NRMSE <= 0.03 / cosine >= 0.999 criteria. Expert weights are compared after alignment by expert ID; TopK order is not interpreted as membership. The largest mixture NRMSE is approximately 3.6531e-5. The native expert-set changes between different incoming trajectories reproduce the original-source survey: source-to-baseline rows 9/17/21, source-to-candidate 9/17, baseline-to-candidate 21. No same-input MoE5 routing-set mismatch was observed. This supports upstream activation sensitivity at this boundary, not a routing-kernel defect or a global causal explanation. The native profile contains exactly three model runs, 7,983 CUDA-provider events, and zero CPU-provider events. Both source/native children exited 0 and were fully reaped; the H100 was released. Preserved negative evidence: the earlier run stopped because six observer contracts incorrectly borrowed flattened HF router shapes for ONNX taps that retain the batch axis. The repair changes those diagnostic contracts and adds explicit raw-layout checks plus failure-path exit receipts; it changes no graph, model arithmetic, backend, weights, source controls, or numerical thresholds. That earlier run remains FAIL/NATIVE_INCOMPLETE. Separately, runtime provenance now explicitly includes 37 current shared-library hashes, with 12 newly covered files verified against installed-package RECORD hashes; their historical hashes remain unobserved, not retrospectively qualified. Evidence manifest: Full decoder remains FAIL (1,034 PASS / 462 FAIL of 1,496 for the latest complete diagnostic). This component result does not establish model readiness, official SM120/121 W4A4 compatibility, a 24 GB footprint, performance, or speculative integration. No production arithmetic fix is claimed. |
CUDA diagnostic update: source-conditioned layers 1-5Scoped controls PASS; full decoder remains FAIL (1,034 PASS / 462 FAIL). This is a counterfactual boundary experiment, not an ordinary-decoder rerun or a production fix. Two unchanged native prefixes (baseline and the previously retained Reciprocal/Mul candidate) each ran three cases, changing only the hidden activation supplied after layer 0. The seven auxiliary feeds, including original prefill caches, embedding, mask and phase, were unchanged. Each worker first reproduced all five retained hidden outputs and six layer-5 router taps byte-for-byte before running counterfactual cases: 22/22 controls PASS. No source-model forwards or CPU floating-point model execution ran in this experiment; reference values came from pinned, previously qualified source evidence.
With frozen-source hidden-01 supplied, hidden-02 and hidden-03 became byte-exact in both prefixes; hidden-04 through hidden-06 still differed. Thus this boundary intervention removes the observed first layer-5 membership differences, but does not establish an exclusive/global cause, validate a production remedy, or qualify full-model parity. The earliest remaining observed BF16 normalized-activation mismatch is one element at layer-3 norm on identical hidden-03 inputs. Earlier FP32 sigmoid differences at router 1 remain; layer-3 norm is not the first remaining numerical difference of every dtype. The parent independently verified all 47 sealed files, 271 pins, 54 protected package entries, 138 raw output arrays and 126 FP64 comparisons. Fixed gates remain finite outputs, NRMSE <=0.03 and cosine >=0.999. Profiles contain 55,116 CUDA events, plus 282 CPU integer/bool metadata events and zero CPU floating-point events. Both workers exited 0 and were fully reaped; the H100 returned to 4 MiB. Sampled peaks were 1,426,530,304 bytes aggregate RSS and 3,637 MiB GPU use; these are sampled observations, not formal peak guarantees or a full-model footprint. Evidence: Provenance remains HF revision Shipping remains HOLD. This experiment does not change the previously recorded decoder FAIL, establish official SM120/121 W4A4 execution or a 24 GB deployment footprint, qualify public GenAI/media generation or speculation, demonstrate a performance improvement, or identify an ORT operator defect. Follow-up: isolated layer-3 normalization, CUDA onlyBoth original endpoints reproduced their frozen controls, and both separately observed endpoints reproduced their respective original controls: 4/4 byte controls PASS. Checkpoint, original native external and standalone inline gamma values were byte-exact; original source math, operator attributes, FP32 intermediate precision and runtime policies were preserved. At coordinate
Observed FP32 NRMSE is The parent verified 42 sealed files, all 336 pins and the unchanged 54-entry canonical package. Both source calls used the original Evidence: |
|
Follow-up fixes pushed in a6b850c (parent 1782df1). Keeping this PR draft; this is not a full-model qualification claim. Addressed:
Validation: exact worktree imports verified. Focused combined suite: 157 passed, 2 skipped, 49 deselected on Python 3.12 / Torch 2.14.1+cu130 / H100, with CUDA-only numerical cases selected; no CPU inference campaign. Cases named sealed and CPU-provider numerical cases were excluded by the selection. Two additional chunk-prefill CUDA replay tests skipped because MOBIUS_CHUNK_PREFILL_FIXTURES was unset (confirmed with pytest -rs); these are not numerical passes. Reporting suite separately: 16 passed on pytest 7.4.3. Ruff lint/format and git diff --check pass. Missing xdist/execnet test plugins were installed into a session-only target directory, not the pinned model runtime. Reproduction from this checkout (P=/home/mobius/validation_lightning_h100_normfix_20261011/runtime/bin/python; D is a directory containing the test-only xdist and execnet packages): PYTHONPATH=src:$D PYTHONDONTWRITEBYTECODE=1 CUDA_VISIBLE_DEVICES=4 HF_HUB_OFFLINE=1 OMP_NUM_THREADS=2 OPENBLAS_NUM_THREADS=2 "$P" -m pytest src/mobius/_configs/per_model/_nemotron_h_puzzle_test.py src/mobius/_configs/per_model/_nemotron_h_omni_test.py src/mobius/components/_chunked_prefill_test.py src/mobius/models/nemotron_h_source_chunk_test.py src/mobius/components/_float32_linear_test.py tests/ci_reporting_test.py -q -k "not sealed and not padded_lane_tree_and_fma_on_cpu and not CPUExecutionProvider" --tb=short -p no:cacheproviderStill OPEN:
No full-model, native packed-kernel, GenAI pairing, performance, 24GB-fit, or standard-only qualification is implied. The already-resolved masks/streaming findings remain unchanged. |
|
Additional oracle investigation and fix: 5ff29b2 (parent a6b850c). I reproduced the exact reported failure signature on the available H100/Torch2.14.1 runtime by explicitly setting torch.backends.cuda.matmul.allow_fp16_accumulation=True: biased CUDA FP16 rank1/2/3 each fail the native-source-versus-independent-oracle assertion with 2/9 elements different and maximum absolute error 2^-20. The two tiny cancelling residuals become zero. This is a demonstrated mechanism for that signature; it does NOT prove what backend setting the original A100/Torch2.13 reviewer used. The component is explicitly Float32Linear, with FP32 accumulation through the bias epilogue. Its native low-precision source cross-check previously left the accumulator contract implicit. The test now temporarily disables FP16 accumulation only for that CUDA FP16 source cross-check, using a scoped monkeypatch context that restores the incoming flag. On Torch versions without the flag, no override is applied. The mathematical independent oracle, exact source/oracle assertion, exact ORT-output assertions, fixture values and production arithmetic remain unchanged; no numerical tolerance was loosened. Before this change: 3/3 selected regressions FAIL under ambient FP16 accumulation=True at the source assertion. After this change: all18 CUDA dense epilogue cases PASS under the same ambient=True setting, and an explicit post-suite check confirms the ambient flag is still True. The process restores its original flag in finally. Tiny backend probes also checked cuBLAS/cuBLASLt and reduced-precision-reduction toggles on the two already-installed Torch versions (2.6/CUDA12.6 and2.14.1/CUDA13); only explicitly enabled FP16 accumulation reproduced the signature in these probes. No model downloads or full exports were performed. Reproducer/verification from the exact checkout with its dependencies: import pytest, torch
prior = torch.backends.cuda.matmul.allow_fp16_accumulation
try:
torch.backends.cuda.matmul.allow_fp16_accumulation = True
code = pytest.main([
"src/mobius/components/_float32_linear_test.py", "-q",
"-k", "dense_epilogue and CUDAExecutionProvider",
"--tb=short", "-p", "no:cacheprovider",
])
assert code == 0
assert torch.backends.cuda.matmul.allow_fp16_accumulation is True
finally:
torch.backends.cuda.matmul.allow_fp16_accumulation = priorLint/format/diff checks pass. Matching the original A100/Torch2.13 environment remains unexecuted, so I am not claiming a fresh pass there. Full-model decoder/cache qualification remains the open major merge gate; this reference-test precision correction does not clear it. PR stays draft. |
Add a reusable Float32AccumulationMamba2Block and wire only Nemotron H Puzzle to its source precision boundaries: include bias in FP32 convolution before one native store, and add FP32 readout and D*x before one native store. Keep default Mamba2/Lightning behavior, model-dtype caches, activations, recurrence and packed projection contracts unchanged. Share causal cache preparation and cover zero-width/ND caches, tiny dtype-sensitive graphs, architecture-specific wiring and eight actual-checkpoint CUDA regressions. Full L1 plus related regressions: 1820 passed, 52 existing skips. Actual-checkpoint boundaries match source exactly; 24 separately labeled runtime-v4 local arrays pass unchanged gates. Full corrected decoder parity is not claimed. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> Signed-off-by: Copilot <223556219+Copilot@users.noreply.github.com> Signed-off-by: rui-ren <15321482+rui-ren@users.noreply.github.com>
Annotate causal convolution outputs only when the input dtype is known. This avoids assigning an optional type to strict Value.dtype setters while preserving graph-wide inference for unresolved values. The six remaining Mamba/model typing diagnostics reproduce on exact parent2c9857a3. Validation: integrated L1 and precision selection1795passed60skips; all75 Mamba regressions including eight actual-checkpoint boundary cases passed without skips. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> Signed-off-by: rui-ren <15321482+rui-ren@users.noreply.github.com>
Record fresh source fidelity, vision and ordinary generation evidence without relabeling failed decoder-state and hidden gates. Distinguish the source-matched activation rebuild from the preserved stale runtime, report actual memory/context bounds, and link the passing L1 fix. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> Signed-off-by: rui-ren <15321482+rui-ren@users.noreply.github.com>
Apply gamma to FP32 normalized activations before one native store. Keep the required native biased-Conv store before complete FP32 Swish and a separate native activation store. Wire reusable variants only into Puzzle outer/final norms and Mamba; standard RMSNorm, ordinary Nemotron-H, Lightning, packed projections and the parent Mamba activation contract remain unchanged. Add deferred-dtype, F32/F16/BF16 prefill/cache, parameter-name and 88-layer wiring regressions, plus opt-in strict source-fixture CUDA controls. All 64 focused CPU tests, 47 verified-runtime CUDA tests and two tiny graph builds pass. Ten production fixtures are bitwise source-exact; full-decoder parity is not inferred or rerun. Signed-off-by: rui-ren <15321482+rui-ren@users.noreply.github.com> Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
ModuleList exposes generic Module elements to static analysis. Validate the existing pre-norm as RMSNorm before accessing its epsilon, retaining the exact per-layer value, parameter path and graph operations. Add a varied-epsilon regression across all88 layer replacements. The new Module.norm diagnostic is removed; the six pre-existing MyPy diagnostics remain unchanged. Signed-off-by: rui-ren <15321482+rui-ren@users.noreply.github.com> Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Describe the distinct outer RMSNorm, Conv output and SiLU rounding boundaries. Record source-exact fixture and full-size graph-equivalence evidence without treating the decoder-session timeout as numerical parity or relabeling the preserved v7 failures. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> Signed-off-by: rui-ren <15321482+rui-ren@users.noreply.github.com>
Record actual H100 image/video parity and exact-head smoke checks without promoting unexecuted decoder gates. Document the controlled no-reuse session result separately from numerical qualification and link the dedicated Olive preservation recipe. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> Signed-off-by: rui-ren <15321482+rui-ren@users.noreply.github.com>
Publish complete no-reuse qualification results without relabeling the preserved default-policy timeout. Distinguish passing SSM states and exact tokens from failed KV, convolution and hidden gates, and record the source-hidden projection control without claiming an upstream cause. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> Signed-off-by: rui-ren <15321482+rui-ren@users.noreply.github.com>
Mark native mixed-precision plans as streamed so default saves use bounded shards and serial materialization. Preserve past-plus-current attention masks during cached no-media decoding while retaining strict prefill validation. Normalize encoded infinity inside attribute-based configurations and type the heterogeneous layer topology explicitly. Add focused regressions for all three review findings. Signed-off-by: rui-ren <15321482+rui-ren@users.noreply.github.com> Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> Signed-off-by: rui-ren <15321482+rui-ren@users.noreply.github.com>
Implement parameter-free 128-lane prefill with BF16 dot operands, FP32 mixed-precision GEMM outputs and FP32 chunk carry. Preserve the padded source scan association and keep D, masks, cache phase selection and all model/default wiring outside the primitive. Add CPU graph/API/shape/type regressions and opt-in sealed CUDA intermediate and production-prefix fixture replay with placement audits. Document numerical limits and the explicit cache-lifecycle adapter proposed for a separately reviewed next phase. Signed-off-by: rui-ren <15321482+rui-ren@users.noreply.github.com> Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Expose a reusable explicit-phase Mamba2 block and unregistered text/Omni model-task pairs. Preserve default graphs, learned parameter paths, native cache ABI and cached FP32 recurrence while keeping source-chunk prefill opt-in. Derive optional GenAI phase metadata from the actual BOOL scalar graph input and provide a poisoned success-only validation caller. Cover checkpoint mask transitions, source production boundaries, opposite offset/phase controls and default isolation. Retain numerical diagnostic failures explicitly: this is not bitwise state, attention or full-model qualification. Signed-off-by: rui-ren <15321482+rui-ren@users.noreply.github.com> Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Read production diagnostic inputs through an explicitly configured, SHA-pinned immutable fixture root instead of the execution output directory. Verify every consumed JSON/NPZ, reject overlapping roots before worker writes, record source provenance, and allow explicit test GPU allocation. Document reproducible fresh-output commands and cover root guards without changing model math or qualification gates. Signed-off-by: rui-ren <15321482+rui-ren@users.noreply.github.com> Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Validate the immutable fixture root unconditionally before importing or starting the worker and before output creation. Cover the missing-environment bypass using a forbidden fake worker and an untouched output sentinel. Preserve source/model math, source fixtures, numerical gates and all other runtime behavior. Signed-off-by: rui-ren <15321482+rui-ren@users.noreply.github.com> Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Record the frozen runtime composition and unchanged 281/1496 full-decoder failures. Separate public token-only fixtures, routing replays and dequantized single-projection matches from full-package readiness, including the negative expanded-FP32 experiment. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> Signed-off-by: rui-ren <15321482+rui-ren@users.noreply.github.com>
L4/L5 jobs can hit their job timeout before pytest prints its final failure summary, losing the traceback for tests that already failed. Use the shared pytest configuration to emit and flush each failed setup/call/teardown report through the terminal reporter on GitHub Actions. Only the xdist controller prints worker reports; normal local output stays unchanged. Preserve selection, capture, final summaries, JUnit and exit status. CPU-only subprocess regressions verify traces survive a later hard exit and validate passing runs and normal completion. This improves diagnostic reliability only; it does not resolve or reclassify existing L4/L5 model failures. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> Signed-off-by: Rui Ren <15321482+rui-ren@users.noreply.github.com>
Exercise the existing reporting subprocess helper with the already-declared pytest-xdist plugin and two workers. Cover GitHub Actions and non-CI runs, requiring exactly one controller-side immediate failure header only in CI while preserving final status, JUnit counts and captured output. Keep the original fourteen reporting regressions unchanged. Signed-off-by: Rui Ren <15321482+rui-ren@users.noreply.github.com> Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> Signed-off-by: Rui Ren <15321482+rui-ren@users.noreply.github.com>
Match direct Puzzle and inherited Omni runtime timestep and FP32 SSM cache defaults to source conversion while preserving explicit overrides. Cover ordinary and experimental task cache interfaces. Reject conflicting known batch/time and state batch dimensions before chunk reshapes. Validate JUnit testcase/error payloads across pytest teardown accounting conventions without changing production logging. Full-model CUDA parity and the reviewer-specific FP16 oracle discrepancy remain open qualification gates. Signed-off-by: Rui Ren <15321482+rui-ren@users.noreply.github.com> Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Explicitly enabled native FP16 accumulation reproduces the reported three exact source-versus-oracle failures (two of nine elements, maximum error 2^-20) before ORT output comparison. Scope the native source cross-check to the component's declared FP32-accumulation contract and restore the caller's backend flag afterward. Keep exact assertions, independent oracle, and production arithmetic unchanged. All 18 CUDA dense cases pass with ambient FP16 accumulation enabled and the flag is restored. This does not establish the reviewer's original backend settings or full-model parity. Signed-off-by: Rui Ren <15321482+rui-ren@users.noreply.github.com> Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Record the linear refresh onto current main, focused CPU graph/config/weight/metadata and precision validation, and the November 9 handoff deadline. Preserve the existing numerical failures and native runtime limits; no architecture, rounding or checkpoint contracts are changed. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> Signed-off-by: rui-ren <15321482+rui-ren@users.noreply.github.com>
77404b2 to
4649d18
Compare
Replace routed all-expert expansion with packed IQ2 MoE banks and lower eligible native projections to packed CUDA kernels. Preserve FP16 GGML scale bits, translate signed Q8 payloads losslessly, and tile longer inputs without unsupported BF16 Pad or invalid padded routes. Keep the default exporter, routing, Mamba, attention masks, caches and media components unchanged. The pinned production configuration shrinks from 156423 to 11263 total nodes, including nested bodies. This is a separate reduction-schedule variant, not a full-model numerical or performance acceptance. Add transactional graph/storage regressions, explicit CUDA integration coverage, builder preflight checks and backend documentation. Validation: 140 tests pass with four explicit custom-runtime integration skips; cached built-in CUDA runtime passes projection/MoE tiles, symbolic control flow and the 128-pair boundary. Pinned lint and new-module MyPy pass; builder diagnostics match the original baseline. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> Signed-off-by: rui-ren <ruiren1225@gmail.com>
|
Implemented the opt-in compact CUDA exporter in a2ba3c3, fast-forwarded from the previous PR head. MOBIUS_BOLT_PACKED_CUDA=1 mobius build \
--model nvidia/NVIDIA-Nemotron-3.5-Bolt-74B-A9B-24GB \
--revision c6cc2e8ffa353403213d6b9b18aaf8eb684d705a \
--task nemotron-h-omni --dtype bf16 --ep cuda \
--external-data onnx --output /fresh/path/bolt-packedThe default path is unchanged. The new path lowers 40 routed expert banks to The original router, Mamba recurrence/normalization, attention masks, cache interfaces, and media components are preserved. This deliberately does not substitute the benchmark's verification-snapshot Mamba interface for ordinary prefill/cached decoding. Packed projection reduction schedules and selected-route expert accumulation differ from the original backend, so this is a separately qualified graph variant.
CUDA evidence uses the cached integrated runtime, not a clean rebuild of the latest ORT review heads: core SHA256 Permanent custom-runtime regression entry: MOBIUS_TEST_PACKED_CUDA=1 python -m pytest \
src/mobius/components/_packed_cuda_test.py -k executes_cuda_projection -qLocal receipts, source/runtime hashes, placements, and regression XML are retained under |
Emit FP32 MoEGroupRouter outputs directly into MoEPacked, retaining the original FP32 router projection and external model scaling. Preserve complete IQ2/Q8 blocks in the native packed ABI, leave NVFP4 on its existing operator, and reject incompatible old packed graphs and altered routing expressions. Version the opt-in graph metadata and document normalization and rounding differences. 31 focused exporter/native-layout graph tests passed; Ruff lint and formatting passed. The existing CUDA Q8 precision test was blocked by the test environment lacking MatMulGGMLQ8_0 and is recorded separately, not counted as numerical success. Full exported-model runtime parity and generation remain unqualified. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> Signed-off-by: rui-ren <ruiren1225@gmail.com>
|
Published Task 5 exporter integration in 9fe55f0. The opt-in version-2 packed CUDA graph now uses full FP32 MoEGroupRouter selection/mixing outputs with the original FP32 router projection, consumes them directly in MoEPacked, and keeps model scaling external. Complete IQ2/Q8 blocks are preserved byte-for-byte; NVFP4 remains on its original operator/layout. Old packed graphs and unsupported routing expressions fail closed. Validation: 31 focused exporter/native-layout graph tests passed; Ruff lint/format passed. Runtime APIs were published to microsoft/onnxruntime#33963 and synchronized through the existing dependent PRs. The available Python ORT lacks MatMulGGMLQ8_0, so its legacy CUDA execution test remains explicitly blocked, not counted as a numerical pass. This is a separately qualified graph variant: normalization epsilon/reciprocal and output-rounding boundaries are documented. Full exported-model numerical parity, cached decoding, generation and speculative integration remain NOT_RUN pending Task 6 with the matching runtime. No full-model readiness or performance claim is made. |
Reload the authoritative nested Omni config when a newer native AutoConfig reuses the model type for a different architecture. Avoid generic PretrainedConfig layer proxies for the legacy Puzzle layout, preserving its original decoder and quantization metadata. Add a regression covering the successful-but-incompatible AutoConfig path discovered during full-checkpoint validation. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> Signed-off-by: rui-ren <ruiren1225@gmail.com>
Supply source-normalized and scaled FP32 full mixing values to MoEPacked instead of scaling its already-rounded BF16 output. Keep shared operator APIs and internal native selection unchanged; non-unit normalized scaling uses an explicit FP32 top-k row denominator, while unit-scale routing remains fully internal. Add executable tests for coefficient precision, biased selection, ties, normalization and zero rows. The isolated production expert probe becomes exact for the cached step; this does not assert full-model acceptance. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> Signed-off-by: rui-ren <ruiren1225@gmail.com>
Emit full FP32 selection scores and mixing values at MoEGroupRouter outputs 1 and 2, omitting only the legacy dense output. Preserve six-input MoEPacked routing and pre-store coefficient scaling. Bump packed graph metadata to ABI 3 so previously lowered five-output graphs fail closed and require re-export. Cover serialized wiring, coefficient preparation, idempotency and old-ABI rejection. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> Signed-off-by: rui-ren <ruiren1225@gmail.com>
Select denominator routes with repeated ArgMax and lower-index tie breaking instead of CUDA TopK. Accumulate unbiased mixing weights in selected-rank FP32 order instead of a tree ReduceSum, preserving the validated V3 native routing arithmetic before scaling and the BF16 output store. Add exact CPU/CUDA coefficient regressions for cutoff ties, non-associative sums, zero weights, top-k boundaries, and unnormalized scaling. The new sum-order cases fail against the previous emitter. Weights, recurrence, cache interfaces, and the ORT runtime are unchanged. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> Signed-off-by: rui-ren <ruiren1225@gmail.com>
|
Published the V3 native routing-normalization source correction in 1cf2447. Changed only:
For externally normalized, scaled routing, repeated Validation on the isolated CUDA environment (ORT 25 passed, zero skipped, including all 12 CPU/CUDA-parameterized coefficient cases. Coverage includes cutoff ties, FP32 non-associativity, zero weights, top-k 1/2/8, and unnormalized scaling. The new normalized top-k-8 tests fail on both providers when executed with the previous emitter and pass with this commit. Repository lintrunner formatting and lint passed for both changed files using pinned Ruff 0.16.2; git diff --check passed. This source change implements the arithmetic already qualified in |
Summary
Add CUDA export support for the actual multimodal architecture and native
mixed-format checkpoint storage of
nvidia/NVIDIA-Nemotron-3.5-Bolt-74B-A9B-24GB,pinned to Hugging Face revision
c6cc2e8ffa353403213d6b9b18aaf8eb684d705a.Full-model numerical acceptance remains blocked. This PR does not establish
shipping readiness, full multimodal generation, speculative integration,
performance, or a 24 GB deployment footprint. Export success and matching
greedy tokens do not clear failed hidden-state/cache comparisons.
This description consolidates the implementation and review follow-ups as of
2026-10-20. Current PR head:
77404b2ba5ae73514d891c584db3f0e68333457a, which merges upstreammainafterthe review fixes at
5ff29b274b5d23e19b1aab3d1e8224316b4a3e0b.Historical artifact results below are not qualification of this exact head.
Implementation
per-layer MoE topology, C-RADIOv4 vision encoder, image/video projector and
source-bound EVS prompt compaction. Bolt is not a Lightning alias and has
no built-in MTP.
decoder,vision_encoderandembeddinggraphs withsparse KV, convolution and FP32 SSM state interfaces. Preserve all 88
runtime layer descriptors, including the trailing stateless MoE.
bytes through transactional, fail-closed streaming weight loading. Do not
substitute affine INT2 or reconstruct the complete checkpoint into dense
BF16 storage.
complete-weight and output/bias rounding boundaries.
same-input evidence requires them: RADIOv4 dense MatMul/bias accumulation
in FP32; Puzzle outer/final RMSNorm, convolution epilogues and SiLU with
explicit rounding boundaries. These do not change default
Nemotron-H/Lightning arithmetic.
Reject unsupported dtype/provider/dequantization routes before large I/O.
Generate graph-faithful prepared-feature GenAI metadata rather than a
fictitious ORT-extensions media-processing pipeline.
on absent-media cached steps, including generated token 18. Retain strict
placeholder accounting when media is present.
component/model/task with an explicit BOOL-scalar cache-phase contract.
It is experimental, not a replacement for the ordinary registered route
or evidence of full-model acceptance.
Review findings addressed
0.0and FP32 SSM state. Supported explicit overrides remain intact; the base Nemotron-H defaults are unchanged. Tests cover direct construction, graph cache I/O and ordinary/experimental task entry points for both families.x,dt,BandC, plus incompatible initial-state batch sizes, before reshaping. Preserve valid symbolic dimensions.The first four follow-ups were published in
a6b850c4and5ff29b27.Enabling native FP16 accumulation reproduced the three biased rank-1/2/3
oracle failures: 2/9 elements differed, maximum absolute difference
2^-20. This establishes a test-reference precision vulnerability, notan ORT production-output defect or proof of the reviewer's original flags.
The exact A100/Torch 2.13 reviewer environment was not rerun.
Review dispositions and reproduction details:
configuration/shape/JUnit fixes,
FP16 oracle fix.
Numerical qualification: artifact-specific, not exact-head acceptance
The fixed gates remain: all required outputs finite, normalized RMSE
<= 0.03, cosine similarity>= 0.999, and exact required greedy tokens.No threshold was loosened to obtain a pass.
The retained source-chunk baseline was exported from clean checkout
a8da7270fa2519eec737f103bc1d662b9accc174. These results apply to thatartifact and its recorded runtime, not automatically to the current PR head
or ordinary/default graph:
Expected and observed ordinary greedy tokens for the retained eight-token
controls are:
Logits and token identity passing do not override failed state/hidden
comparisons. Actual-source and native same-input component replays, routing
controls and prefix conditioning localize differences but do not establish a
full-model remedy.
Preserved controls and negative experiments
These are separate artifacts/runtime variants, not successive claims that
the production model passes:
All negative evidence, reference definitions and original packages remain
preserved. Dequantized or expanded-precision diagnostics are not native
IQ2/24 GB deployment qualification. Earlier v5/v6/v7 results are historical
controls, not the current acceptance result.
The qualified layer-0 diagnostic reproduces source/native control endpoints
bitwise and passes all 12 single-block gates. Its first observed BF16
activation drift is grouped norm (4/221,184 values, NRMSE 6.97e-6), but that
has not been established as the cause of the full decoder failure.
Audited evidence and scope.
Validation and CI
with 2 skips and 49 deselections. The two skips require sealed CUDA
chunk-prefill replay fixtures; they are not numerical passes.
deliberately enabled; the source-check scope restored that ambient
setting afterward.
5ff29b27, lint, L1 smoke, L3 synthetic parity, fast integration,build and all six Windows/Linux Python matrix checks passed. L4/L5 were
cancelled; neither is a passing suite.
77404b2, L1 smoke, L3 synthetic parity and lint checkshave passed. The platform unit matrix, fast integration and L4/L5 were
still running at this description update. No all-green CI or
exact-head full-model CUDA acceptance is inferred.
CI graph/API/synthetic coverage is separate from actual-checkpoint
numerical qualification. Tests requiring external sealed evidence or a
pinned local source checkout do not become qualified through skips.
Reference and runtime boundaries
No unnecessary checkpoint redownload or complete dense reconstruction.
with 394 separately validated vision/projector sources. It uses declared
eager text attention, source Mamba Triton semantics and explicit cache
precision; diagnostic projection reconstruction is labeled separately.
Torch 2.14.1+cu130 and Transformers 5.19.0, plus an isolated Python 3.10
ORT CUDA worker. Exact imported paths, source/runtime hashes, commands,
arrays, provider profiles and token sequences remain in artifact receipts.
These are not claims about stock released ORT or a clean build at every
later companion PR head.
explicitly allowed and audited; strict whole-package
CPU-fallback-disabled loading is not qualified.
H100/SM90. The custom H100 packed route does not prove those backend
semantics, public media generation, or the model's advertised footprint.
Companion PRs
Each is a separate deliverable. Their publication, scoped tests or CI do
not establish full-package pairing or model readiness here.
Remaining acceptance gate
Provide reproducible prefill and reused-state decoder qualification for
the exact intended production graph and runtime, including logits,
hidden states, KV/conv/SSM states and exact ordinary generation. Locate and
correct the first divergence before changing arithmetic; retain original
failures and unchanged gates. Experimental source-chunk and local backend
variants must remain explicitly distinguished from the ordinary route.
Until that passes, full multimodal generation, speculative integration,
performance and 24 GB deployment remain unqualified.
Documentation:
docs/nemotron-h-omni.mdanddocs/nemotron-h-omni-processing.md.