Skills DirectorySearch all skills ↗AI agent skill · by @BBuf
Model Compute Simulation
Build an operator-level compute template for an LLM and estimate FLOPs/MFU for a serving shape. Use when you need tensor shapes, per-op FLOPs, kernel-to-op MFU mapping, or parallelism what-if analysis.
Archived 2026-10-23 · Source updated 2026-10-05. Check the original repository for the latest version.
SKILL.md instructions
# Model Compute Simulation
## Overview
Use this when the question is about operator order, tensor dimensions, FLOPs,
MFU, or parallelism checks. The simulator loads a model config, builds the
representative operator sequence, prints tensor shapes and FLOPs, and can
estimate MFU from measured latency.
## Inputs
Before running a simulation, collect or verify these inputs:
| Item | Why it matters | How to obtain | Default if user skips |
|---|---|---|---|
| Model name | Resolves to config in `model-config-index.json`; determines entire architecture | Ask user or infer from trace context | — (required) |
| Config accuracy | Indexed values may differ from actual serving config (e.g. `routed_expert_intermediate_size`, `compress_ratios`) | Ask user to provide `config.json` or verify key params against HuggingFace | Use indexed values with a caveat |
| GPU type | Determines peak FLOPS for MFU denominator | Ask user | — (required for MFU) |
| dtype (bf16 / fp8) | Affects peak FLOPS selection; fp8 doubles peak | Ask user | bf16 |
| Batch size & seq len | Directly affects FLOPs and tensor shapes | Ask user | B=1, S=1 (decode) |
| TP / DP / EP | TP splits GEMM FLOPs across GPUs; EP splits expert FLOPs | Ask user | Read the run manifest; otherwise report assumptions explicitly |
| Measured latency (ms) | Required for MFU numerator; must be per-GPU forward-pass wall-clock | Ask user or extract from a profiler trace | — (optional, no MFU without it) |
If the model is not in `model-config-index.json`, ask the user for a
`config.json` path or add an indexed config before running estimates.
The 2026-10-05 index pins public HF config revisions for V4 Flash/Pro/V4.1,
K3, GLM-5.2/5.3/5.3-Flash, Qwen3.8-Flash-Next, Hy3/Hy4, Inkling, dots3,
Step-3.7, Nemotron-3/3.5, LongCat and Gemma4, and corrects older model dimensions.
`Qwen/Qwen3.8-A95B` returned HTTP 401 at this audit and is not indexed.
Use `--config /path/to/config.json` for a raw nested HF config; the local
normalizer expands text wrappers and expert/layer aliases.
## Shape and timing limits for speculative MoE
For DSV4.1, obtain actual target/draft row counts from the trace: request BS=1
can issue six target-verify rows with DSPARK block size 5. Do not substitute
acceptance length for the number of rows computed. Account for TP-sharded
shared-expert shapes separately from routed expert assignments and EP.
Do not divide an unverified model template by TP/EP and call it measured FLOPs.
The simulator applies uniform TP division outside routed experts and uniform
EP division inside routed experts. This is a what-if approximation: replicated
routers, norms, latent projections and backend-dependent shared-expert
ownership require a run-specific adjustment before interpreting MFU.
The pinned V4.1 entry is a static target-model template; DSPARK draft, vision,
Engram/PLE memory traffic and backend dispatch require separate evidence.
Kernel-duration sums include overlap and possibly PDL dependency waits. Use
wall-clock time for a whole-pass MFU denominator, and measured GPU SKU/dtype
peaks rather than assuming every FP8 operation doubles useful FLOPs. Follow the
[dispatch and timing principles](../llm-torch-profiler-analysis/references/heuristics.md#establish-dispatch-and-numerical-contracts)
when assigning kernel names to attention, shared experts, routing or mHC.
## Workflow
### Step 1: Load model config
Resolve the model name and load its configuration parameters:
```bash
python3 skills/model-compute-simulation/scripts/model_compute_simulator.py "<model name>" --list-models
```
The script resolves the model name against `references/model-config-index.json`, which stores public HuggingFace config parameters (hidden_size, num_experts, MLA ranks, etc.).
If the model is not indexed, tell the user to provide a `config.json` path or request an index update.
### Step 2: Generate execution flow and tensor dimensions
Run the simulator with batch size, sequence length, and parallelism configuration:
```bash
python3 skills/model-compute-simulation/scripts/model_compute_simulator.py "Qwen3-235B-A22B" \
--batch-size 1 --seq-len 1 \
--tp 8 --dp 1 --ep 8 \
--gpu b200 --dtype bf16
```
The simulator prints:
- Per-layer operator sequence with FLOPs and tensor shapes (shape_in → shape_out)
- Attention vs MoE/FFN FLOPs proportion per layer
- Total model FLOPs for a single forward pass
For **decode**: use `--seq-len 1 --context-len TOTAL_KV_LENGTH`.
For extend, `--context-len` is the prefix length, with causal work for the new
`--seq-len` tokens. Full/window attention includes KV context in QK and AV FLOPs.
The LM head computes last-token logits unless `--all-token-logits` is given.
Every run prints batch/shape/parallelism/GPU/dtype assumptions to stderr.
Sparse selection, KDA/Mamba scalar work, norm/Sinkhorn and zero-expert top-k
are architecture estimates, not measured instruction counts. LongCat routed
FLOPs assume all top-k are nonzero (an upper bound until routing is measured).
For **prefill**: use `--seq-len <prompt_length>`.
### Step 3: Estimate MFU with measured latency
Provide the measured forward-pass latency to compute MFU:
```bash
python3 skills/model-compute-simulation/scripts/model_compute_simulator.py "Qwen3-235B-A22B" \
--batch-size 1 --seq-len 1 \
--tp 8 --dp 1 --ep 8 \
--gpu b200 --dtype bf16 \
--measured-ms 15.0
```
MFU = theoretical_min_time / measured_time × 100%
The simulator prints:
- Overall MFU
- Per-layer MFU (uniform layer-time assumption)
- Per-operator FLOPs proportion (for identifying which ops dominate)
GPU peak FLOPS are loaded from `references/gpu-specs.json`. The bundled
hardware table includes H100/H200/B200, B300, GB300, MI355X, RTX PRO 6000
and DGX Spark, each with official URLs. H20 is retained as an unverified
historical estimate. Unknown dense peaks are null and MFU for that dtype is
refused; RTX Server and Spark sparsity is unconfirmed. B300/GB300 memory
capacity is physical maximum, not usable allocation. `--dtype` accepts bf16,
fp8/mxfp8 and fp4/mxfp4/nvfp4; INT8 must use its separately published peak. Use aliases such as `--gpu h100`, `--gpu b2000`, or `--gpu b200` when
running on those local boxes.
### Step 4: Per-operator MFU with kernel-level latency
When you have per-kernel measured latency, compute per-operator MFU by mapping
kernel durations to the compute flow.
#### Method A: `--kernel-flow` (kernel-level MFU, recommended)
Provide per-kernel detail as JSON, then feed it to the simulator for
kernel-level MFU analysis. This preserves every kernel row from the compute
flow and adds FLOPs/MFU columns.
```bash
python3 skills/model-compute-simulation/scripts/model_compute_simulator.py "Qwen3-235B-A22B" \
--batch-size 1 --seq-len 8192 \
--tp 8 --dp 1 --ep 8 \
--gpu b200 --dtype bf16 \
--measured-ms 100 --kernel-flow @/tmp/layer3_detail.json
```
The `--kernel-flow` parameter accepts a JSON string or `@file` path. It produces
a **kernel-level MFU table** that preserves all kernel rows from the compute
flow and adds:
- `Mapped Op`: which operator this kernel maps to
- `FLOPs`: operator's total FLOPs
- `Theo(us)`: theoretical minimum time
- `MFU%`: measured FLOPs utilization
- `shape_in→shape_out`: operator tensor dimensions
When `--kernel-flow` is provided, the static per-operator template is omitted
because the kernel-level MFU table already carries per-kernel shape and FLOPs
information. The output keeps the model summary, serving configuration, total
FLOPs, overall MFU calculated from the measured layer duration, and the
kernel-level MFU table. In `--format json`, the raw evidence is stored under
`kernel_flow` and no text table is appended.
Mapping rules:
- **Direct-match kernels** (mla, moe, mhc, rmsnorm, hadamard, rope, quant, topk, etc.): time is assigned directly to the corresponding operators
- **Generic GEMM kernels** (gemm_fp8, gemm_bf16): time is distributed to remaining unassigned projection GEMM operators by FLOPs share
- **Overhead kernels** (allreduce, moe_align, moe_sort, other): rows preserved, FLOPs/MFU marked as N/A
**FP8 kernel MFU correction**: Kernels in categories `moe` (fused_moe_kernel)
and `gemm_fp8` use fp8 math internally even when `--dtype bf16` is specified.
For these kernels, the MFU denominator uses the GPU's fp8 peak FLOPS
(2x bf16 peak) instead of bf16 peak. The resulting MFU is marked with a
superscript `⁸` (for example, `63.7%⁸`) to show that the fp8 denominator was
used. `gemm_bf16` kernels still use the bf16 peak FLOPS denominator.
#### Method B: `--kernel-detail` (operator-level MFU, legacy)
Same input as `--kernel-flow` but outputs an **operator-level** summary table
(aggregated by operator, not per-kernel). Use when you want a compact view.
```bash
python3 skills/model-compute-simulation/scripts/model_compute_simulator.py "Qwen3-235B-A22B" \
--batch-size 1 --seq-len 8192 \
--tp 8 --dp 1 --ep 8 \
--gpu b200 --dtype bf16 \
--measured-ms 100 --kernel-detail @/tmp/layer3_detail.json
```
#### Method C: `--kernel-ms` (category-level approximation)
Use a hand-aggregated category map only when per-kernel detail is unavailable:
```bash
python3 skills/model-compute-simulation/scripts/model_compute_simulator.py "Qwen3-235B-A22B" \
--batch-size 1 --seq-len 8192 \
--tp 8 --dp 1 --ep 8 \
--gpu b200 --dtype bf16 \
--measured-ms 100 --kernel-ms '{
"mla": 4.922, "moe": 1.644, "allreduce": 0.769,
"hadamard": 0.348, "mhc": 1.388, "gemm_fp8": 1.692,
"gemm_bf16": 0.125, "rmsnorm": 0.227, "quant": 0.311,
"rope": 0.209, "topk": 0.122, "activation": 0.071,
"other": 0.437
}'
```
The `--kernel-ms` parameter accepts a JSON object mapping **kernel category** names
to their measured durations in **milliseconds**. It uses FLOPs-proportional
distribution across entire categories, which is less precise than `--kernel-detail`
because generic GEMM categories (gemm_fp8, gemm_bf16) span multiple operator categories.
Choose only one of `--per-layer-ms`, `--kernel-ms`, `--kernel-detail`, or
`--kernel-flow`. All measured durations must be positive. The three kernel
inputs also require `--measured-ms` from whole-pass wall-clock timing; the
100 ms in these examples is a placeholder to replace with the measurement.
Kernel-duration sums cannot establish elapsed time when streams overlap.
Output includes:
- Model architecture summary (layers, hidden_size, attention_type, MoE config)
- Per-layer compute flow: operator sequence with tensor dimensions, FLOPs, shape_in→shape_out
- Per-operator MFU table: each operator's FLOPs, theoretical time, measured time (from trace), MFU%
- Kernel → operator mapping explanation (direct-match vs FLOPs-proportional vs overhead)
- Overall and per-layer MFU
## When To Use It
- when you need compute-level detail for a known model or config
- when the user asks about execution flow, tensor dimensions, or FLOPs for a specific serving shape
- when the user asks about MFU and can provide measured forward-pass latency
- when comparing compute profiles across different parallelism configurations
## Useful Commands
List known model IDs:
```bash
python3 skills/model-compute-simulation/scripts/model_compute_simulator.py --list-models
```
List known GPU types:
```bash
python3 skills/model-compute-simulation/scripts/model_compute_simulator.py --list-gpus
```
Emit JSON for automation:
```bash
python3 skills/model-compute-simulation/scripts/model_compute_simulator.py "GLM-5" --format json
```
## Reporting Checklist
Include:
1. **Model architecture summary**: model name, config source, num_layers, hidden_size, attention_type, MoE config (num_experts, topk, shared_experts), MHC, head_dim
2. **Serving configuration**: batch_size, seq_len, TP, DP, EP, GPU, dtype
3. **Per-layer compute flow** (showing first representative layer in detail):
- Operator sequence table: name, category, FLOPs, shape_in → shape_out
- Attention vs MoE/FFN FLOPs proportion
4. **Total model FLOPs** for a single forward pass
5. **Kernel-level MFU table** (when `--kernel-flow` provided):
- Preserves ALL kernel rows from the compute flow (never deleted)
- Per-kernel columns: `# | Half | Category | Simplified Name | dur(us) | % | Mapped Op | FLOPs | Theo(us) | MFU% | shape_in→shape_out`
- Direct-match kernels: show mapped operator FLOPs/MFU
- Overhead kernels: show N/A for FLOPs/MFU, row preserved
6. **Operator-level MFU table** (when `--kernel-detail` or `--measured-ms` provided):
- Each operator: name, category, total FLOPs, per-GPU FLOPs, theoretical time, measured time (from trace), MFU%
- Kernel category → operator mapping explained
7. **Overall MFU** and **per-layer MFU**
8. **One-line summary**: dominant compute category, MFU status, key bottleneck
## Trace-Based Validation (extract_compute_flow_from_trace.py)
Use `scripts/extract_compute_flow_from_trace.py` to extract the real operator sequence and tensor dimensions from a torch profiler trace, then compare against the static template as ground truth validation.
```bash
# Extract compute flow from a trace
python3 skills/model-compute-simulation/scripts/extract_compute_flow_from_trace.py \
--input /path/to/trace.json.gz --format text
# Compare trace against static template
python3 skills/model-compute-simulation/scripts/extract_compute_flow_from_trace.py \
--input /path/to/trace.json.gz \
--compare qwen3-235b-a22b \
--batch-size 1 --seq-len 1 --tp 8 --ep 8
```
Trace/template operation presence is compared through semantic families such
as `matmul`, `attention`, `norm`, and `router`; raw `aten::*` names are not
compared directly with semantic template names. A positive `--min-flops`
threshold removes operations whose FLOPs could not be determined.
### Compute Flow Confirmation Hierarchy
When the static template or trace extraction cannot fully confirm the compute process (e.g. ambiguous scope, missing shapes, new model architecture), follow this escalation hierarchy:
1. **Static template** (`model_compute_simulator.py` + `model-config-index.json`) — fast, covers known models
2. **Trace extraction** (`extract_compute_flow_from_trace.py`) — validates template against real execution
3. **Inference framework source code** — when trace is insufficient (missing `Input Dims`, CUDA Graph replay, compiled kernels without scope), read the model's forward flow directly from the serving framework source:
- **SGLang**: `python/sglang/srt/models/<model_name>.py` — contains the `forward()` method and `declare_attn`/`declare_ffn` stage boundaries in `python/sglang/srt/layers/layer_boundary/factories.py`
- **vLLM**: `vllm/models/<family>/{common,nvidia,amd}/` for migrated 2026 families; older models remain under `model_executor/models`
- **TensorRT-LLM**: `tensorrt_llm/_torch/models/modeling_<family>.py` + model configs
When consulting framework source, focus on:
- The `forward()` method: operator call order and residual connections
- QKV / O projection: whether LoRA-style down/up projections are used (`q_lora_rank`, `o_lora_rank`)
- MoE routing: top-k selection, shared vs routed expert split
- TP/EP slicing: which dimensions are split and how FLOPs divide across GPUs
- Any model-specific ops not in the static template (e.g. MHC, Hadamard, indexer)
**Action**: If the framework source reveals discrepancies with the static template, update `model-config-index.json` and/or `build_layer_ops()` accordingly.
### Limitations of Trace Extraction
| Limitation | Detail | Workaround |
|---|---|---|
| `record_shapes=True` required | Trace must be captured with shape recording enabled; without it, `Input Dims` fields are absent and FLOPs cannot be computed | SGLang live capture and vLLM `torch_profiler_with_stack=true` already enable this; TensorRT-LLM requires a `py_executor.py` override adding `record_shapes=True` |
| CUDA Graph mode | During graph replay, `cpu_op` events may only appear once (at capture time); shape information for replayed iterations is not re-recorded | The script detects graph capture phases and annotates affected ops; use eager-mode traces for full coverage |
| TP-sliced dimensions | Trace shows post-TP-split dimensions (e.g. `H/TP`), not the full-model view | Use `--tp` in `--compare` mode to scale trace FLOPs back to full-model equivalents |
| Scope attribution quality | Python scope depends on `with_stack=True`; some frameworks or compiled paths may produce shallow or missing scope chains | Graceful degradation: ops with unresolved scope are categorized as "other" |
| Not a replacement for static templates | Trace extraction is a validation and discovery tool; static templates remain the primary fast-analysis path | Use trace extraction to verify templates for new models, then update `model-config-index.json` if discrepancies are found |
## References
- `references/model-config-index.json`: model configuration parameters (hidden_size, expert counts, MLA ranks, etc.).
- `references/gpu-specs.json`: GPU peak FLOPS specifications for MFU calculation.
- `scripts/extract_compute_flow_from_trace.py`: trace-based compute flow extraction and template validation tool.
Source audit: [SGLang V4](https://github.com/sgl-project/sglang/blob/b1bbd74f287f13ed1276b0403a01ebb55c597e93/python/sglang/srt/models/deepseek_v4.py),
[K3 latent experts](https://github.com/sgl-project/sglang/blob/b1bbd74f287f13ed1276b0403a01ebb55c597e93/python/sglang/srt/models/kimi_k3.py),
[NVIDIA HGX dense/sparse specs](https://www.nvidia.com/en-us/data-center/hgx/),
[AMD MI355X specs](https://www.amd.com/en/products/accelerators/instinct/mi350/mi355x.html).
HF field provenance is `hf_config_source` at `hf_config_revision`; construct
`https://huggingface.co/<repo>/blob/<revision>/config.json` for the exact source.
No GPU run is implied by a config refresh.