Introduction

A source-first runbook for moving PyTorch and Hugging Face projects to Apple MLX: inspect before executing, prove parity before optimizing, and keep every speed claim attached to a local receipt.

The dangerous bug in a model port is the one that looks plausible. A tensor shape lines up, a checkpoint loads without complaint, a sample decodes, and the agent moves on while a mask, RoPE base, layer norm epsilon, or transposed weight is quietly wrong.

MLX Porter exists to make that harder to do. It gives humans and agents one route through source provenance, architecture classification, oracle capture, eager MLX implementation, parity checks, and only then Apple Silicon optimization.

No parity, no optimization. No benchmark metadata, no speed claim.

Explainer diagrams

The runbook is easiest to follow when the agent can see which proof belongs to which phase. These diagrams show the expected path and the places where work must stop instead of drifting into guesswork.

Porting pipeline

Move left to right. Do not jump to the next phase until the current proof exists.

01 Inspect source Config, license, checkpoint format, remote-code risk.

02 Classify family Dense decoder, ASR, diffusion, audio, MoE, SSM, CV.

03 Capture oracle Deterministic source tensors and expected outputs.

04 Port eager MLX Readable graph first, no compile, no kernel shortcuts.

05 Validate and optimize Parity, quality, benchmark metadata, rollback condition.

Parity ladder

Each rung narrows the bug surface before a model-level claim is allowed.

Weights accounted for Every source tensor is loaded, transformed, ignored, or rejected explicitly.

Intermediate tensors match Attention masks, RoPE, norms, scheduler steps, or feature extraction are checked directly.

End output matches Logits, decoded text, waveform metrics, latent trajectory, or class output matches the oracle.

Quality gate passes The port passes the model-family metric before performance claims enter the conversation.

Optimization decision loop

Optimization is a measured loop, not a pile of tricks. One change enters, one gate decides.

Profile Find the real bottleneck Decode, prefill, memory pressure, preprocessing, or scheduler overhead.

Change one thing Apply one scoped method Compile, quantize, cache, batch, fuse, stream, or write a kernel only when supported.

Re-prove Run parity and benchmark Same baseline, hardware, precision, shape, and metric window.

Pass Record the receipt Promote the change with speed, quality, metadata, and rollback notes.

Fail Rollback or hold Keep it out of the supported path until a reproducible MLX route exists.

Repeat Choose the next bottleneck Do not stack speculative changes until the measured one is understood.

What that looks like

For a small Transformer block, do not start by asking for the fastest MLX version. Start by freezing the source behavior and making the first MLX implementation boring enough to debug:

runbook.txt

load mlx-model-porting/SKILL.md
classify architecture family
inspect config + static checkpoint metadata
capture PyTorch oracle tensors
implement eager MLX without compile or kernels
map every weight, with explicit transforms
compare selected tensors
profile the real bottleneck
apply one optimization and re-run parity

If an agent cannot explain which source tensor proves the MLX tensor, the port is not done. If it cannot name the baseline, hardware, precision, prompt shape, and rollback condition, the optimization is not done either.

How it fits together

The docs and repo are split into independent pieces so a small model can follow the path one step at a time instead of discovering the whole porting strategy at runtime.

The skill contract tells the agent how to classify the request, which files to read, and when to stop.
The architecture runbook gives the family-specific checks: attention, diffusion, ASR, audio codecs, MoE, SSM, graph models, and more.
The validation scripts turn claims into gates: source validation, skill audit, golden scenarios, and unit tests.

Frontier systems may discover these decisions dynamically. MLX Porter distills them into a runbook that a smaller open model can execute with fewer wrong turns.

The rules: what to do, what to avoid

Do inspect statically first. Read configs, safetensors headers, source files, licenses, and remote-code risk before running model code.
Do build an oracle. Capture deterministic source outputs and intermediate tensors before writing the MLX path.
Do start eager. The first MLX graph should be readable, not fused, compiled, quantized, or kernel-heavy.
Do not treat loading as correctness. A checkpoint load without missing keys is only one gate, not the port.
Do not make silent speed claims. Optimization needs benchmark metadata, a baseline, and a rollback condition.

Start here

Skill contract

Load this first. It routes the agent to the relevant runbook, scripts, and stop conditions.

Architecture runbooks

Pick the family: dense decoder, ASR, diffusion, audio, MoE, SSM, graph, CV, VLM, and more.

Porting patterns

Common starts for attention blocks, checkpoint maps, Whisper-style ports, and diffusion blocks.

Validation status

What is tested, what is reviewed, and where unsupported techniques must remain blocked.

Choose an architecture route

Each route narrows the parity problem. Do not mix route assumptions unless the model really crosses families.

Starting point	First proof to collect	Common trap
Transformer attention	QKV projections, masks, RoPE, KV cache, and selected logits	Changing SDPA, cache layout, and quantization before tensor parity
Whisper-style ASR	Log-mel features, encoder states, token IDs, WER/CER, timestamps	Comparing decoded text while preprocessing already drifted
Diffusion block	One denoiser timestep, scheduler math, and fixed latent trajectory	Optimizing sampling while the scheduler or conditioning path differs
Checkpoint loading	Static tensor inventory, explicit transforms, and missing-key policy	Accepting partial loads or shape-compatible transposes silently

Validation gates

The lightweight local path exercises the repo without downloading a real model:

offline-smoke.sh

python3 mlx-model-porting/scripts/manifest.py check
python3 mlx-model-porting/scripts/audit_skill.py --strict
python3 mlx-model-porting/scripts/validate_sources.py
python3 -m unittest discover -s tests -v

A real model port adds architecture-specific tensor parity, quality metrics, generation or inference benchmarks, and rollback conditions.

For agents

The intended agent behavior is simple: read the compact contract, choose one family route, keep facts sourced, and fail loudly when a required proof is missing. The runbook should feel like rails, not a suggestion list.