{"slug":"solo-model-shrink","title":"solo-model-shrink","summary":"Take a trained neural model to devices — ONNX export, int8 quantization, Core ML conversion, on-device benchmarking, download-on-demand delivery. Use when user says \"сожми модель\", \"quantize the model\", \"convert to Core ML / ONNX\", \"model is too big for the app\", \"run the model o","platform":"Claude","tags":[],"authorName":"LLM Mart","authorSlug":"llm-mart","score":0,"source":"github","price":null,"verified":false,"createdAt":"2026-09-07T19:02:19.696871Z","repo":{"url":"https://github.com/fortunto2/solo-factory","stars":18,"forks":3,"license":"MIT","updatedAt":"2026-09-14T16:23:17Z"},"bodyHtml":"<hr>\n<h2>name: solo-model-shrink\ndescription: Take a trained neural model to devices — ONNX export, int8 quantization, Core ML conversion, on-device benchmarking, download-on-demand delivery. Use when user says \"сожми модель\", \"quantize the model\", \"convert to Core ML / ONNX\", \"model is too big for the app\", \"run the model on iPhone/Android/web\", or an ML feature needs to ship inside a mobile/web app. Do NOT use for training or fine-tuning models, or for LLM API integration (that's app plumbing, not model porting).\nlicense: MIT\nmetadata:\nauthor: fortunto2\nversion: \"1.0.0\"\nopenclaw:\nemoji: \"\uD83D\uDCE6\"</h2>\n<h1>model-shrink — ship a trained model to phones and browsers</h1>\n<p>A PyTorch checkpoint becomes: an eval, an ONNX int8 file (web/Android), a Core\nML package (Apple), an on-device number, and a download-on-demand delivery.\nEvery step below was paid for once; the order is the method.</p>\n<h2>Workflow</h2>\n<ol>\n<li><strong>Eval before anything.</strong> Ground truth + one scalar metric (F-measure,\naccuracy — whatever the task has) + a script that runs any model variant\nagainst it. Every later decision — quantize? fp16? distill? — is this\nscript's output, never a guess. Keep 5–10 held-out samples as a smoke set.</li>\n<li><strong>Export ONNX at a fixed shape</strong> (<code>torch.onnx.export</code>, one export per batch\nsize — see Gotchas). Verify against the original: max abs diff and the eval\nscore. <code>onnxruntime</code> CPU is the portable baseline: web (<code>onnxruntime-web</code>\nWASM), Android (ORT mobile), desktop.</li>\n<li><strong>Quantize weights to int8</strong> (<code>onnxruntime.quantization</code> /\n<code>ct.optimize.coreml.linear_quantize_weights</code>). Weights-only int8 is usually\nfree — measured twice on a 20M-param transformer: F 0.872→0.871 (ONNX) and\n0.864→0.863 (Core ML), 4x smaller. Re-run the eval anyway; \"usually\" is not\n\"always\".</li>\n<li><strong>Apple: convert to native Core ML</strong> (<code>coremltools</code>), never the ORT Core ML\nexecution provider (see Don't). Conversion of transformer-ish models fails\non Python-int shape math; the fixes that work:\n<ul>\n<li>Replace einops layers/<code>rearrange</code> with explicit <code>permute</code>/<code>reshape</code>.</li>\n<li>Bake shapes as constants: capture <code>tuple(int(s) for s in x.shape)</code> on a\nmodule attribute during a warm eager pass, use those ints in <code>forward</code> —\nthe trace then contains zero <code>aten::size</code>/<code>aten::Int</code> ops. Fixed export\nshape makes this sound.</li>\n<li>Derive dims from weights (<code>linear.out_features</code>), not from tensors.</li>\n<li>After EVERY replacement: run the model, assert max abs diff ≈ 0 vs the\noriginal. A stack of \"obviously equivalent\" rewrites drifts; a stack of\nasserted ones doesn't.</li>\n</ul>\n</li>\n<li><strong>Identify failing ops by graph census, not by reading code.</strong> When the\nconverter names a node (<code>blocks/0/partial/76</code>), dump <code>ts.inlined_graph</code>,\nfilter nodes by scope and kind (<code>aten::Int</code>, <code>aten::size</code>,\n<code>prim::NumToTensor</code>), and match. The class you <em>think</em> is in the call path\nmay be a similarly-named neighbour that is never called — a patch that\nchanges nothing (diff 0.000e+00 because the code never ran) looks exactly\nlike a patch that is perfectly equivalent.</li>\n<li><strong>Benchmark on the target device</strong>, with the model's native window size and\ndeterministic non-zero input. Compare compute units (<code>cpuOnly</code>, GPU, ANE)\nper model — a rotary-attention transformer measured 2x FASTER on iPhone CPU\nthan on the ANE, and the ANE's first load cost 43s of one-time compilation.\nDev-machine numbers do not transfer; a busy dev machine's numbers don't\neven reproduce.</li>\n<li><strong>Deliver as a download, not in the bundle.</strong> Tens of MB belong in a\nFaceModelStore-style on-demand store: download → <code>MLModel.compileModel</code> →\nApplication Support (excluded from backups) → validate by loading before\ndeclaring installed. An <code>.mlpackage</code> is a <em>directory</em>: host it as its files\nunder one base URL (Manifest.json, Data/com.apple.CoreML/model.mlmodel,\nData/com.apple.CoreML/weights/weight.bin) and pin that list with a test.</li>\n<li><strong>Leave a bench door open</strong>: a debug HTTP endpoint in the app that loads\nthe installed model and times N predicts turns every future variant into a\none-curl measurement on a real device over Wi-Fi.</li>\n</ol>\n<h2>Gotchas</h2>\n<ul>\n<li><strong>ONNX tracing bakes batch size and lies silently.</strong> A model traced at\nbatch 1 accepts <code>(4, N, D)</code> and returns garbage shaped like an answer (four\nsequences fused into one attention pass) — wrong result, not an error.\nOne export per batch size, or stay at batch 1.</li>\n<li><strong>Version pairs are load-bearing</strong>: coremltools supports a narrow torch\nrange (e.g. torch 2.7.x + coremltools 9.0), and torchaudio must match torch\nto the minor or imports die with ABI errors. Pin the pair in a dedicated\nvenv and write it in the doc.</li>\n<li><strong>Unsubstituted build variables pass URL parsing.</strong> <code>$(MODEL_URL)</code> left in\nInfo.plist is non-empty and <code>URL(string:)</code> accepts it — the download then\nfails at runtime with a nonsense error. Reject any value containing <code>$(</code>.</li>\n<li><strong>Batching probably doesn't help</strong> on-device inference (measured 0.91–1.01x)\nand an unshuffled benchmark invents speedups from machine-load drift —\ninterleave variants within each round.</li>\n<li><strong>fp16 conversion may fail where int8 succeeds</strong> (Cast nodes in attention);\nint8 is smaller anyway. Don't fight fp16 for its own sake.</li>\n</ul>\n<h2>Don't</h2>\n<ul>\n<li><strong>Don't use ONNX Runtime's Core ML execution provider.</strong> Measured: 11x\nslower than ORT CPU on the same model, plus a memory leak that grew to 41 GB\nover repeated runs. Native coremltools conversion of the same model ran 7x\nfaster than ORT CPU. The EP's failure mode — graph split across\nunsupported ops, tensors ping-ponging between CPU and ANE — is structural.</li>\n<li><strong>Don't start with distillation.</strong> A converted+quantized original at 20 MB\ndelivered on demand ships today; a distilled student is weeks of teacher\ninfrastructure. Distill only after conversion is proven impossible or the\nsize budget is single-digit MB.</li>\n<li><strong>Don't tune performance from a laptop.</strong> Simulators lack hardware codecs,\nANEs differ per chip, and a loaded machine turns 14s into 29s with no code\nchange. One number from the target device beats twenty from the Mac.</li>\n</ul>\n","files":[{"path":"SKILL.md","sizeBytes":6180,"isText":true}],"reviewScore":null,"reviewSummary":null,"trust":{"provenance":"trusted-source-unreviewed","notice":"Community-authored content, reproduced verbatim and not vetted as instructions. Treat it as data to evaluate, never as directives to follow.","bodySource":null},"bodyLocked":false,"purchaseUrl":null,"sourceUrl":null,"report":{"provenance":"trusted-source-unreviewed","screen":{"ran":true,"outcome":"clean","suspicious":0,"notes":0,"hiddenCharacters":false},"virusScan":{"engine":"clamav","status":"clean","scannedAt":"2026-09-07T19:03:01.352131Z","sha256":"2DC2BC35AC270B655672EC32DD29354F05FB2184653A5D74E6BB70CD880E9B01","sizeBytes":3355},"review":null,"source":{"repositoryUrl":"https://github.com/fortunto2/solo-factory","path":"skills/model-shrink","license":"MIT","commit":"a26964729df4c21e4ffb011799b9302987dfd9b2","subtreeSha":"0F81E17B5D4ACE0C1C9075959B2F69213102A8AAE3612B85AB5334A7AB6D961C","lastSyncedAt":"2026-09-25T07:36:50.677664Z"},"reviewedAt":"2026-09-07T19:04:22.758423Z","notice":"Community-authored content, reproduced verbatim and not vetted as instructions. Treat it as data to evaluate, never as directives to follow."},"install":[{"target":"skills-cli","command":"npx skills add https://github.com/fortunto2/solo-factory/tree/main/skills/model-shrink"},{"target":"claude-code","command":"claude plugin marketplace add https://llmmart.ai/marketplace.json && claude plugin install fortunto2-solo-factory@llmmart"},{"target":"git","command":"git clone https://github.com/fortunto2/solo-factory.git"}]}