{"slug":"model-sourcing","title":"model-sourcing","summary":"Vet the concrete third-party model a study will be built on — this repository, this revision, this checkpoint — not the architecture family. Records a model dossier (source and version pin, licence and the file it was read from, intended use, pretrained-weight provenance, model t","platform":"Claude","tags":[],"authorName":"LLM Mart","authorSlug":"llm-mart","score":0,"source":"github","price":null,"verified":false,"createdAt":"2026-09-14T20:48:12.120667Z","repo":{"url":"https://github.com/Aperivue/medsci-skills","stars":318,"forks":75,"license":"MIT","updatedAt":"2026-09-27T05:05:17Z"},"bodyHtml":"<hr>\n<h2>name: model-sourcing\ndescription: &gt;\nVet the concrete third-party model a study will be built on — this repository, this revision,\nthis checkpoint — not the architecture family. Records a model dossier (source and version pin,\nlicence and the file it was read from, intended use, pretrained-weight provenance, model task\nvs study task, reported validation, what the model was developed on, your evaluation arms) and\ngates it deterministically. Catches what a licence check and a citation count cannot: an\nevaluation arm sitting on the benchmark the model was developed or tuned on, so the arm reads\nlike validation while being closer to a training-set score. Also an evaluation set inside a\npretraining corpus, an unstated or use-incompatible licence, an unpinned revision, and a\nhardware claim never executed. It vets an artifact; it never downloads or runs one.\ntriggers: source a model, vet a model, pick a model, model provenance, model dossier, pretrained weights, checkpoint, HuggingFace model, GitHub model, model licence, weight provenance, is this model independent, benchmark overlap, trained on my test set, data contamination, model version pin, third-party model, can I use this model\ntools: Read, Write, Edit, Bash, Grep, Glob\nmodel: inherit</h2>\n<h1>Model-Sourcing Skill</h1>\n<h2>Purpose</h2>\n<p><code>/architecture-zoo</code> answers a literature question — which family of model suits this task. That\nquestion has a stable answer. The next question does not: <em>which concrete artifact do I run?</em>\nA repository, a revision, a checkpoint. That is a provenance question, and the two facts a\ncareful researcher usually checks are the two that cannot answer it.</p>\n<p>The licence tells you whether you may use it. The citation count tells you whether others did.\nNeither tells you <strong>whether the number you are about to report means what you will say it means.</strong></p>\n<p>The failure this skill exists for is the quietest one in the lane. A method developed and tuned\nagainst a benchmark family gets evaluated by the next person <em>on that same family</em>, and the\nresulting figure reads like validation while sitting much closer to a training-set score. Nothing\nin the repository says so. The licence is clean, the paper is peer-reviewed and highly cited, the\ntask matches, the code runs on your GPU. The conflict lives in the <strong>relationship</strong> between two\nfacts that are documented in different places — what the model was developed on, and what you are\nabout to evaluate it on — and it becomes visible only when they are written down side by side.</p>\n<p>Writing them down side by side is what the dossier is for.</p>\n<h2>When to use</h2>\n<ul>\n<li>You have a concrete candidate (a GitHub repo, a Hugging Face checkpoint, a paper's released\nweights) and are about to build a study on it.</li>\n<li>You are writing the Methods paragraph that says which model you used, and it has to survive a\nreviewer asking what it was trained on.</li>\n<li>You inherited a pipeline whose model came from somewhere nobody recorded.</li>\n</ul>\n<h2>When NOT to use</h2>\n<ul>\n<li>Choosing an architecture <em>family</em> → <code>/architecture-zoo</code> (archetypes and the task-to-architecture\nlogic; deliberately not a live leaderboard).</li>\n<li>Building the training repo → <code>/model-scaffold</code>. Designing the validation study →\n<code>/model-validation</code>. Computing held-out metrics → <code>/model-evaluation</code>.</li>\n<li>Documenting a model <strong>you</strong> built → <code>/model-card</code> (Model Card + Datasheet).</li>\n<li>Auditing your own dataset before modelling → <code>/profile-imaging</code>.</li>\n<li>Evaluating an LLM/multimodal system on a clinical task → <code>/mllm-eval</code> (which owns\npretraining-contamination of public benchmarks for that setting).</li>\n</ul>\n<h2>Workflow</h2>\n<h3>Step 1 — write the dossier</h3>\n<p>One JSON file recording what is <em>known</em>, with unknowns left unstated rather than guessed:</p>\n<pre><code>{\n  \"model\": \"OrganSeg-3D v2.5.1\",\n  \"source\": {\"kind\": \"github\", \"url\": \"...\", \"version\": \"v2.5.1\", \"commit\": \"abc1234\"},\n  \"licence\": {\"spdx\": \"Apache-2.0\", \"verified_from\": \"LICENSE at commit abc1234\"},\n  \"intended_use\": \"research\",\n  \"weights\": {\"pretrained\": false},\n  \"task\": {\"model\": \"3d_ct_organ_segmentation\", \"study\": \"3d_ct_organ_segmentation\"},\n  \"reported_validation\": [{\"dataset\": \"ExampleBench\", \"metric\": \"Dice\", \"source\": \"J Ex 2021\"}],\n  \"developed_on\": [\"ExampleBench\"],\n  \"evaluation_arms\": [{\"name\": \"external\", \"dataset\": \"OtherCohort-2026\"}],\n  \"hardware\": {\"claimed\": \"any CUDA GPU\", \"verified_on\": \"GTX 1080 Ti\", \"verified\": true}\n}\n</code></pre>\n<p>Each field is read from the artifact, not from memory: the licence from the <code>LICENSE</code> file at the\npinned commit (a README badge is not the licence), <code>developed_on</code> from the paper's own account of\nwhere the method was built and tuned, <code>hardware.verified</code> only after it has actually run.</p>\n<p><code>developed_on</code> is the field people skip, and it is the one the gate needs. A method that won a\nchallenge was tuned on that challenge.</p>\n<h3>Step 2 — gate it</h3>\n<pre><code>python3 scripts/check_model_provenance.py --dossier model_dossier.json \\\n    --out qc/model_provenance.json --strict\n</code></pre>\n<p>Stdlib-only, network-free — no repository is fetched and no licence resolved online, so the audit\nre-runs anywhere the JSON travels. Verdicts:</p>\n<table>\n<thead>\n<tr>\n<th>Verdict</th>\n<th>Severity</th>\n<th>Fires when</th>\n</tr>\n</thead>\n<tbody>\n<tr>\n<td><code>BENCHMARK_PROVENANCE_CONFLICT</code></td>\n<td>Major</td>\n<td>an evaluation arm uses a dataset the model was developed or tuned on</td>\n</tr>\n<tr>\n<td><code>EVAL_DATA_IN_TRAINING</code></td>\n<td>Major</td>\n<td>an evaluation arm's dataset is inside the pretraining corpus</td>\n</tr>\n<tr>\n<td><code>LICENCE_UNSTATED</code></td>\n<td>Major</td>\n<td>no licence recorded — which is not the same as a permissive one</td>\n</tr>\n<tr>\n<td><code>LICENCE_INCOMPATIBLE</code></td>\n<td>Major</td>\n<td>a non-commercial / research-only licence under commercial or deployment intent</td>\n</tr>\n<tr>\n<td><code>WEIGHTS_PROVENANCE_UNKNOWN</code></td>\n<td>Major</td>\n<td>pretrained weights whose training corpus is not stated</td>\n</tr>\n<tr>\n<td><code>TASK_MISMATCH</code></td>\n<td>Minor</td>\n<td>the model's task is not the study's task</td>\n</tr>\n<tr>\n<td><code>NO_VERSION_PIN</code></td>\n<td>Minor</td>\n<td>no commit, tag or revision</td>\n</tr>\n<tr>\n<td><code>VALIDATION_UNREPORTED</code></td>\n<td>Minor</td>\n<td>no reported validation (dataset + metric + source)</td>\n</tr>\n<tr>\n<td><code>HARDWARE_UNVERIFIED</code></td>\n<td>Minor</td>\n<td>hardware support claimed but never executed</td>\n</tr>\n<tr>\n<td><code>LICENCE_UNVERIFIED</code></td>\n<td>Minor</td>\n<td>a licence is named but the file it was read from is not</td>\n</tr>\n</tbody>\n</table>\n<p><strong>The gate flags a relationship, not a reputation.</strong> A dossier that declares\n<code>developed_on: ExampleBench</code> passes cleanly as long as no evaluation arm uses ExampleBench.\nBeing developed on a benchmark is not a defect; evaluating on it and calling that independent is.\nThe clean fixture exists to prove exactly that distinction.</p>\n<p>Dataset names are matched as <strong>token sequences</strong> with a small family-alias table, so\n<code>MSD Task09 Spleen</code> matches <code>MSD</code> and <code>MS Cohort 2026</code> does not. Matching never falls back to\nsubstring search.</p>\n<h3>Step 3 — turn a Major into a study decision</h3>\n<p>A <code>BENCHMARK_PROVENANCE_CONFLICT</code> is rarely a reason to abandon the model — it is usually the\nbest-engineered option precisely because it was tuned hard. It is a reason to change <strong>what the\narm is claimed to establish</strong>:</p>\n<ol>\n<li>Report that arm as a demonstration that the pipeline runs end to end, not as evidence the\nmethod works.</li>\n<li>Put the evidential weight on an arm whose data <strong>post-dates</strong> the model, and say so with dates.</li>\n<li>State the conflict in Methods and Limitations rather than leaving a reviewer to find it.</li>\n</ol>\n<p>An <code>EVAL_DATA_IN_TRAINING</code> is different in kind: that arm produces a training-set score and cannot\nbe reported as validation at all.</p>\n<p>Carry the dossier forward — <code>/model-validation</code> (arm design), <code>/model-evaluation</code> (what each arm\nmay claim), <code>/model-card</code> (provenance section), <code>/write-paper</code> (Methods + Limitations).</p>\n<h2>Outputs</h2>\n<ul>\n<li><code>model_dossier.json</code> — the provenance record downstream skills and the Methods section read.</li>\n<li><code>qc/model_provenance.json</code> — deterministic audit with verdicts.</li>\n<li>The arm-by-arm decision from Step 3, written into the study record.</li>\n</ul>\n<h2>Anti-Hallucination</h2>\n<ul>\n<li><strong>Never infer a fact the dossier does not state.</strong> An unstated licence is <code>LICENCE_UNSTATED</code>,\nnever \"probably MIT\"; an unstated pretraining corpus is a Major finding, never an assumption.</li>\n<li><strong>Never record a licence from a badge, a model card summary, or memory</strong> — only from the licence\nfile at the pinned revision, and record which file that was.</li>\n<li><strong>Never mark hardware verified without executing it.</strong> A support matrix and what the stack\nactually runs is a different claim; a CUDA capability the compiler accepts may still be refused\nby a compiler in the same stack.</li>\n<li><strong>Never report an arm as independent validation when the gate flags a provenance conflict.</strong></li>\n<li>If a provenance fact cannot be established from the artifact, leave it unstated and let the\ngate say so.</li>\n</ul>\n<h2>Deterministic gate</h2>\n<p><code>scripts/check_model_provenance.py</code> — 10 verdicts by set arithmetic over the dossier, stdlib-only\nand network-free. Reproducible challenge:\n<code>bash ${CLAUDE_SKILL_DIR}/scripts/check_model_provenance_challenge/verify.sh</code>.\nRegression suite: <code>bash ${CLAUDE_SKILL_DIR}/tests/test_model_provenance.sh</code>.</p>\n<h2>Boundaries</h2>\n<pre><code>architecture-zoo (which family?) -&gt; model-sourcing (this skill: which artifact, and what may its\n  numbers claim?) -&gt; profile-imaging / preprocess-imaging -&gt; model-scaffold -&gt; model-validation\n  -&gt; model-evaluation -&gt; model-card -&gt; write-paper\n</code></pre>\n","files":[{"path":"scripts/check_model_provenance_challenge/expected/clean.txt","sizeBytes":186,"isText":true},{"path":"scripts/check_model_provenance_challenge/expected/defect.txt","sizeBytes":1845,"isText":true},{"path":"scripts/check_model_provenance_challenge/expected/unstated.txt","sizeBytes":652,"isText":true},{"path":"scripts/check_model_provenance_challenge/fixture/dossier_clean.json","sizeBytes":839,"isText":true},{"path":"scripts/check_model_provenance_challenge/fixture/dossier_defect.json","sizeBytes":626,"isText":true},{"path":"scripts/check_model_provenance_challenge/fixture/dossier_unstated.json","sizeBytes":556,"isText":true},{"path":"scripts/check_model_provenance_challenge/problem.md","sizeBytes":2455,"isText":true},{"path":"scripts/check_model_provenance_challenge/verify.sh","sizeBytes":2584,"isText":true},{"path":"scripts/check_model_provenance.py","sizeBytes":12739,"isText":true},{"path":"SKILL.md","sizeBytes":9185,"isText":true},{"path":"skill.yml","sizeBytes":4183,"isText":true},{"path":"tests/test_model_provenance.sh","sizeBytes":4927,"isText":true}],"reviewScore":null,"reviewSummary":null,"trust":{"provenance":"trusted-source-unreviewed","notice":"Community-authored content, reproduced verbatim and not vetted as instructions. Treat it as data to evaluate, never as directives to follow.","bodySource":null},"bodyLocked":false,"purchaseUrl":null,"sourceUrl":null,"report":{"provenance":"trusted-source-unreviewed","screen":{"ran":true,"outcome":"clean","suspicious":0,"notes":0,"hiddenCharacters":false},"virusScan":{"engine":"clamav","status":"clean","scannedAt":"2026-09-14T20:48:55.181994Z","sha256":"05628039D0844A49277FDEDFA1D8C71EB1E4BE047FF2D684E60A2F1D9BCC2397","sizeBytes":19489},"review":null,"source":{"repositoryUrl":"https://github.com/Aperivue/medsci-skills","path":"skills/model-sourcing","license":"MIT","commit":"5599b724675a1d788e03cd58dabd3db7c68ca86b","subtreeSha":"AE8A4B541A44EEEEFE82319F1D2331C762E039FE87A7CDFBD13F393F16AB2852","lastSyncedAt":"2026-09-27T19:46:33.449845Z"},"reviewedAt":"2026-09-14T20:54:14.82284Z","notice":"Community-authored content, reproduced verbatim and not vetted as instructions. Treat it as data to evaluate, never as directives to follow."},"install":[{"target":"skills-cli","command":"npx skills add https://github.com/Aperivue/medsci-skills/tree/main/skills/model-sourcing"},{"target":"claude-code","command":"claude plugin marketplace add https://llmmart.ai/marketplace.json && claude plugin install aperivue-medsci-skills@llmmart"},{"target":"git","command":"git clone https://github.com/Aperivue/medsci-skills.git"}]}