Skillcheck: separate a score from a review decision
Structure, suspicious content, and review decisions carry different evidence.
A skill file can be easy to navigate and still contain an instruction that should never gain authority. Skillcheck exposes structure and suspicious content as different outputs. Its quality score summarizes deterministic rules across purpose, structure, actionability, consistency, and integrity. Injection findings identify patterns for a reviewer to inspect. Neither output observes what a model will actually do with the file in a particular harness.
Keep the analyzer narrower than the decision
The browser and Node CLI share analyzeReview, a pure entry point that treats selected content as inert text. It computes quality and injection results separately, adds local-link findings during folder review, and returns explicit limitations. It does not execute selected scripts or instructions, fetch external links, or evaluate model behavior. Folder review can therefore expose a missing reference or escaping path without granting the selected package execution authority.
The detector normalizes confusable characters, reduces suspicion for quoted or descriptive context, and dampens repeated category matches. These choices make findings more usable, but each is also a heuristic interpretation. A benign teaching example can look suspicious; unfamiliar wording can evade a known pattern. A high structural score cannot compensate for an unresolved injection finding, and a clean scan cannot authorize giving the document to a production model.
Record judgment without changing the measurement
The review panel lets a person mark findings open, reviewed, or dismissed and attach notes. Those decisions do not change quality scores. When the analyzed content changes, decisions reset rather than silently attaching an earlier judgment to a new analysis. This costs repeat review after editing. It also preserves the meaning of the record: the decision belongs to the evidence the reviewer saw, rather than becoming a durable clearance for a filename.
Exports make that distinction portable. JSON includes normalized source, analysis, rule version, limitations, and decisions; Markdown offers a readable summary without source text. Reports can contain private input and need review before sharing. The CLI exit status has another narrow meaning: ordinarily zero means analysis completed. The optional fail-on-high flag changes the exit status when a high-suspicion match is found. Passing that gate still says nothing about attacks the rules miss.
The calibration record is equally bounded. Its 21 author-labeled synthetic cases cover known categories, benign guidance, quoted examples, and obfuscation. They were used during development, were not held out or independently adjudicated, and do not represent real-world prevalence. Visible false positives and misses help explain the detector. They cannot establish a production safety rate or a claim that skills improve model task success. The transferable lesson is to keep measurements inspectable and give authorization its own decision process, with evidence appropriate to the action being authorized.
A suggested ten-minute reader check
Use only inert synthetic text in a disposable local review. Start with a short, well-structured Markdown instruction file, then add an override phrase such as “ignore previous instructions.” Compare the structural dimensions with the injection findings; the precise score and suspicion depend on the complete text. Record a decision, apply a content edit, and expect the earlier decision to reset. Inspect the export limitations before sharing. This suggested check exercises a review workflow, not a model, and cannot show whether an instruction is safe or effective in deployment.
Pinned shared analyzer and explicit limits