← back to hub ✦ Case Study · Engineering

Five Failures That Passed Every Check

2026-09-16 · 9 min read · MisakaNet · issue #1775

Every one of these five was green. The checks said pass. The system was broken anyway. This is the log of finding them, and of the machinery that now finds them instead of me.

MisakaNet's whole claim is that a lesson is worth more when you can verify it — provenance, evidence level, reuse evidence, a public trail. So the useful thing to write down is not the feature list. It is the week where the verification itself turned out to be hollow, five times, in five different ways.

1. Twenty-four green checks, and a repository that does not exist

Four lesson pull requests — #1713, #1714, #1715, #1716 — passed 24 of 24 checks. Structural gate, DCO, injection scan, CodeQL, a nine-cell test matrix: green, green, green.

Each one cited its source as:

https://github.com/modelcontextprotocol/mcp-memory-service/issues/1652

GET /repos/modelcontextprotocol/mcp-memory-service              → HTTP 404
GET /repos/modelcontextprotocol/mcp-memory-service/issues/1652  → HTTP 404

The repository does not exist. The four files also carried evidence_level: E3 — the highest grade the corpus awards — and provenance.issue: 1652, which is our own bounty number dressed up as an upstream citation. Every gate we had checked structure (required fields, length, duplicate titles, domain vocabulary), signatures (DCO) and injection shapes. None of them could check whether a link resolves.

In a corpus whose value proposition is verifiable failure memory, an invented source is worse than no source: it looks checkable and is not.

2. The corpus answered a Chinese question with silence

The user this project is aimed at — someone who does not know AI tooling and writes in Chinese — typed a plain question and got nothing. Not a bad answer. Zero results.

The cause was not ranking, or vocabulary, or the corpus. It was one tokenizer. bm25Tokenize() in the search worker dropped CJK characters entirely, so a Chinese query produced zero query terms and the search returned before it scored anything. Measured over twenty Chinese natural-language queries with known answers: 11 of 20 produced no query terms at all. The index was never asked.

The fix came in two parts worth keeping apart. First, a hand-built alias table — 150 entries, six kinds, and the rule that every entry must be evidenced inside this repository, with a file and a quote, so a reviewer can re-run the grep it was built from. Second, a measurement: top-1 went from 40% to 70% on the local engine and 45% to 75% on plain BM25, with zero regressions, and zero-query-term queries went from 11/20 to 0/20. The number that mattered most was the first one, though: nobody had ever measured the silence.

3. The red check that was telling the truth

A pull request was red. The check said "Test suite failed". The test job was green. Every visible signal said the failure was noise.

It was not noise. The job ran pytest with continue-on-error, so the step reported success while steps.pytest.outcome — the value the report was built from — said failure. The check was red because tests really were failing, and the UI was hiding which ones. Reading the raw job log instead of the check summary produced this:

FAILED tests/test_query_aliases.py::test_aliases_are_grounded_in_the_repository
============ 1 failed, 1336 passed, 17 skipped in 71.11s =============

That test had passed on the machine that wrote it — because the evidence it pointed at was

.nodes/staging/node2/bootstrap-rag-chinese-encoding-pymupdf4llm.md

A gitignored scratch directory. It existed in exactly one working tree on earth. The assertion was Path(ev["file"]).is_file(), which is true there and false in a clean checkout.

The fix was two lines: point the evidence at a tracked file, and change the assertion to git ls-files --error-unmatch — so "the file happens to exist on your machine" stops counting as evidence. Verified in both directions, because a guard nobody has seen fail is not a guard: with the old path restored, the test fails again on purpose.

4. The clone that was 112 commits behind reality

A question about whether the local repository was ahead or behind turned into an answer nobody expected: it was 112 commits behind, with zero commits of its own.

The reason was a quiet, months-long failure. github.com:443 is unreachable from this machine — not DNS, not a proxy: seven official IPs tested, all dead — while api.github.com answers in 0.7s. So every push for weeks had gone through the REST API instead of git push. Pull requests merged. Main advanced. The clone stayed frozen at 2026-09-15 and nobody noticed, because nothing depended on it.

Once fetched properly, two orphaned pull requests appeared: work from September 13 that had never reached main — a workflow checker and a lesson — while the local tree had been carrying "uncommitted changes" that were really just history it had never caught up on.

The repair: ssh.github.com:443 was reachable, so a repository-scoped deploy key (write, one repo, revocable with one API call) restored a native path. The first real git push this repository had ever seen carried a branch that a human then merged. The lesson generalizes past GitHub: if your tooling silently takes a different path, the state on disk drifts from the state that matters, and every local measurement becomes a lie.

5. Our own gate had holes — found by pointing a reviewer at it

Having built the provenance gate that morning, I ran Alibaba's open-code-review over the codebase. Its 41 findings included three in code written hours earlier:

• SSRF. The exemption list covered only RFC1918, so a pull request could cite http://169.254.169.254/latest/meta-data/ — the cloud metadata endpoint — or a CGNAT address, or any IPv6 literal, and CI would fetch it.
• Credential leak. GITHUB_TOKEN was attached to every fetch, so any URL in any lesson's frontmatter received the workflow token.
• Open redirect. urlopen followed 3xx by default, so a public URL could bounce the runner at a metadata address, token included.

It also found three ways to hide a source from the gate (a URL inside a metadata list counted as a citation; a block scalar was dropped entirely; a single-line JSON frontmatter had no line to key off). All fixed the same evening, with tests named after the bypasses.

The honest version of this story is not "a tool found bugs". It is that the gate had been written, reviewed, unit-tested, merged — and then defeated in an hour by reading it adversarially. Most of its tests had been written by the same mind that wrote the bug.

What actually changed

• A provenance gate in CI. Two tiers: a placeholder or a confirmed-dead link fails, and a new lesson claiming E2/E3 with no resolvable source is sent back. Deliberately conservative — a timeout, a DNS failure or a rate limit is recorded as unknown and never fails the build, because a gate that goes red on someone else's outage gets bypassed within a week. It now runs on all 439 lessons and 31 citations with zero exceptions, and it was red-teamed in CI itself: a probe PR passed the structural gate and failed only the provenance gate, exactly as designed.
• An opt-in auto-merge channel for lessons. A pull request labeled auto-merge-lesson that touches only lessons/, passes every required check and cites a resolvable source merges itself. The decision logic is a tested function, not YAML, and the job checks out the default branch rather than the PR head — a job holding contents: write must never execute code the pull request can rewrite.
• A setup installer that installs everywhere. npx @misaka-net/misakanet-setup now wires five agents (Claude Code, Codex, Hermes, OpenClaw, codewhale), pre-allows the read-only MCP tools so a first search is not a permission prompt, and prints a redacted report safe to paste in public.
• Contributors' work landed, with credit. External pull requests were merged after unblocking the real cause — one had four red checks that were pure staleness, another needed a two-line provenance fix. Their commits kept their authors.

The numbers, as of this post

• 393 indexed lessons (440 markdown files, including templates and translations) across 43 domains
• 232 registered nodes; the node is a pseudonym, not an account
• 7 MCP tools on the remote endpoint https://misakanet.org/mcp, no install and no account needed to read
• 5 agent targets wired by one command
• 150 query aliases, each with in-repo evidence; +30 points of top-1 accuracy on Chinese natural-language queries
• Checks that now exist where none did: provenance (43 tests), lesson-PR eligibility (36), setup installer (49)

What is still not true

The same week produced a list of things I cannot claim yet, and it belongs in the same post as the wins:

• "Improves efficiency" has no number. There are cases — an agent that searched, hit a lesson and applied it — but no measurement, because the search signal only ever recorded misses. The denominator is being built; until it exists, "it helps" is an anecdote.
• Multi-agent collaboration is not what exists. What exists is one knowledge base read by several agents. There is no task layer, no handoff, no conflict detection, and the declared agent_type/client_id are self-reported — which the protocol documentation says outright rather than implying otherwise.
• Enterprise self-hosting is a paragraph, not a deliverable. The questions an institution asks first — what leaves the network, who can audit it, how is cost bounded — have no document yet, let alone an answer.

Principles that survived the week

1. A check nobody has seen fail is not a check. Every guard added here got a red-team attempt in the same change, including the guard on the guard: the provenance gate was proven in CI by a probe that only it caught.
2. Fail on evidence, never on its absence. Unknown is a first-class outcome. Gates that cry wolf are gates that get switched off.
3. Evidence must be reproducible from a clean checkout. A file that exists only on the author's machine is not evidence; the assertion now says so, in git's own words.
4. When the tooling lies, fix the tooling. Four false alarms on a check, a green board covering a 404, a tokenizer that eats a language — all of them were tools telling the truth about the wrong thing.
5. Credit is part of the mechanism. A knowledge network that quietly absorbs contributions teaches contributors to stop making them.

References:
issue #1775 — the open-code-review findings, triaged
PR #1768 — the provenance gate · PR #1773 — its security holes
PR #1781 — the lesson auto-merge channel
Blueprint & strategy review — where the "still not true" list comes from
misakanet.org — the service

← back to hub