How well does this work?

Every quality claim on this site has a number behind it, produced by a script in the public repository and reproducible by anyone. This page reports those numbers, including the ones that are poor.

What is measured, and what is not

This tool finds documents. It writes nothing: there is no model in the request path composing sentences, so the usual measures of a question-answering system — whether the answer is grounded in the source, whether it hallucinates, whether its citations are real — have nothing to attach to here. They are not omitted because they are inconvenient; they do not apply.

What is measured instead:

• Retrieval. Given a question with known correct cases, does the tool put one of them in front of you?
• Contribution. Each part of the retriever, measured on its own, so that no component is kept on faith.
• Filter integrity. When you narrow a search, nothing you excluded may come back. This is pass/fail, not a score.
• Robustness. Hostile, malformed, and impossible queries must not break the service or invent a match.

Is the underlying data correct?

Every score below assumes the record is faithful to the court’s PDF. If the dates were wrong, a date filter would quietly lie and no retrieval measurement would notice. So a random sample is audited field by field against the original document, and cheap invariants run over the whole collection.

120 documents, drawn in proportion to the collection’s courts and document kinds, with all 120 source PDFs re-downloaded and re-extracted independently of the pipeline. A field that the PDF can neither confirm nor contradict is counted as not checkable rather than scored either way.

Invariants across all 68,299 documents:

• No dates outside the collection's range
• No documents with no chunks
• No chunks orphaned from their document
• No chunks with no embedding
• No counties outside Michigan's 83
• No documents with no source URL
• No relative source URLs
• No multi-page opinions under 100 words
• 1 bodies far shorter than their page count implies — worth a look: usually a scanned exhibit with no text layer, occasionally a failed extraction. The one current match is a Supreme Court opinion whose last 20 pages are a scanned exhibit; all 896 extractable words were captured.

The question set

569 questions, each written against a document already in the collection and each carrying the specific cases that count as correct. They are grouped by the kind of search they represent, because a name lookup and a doctrinal description are different tasks and averaging them hides both.

229 of 569 questions have been verified; 340 are drafts. Verified questions are name and docket lookups, checked mechanically against the document itself. The 340 drafts were drafted from the text of the decisions and are awaiting review by a law student against the sources. Numbers from draft questions are marked throughout and should be read as a signal, not a result.

How well it finds cases

Each question was sent to this site’s own search, exactly as you would type it, and scored by whether a correct case came back. Ranges are 95% confidence intervals over questions.

Measured 2026-09-01 against 68,299 documents and 479,604 passages (ranking function f54825880523). The collection has grown since; a number is only meaningful with the index it was measured on, which is why the fingerprint is recorded rather than the date alone.

Two results deserve to be read plainly. Finding several cases on one topic scores 22.3% — this tool is built to find a case, and it is weak when you want the full set. Searching by statute or court rule scores 43.2%, measured as the share of the top ten that are genuinely on that provision, so a partial credit score of 0.44 means roughly four of ten results are on point rather than that the search failed.

Are some kinds of document harder to find?

The table above slices by the kind of question. That leaves a blind spot: an entire class of documents could be unreachable and none of those numbers would move. Regrouping the same results by the document that should have been found closes it.

No cohort collapses. The first run of this cut found something more useful than a gap: two large parts of the collection that nothing tested — 262 documents read by OCR, covered by a single question, and 10,153 documents under 100 words, covered by three. 109 questions were written for them, and both cohorts are now measured rather than absent.

A cohort tested only by name and number lookups is not a cohort shown to be easy. The scanned documents and the very short ones score at or near 100%, and both are covered almost entirely by questions asking for a docket number or a case caption — the kind whose answer the document itself settles. It means their identifiers resolve, not that a reader could find them from a description of what they decided. Doctrinal questions for those cohorts have to be written by someone who has read them.

Does the same search give the same answer?

The vector index is approximate, and a broad filter deliberately trades exactness for speed. Nothing in that design guarantees the same query returns the same cases twice — and someone who searches, reads three cases, then searches again to find the fourth will treat a reshuffled list as their own mistake.

37 of 40 queries returned an identical top ten across 5 consecutive runs. Averaged over every pair of runs, 98.3% of the top ten was shared, and the position of the correct case never moved by more than 1 place. The approximation is visible in the tail of the list, not where it would mislead a reader.

Asked a different way: of 34 questions the tool answers correctly, 33 still find the same case under every reworded version, and 1 do not. The rewordings are generated by a language model and are unreviewed — they are test input, never text shown on this site, and a rewording that quietly changed the legal question would look like a retrieval failure here.

What each part of the retriever contributes

A search runs several methods at once and merges them: vector search for meaning, full-text search for exact words, a case-name match, and an exact docket lookup. Keeping a component because it seems principled is not evidence. Each is measured alone, and against the whole, on the same questions.

460 questions, drafts included. A difference whose p-value is above 0.05 is not a difference, however tidy the ordering looks.

Which embedding model

The vector arm depends on a model that turns text into coordinates. The one in production was chosen because it was available, not because it was measured. A benchmark that runs on one laptop, with no paid API, compares seven of them over the same passages and questions so that the choice can be made on evidence and re-made by anyone.

This measures the vector arm in isolation, on draft questions. It is not a prediction of what the site would score with a different model, because the site also runs full-text and name matching that cover for exactly the weaknesses this test exposes. Docket-number search, for instance, scores near zero for every model here and 93.3% in production, because a number is found by matching it, not by understanding it.

Filter integrity

When you limit a search to one court, one year, or one county, a document from outside that limit must never appear. This is checked as pass/fail rather than scored, and every returned document is verified against the database directly — not against the labels the interface printed, which would pass even if the filter were ignored.

84 of 84 checks passed. No document escaped a filter, no impossible filter combination fell back to an unfiltered search, and every question whose answer lay outside its filter correctly failed to return it.

Adversarial probes

Because nothing here generates text, most prompt-injection attacks have no target: “ignore previous instructions” is only ever a string of search terms. Saying that is not the same as testing it. The probes cover injected instructions and markup, SQL, malformed boolean operators, invented case names, nonsense, control characters, and questions the collection cannot answer.

17 robust, 3 partial, 1 failed, of 21 probes.

The “partial” results are searches for cases that do not exist — an invented party name, a decision dated after the collection ends. The tool returns topically similar real cases rather than nothing. That is defensible behaviour for a search box, and it is listed as partial rather than robust because a reader in a hurry could mistake a plausible neighbour for the case they named.

Accessibility

Someone using a screen reader, a keyboard alone, or a magnified display has the same claim on finding a case as anyone else. Every page is audited with axe-core in both colour themes and at a phone width, including two states a crawler never reaches on its own: a search with results, and the filter panel open.

No machine-detectable violations across 22 page, theme and viewport combinations. An automated audit finds perhaps a third of what a person using assistive technology would. A clean result means no detectable violations, not that the site is accessible; a screen-reader pass by a person is still owed and has not happened.

The first run was not clean. Links inside paragraphs were distinguished from the text around them by colour alone, the scrolling diagrams could not be reached by keyboard, the results page had no level-one heading, and there was no skip link, so a keyboard user tabbed through the whole header on every page. All four are fixed.

Known weaknesses and open work

• Most of the question set is unreviewed. 340 questions are drafts awaiting verification against the sources. Until that review happens, the numbers above are a signal about the system, not a finding about it.
• Finding every case on a topic is weak. The tool is built to surface a relevant case. Exhaustive retrieval — what a practitioner actually needs before filing — is measurably worse and is not solved.
• Name matches currently outrank everything. A search whose words happen to resemble a case caption puts those cases ahead of every passage in the collection, which is right for a name lookup and wrong for a question. The ablation above measures the fix.
• A null byte in a query returned a server error until it was found by the probes here and fixed. It is listed because a page that only reports its successes is not an evaluation.

Reproducing this

Every script, question, and raw result is in the repository. The model comparison needs no API keys and no database — it runs against a fixed subset with a local model server, so the cost is electricity.

python eval/run_gold.py              # how well it finds cases
python eval/run_arms.py              # what each component contributes
python eval/run_filter_integrity.py  # filters must not leak
python eval/run_adversarial.py       # hostile and malformed queries
python bench/build_subset.py         # the offline model benchmark
python bench/embed_local.py --model all
python bench/score.py --models all

Raw per-question results, including every miss, are in eval/results/ and bench/results/.