Product teardown

Should My Agent Answer This?

When an AI assistant cannot find an answer in your documentation, it declines, somebody complains, and you learn something is missing. When it finds an article that is nearly right, it answers confidently and wrong, the metric records a success, and nobody ever tells you. This is a pipeline that goes looking for the second case on purpose, and this is the thinking behind it.

10% to 66% How much more often a model answers wrongly when it is handed partial context instead of none. Measured by Google Research
14 Near misses: an article people ask about that leans on something no page in the documentation set explains
11 of 14 Held up when three independent readers each went back to the pages, unanimously. With only fourteen items the true rate could sit anywhere from about half to about nine in ten
No difference Graph retrieval against ordinary similarity search, over 300 real questions. The clever version did not win, and that is in the hero rather than buried

Thirty seconds

TL;DR

Too long; didn't read. The whole page in five sentences.

When an AI assistant answers from documentation that is nearly right, it answers confidently and wrong, and the metric counts a success. I built a pipeline that maps a documentation set, crosses it with questions users actually asked, and names the articles most likely to fail that way and the fact each is missing. On one project's documentation it found fourteen gaps, and three independent readers confirmed eleven. Using the same map to pick what the assistant reads did not beat ordinary search. Each finding is a work item for the content owner, not a score for the system owner.

Three minutes

Summary

  • The problem. Three things can happen when an assistant is asked a question: it answers, it declines, or it finds an article that is nearly right and answers confidently and wrong. Every signal a content team has (complaints, escalations, empty searches, the deflection rate) surfaces the decline and hides the confident wrong answer, because the person who was misled left satisfied. Google Research measured the danger: handed no context, models are wrong about 10% of the time; handed insufficient context, about 66% (Part 1, Part 2).
  • The thesis. Since no signal arrives, the signal has to be manufactured from the documentation itself, before deployment. A map of the documentation as boxes and arrows, one box per section and per idea, an arrow wherever a section explains an idea or quietly assumes it, can express an absence: an idea that sections depend on, that users ask about, and that no section explains. Ordinary similarity search cannot express that, because it always returns the nearest piece however far away it is (Part 3).
  • The method. 60 pages of the FastAPI documentation were mapped, with no arrow allowed to exist without the verbatim sentence that justifies it, and crossed with 300 answered questions from the project's own discussion board, whose accepted answers served as the known-correct answers. Every one of the fourteen near misses went to three independent readers told to search rather than judge from memory, and the bar was set in advance: none of the 60 pages explains the idea, and the thread's answer supplies a fact the pages lack. The same map was then tried a second way, choosing what the assistant reads, against ordinary similarity search over the same 300 questions, which is the control (Part 3, Part 5, Part 6).
  • The results. 153 findings in all, 14 of them near misses. Eleven of the fourteen held up, unanimously; measured against all 155 pages of FastAPI's documentation rather than the 60 indexed, eight did, because three are documented on pages outside the index. As a way of choosing what the assistant reads, the map did not beat similarity search: 23 correct answers against 24, 3 confidently wrong against 5, on a comparison that could only have detected a much larger difference (Part 4, Part 5, Part 6).
  • What it means. The map earns its cost as a diagnosis and has not earned it as a retriever. The null on retrieval has a known cause: 282 of the 300 questions came from people who said they had already read the documentation and not found the answer, which makes them excellent for finding gaps and useless for judging answering. Both precision numbers are published, because a gap-finding tool is judged on its false positives, and picking the flattering one would be the same failure this project is about (Part 5, Part 6, Part 7).
  • Cautions, and how to check. Recall, the share of real gaps found, is unmeasured and cannot be measured without the hand labeling this design avoids, so fourteen is a floor. The scorer of the retrieval comparison was not blind to which path produced each answer, which is on the record and, if anything, favored the map. Only the 14 near misses were checked; the other 139 findings are labeled as unchecked in the explorer. Every number on this page is tested against the data files, and every finding carries the sentence, file and line it was read from (Part 6, Part 7).

Part 1

The three outcomes

First, what actually happens when someone asks

Everything below depends on one mechanism, so it is worth thirty seconds. When a company points an AI assistant at its documentation, the assistant does not read your documentation. It cannot: there is far too much of it to fit in a single question.

Instead your pages are chopped into small pieces ahead of time, usually a few paragraphs each. Those pieces are called chunks, and where the cuts fall matters more than it sounds. When somebody asks a question, separate software picks the handful of pieces that look closest to the question. That step is called retrieval. Only those few pieces are handed to the model, and the model writes its answer from them and nothing else.

How an AI assistant answers from your documentation Four steps. A question arrives. Retrieval picks a handful of pieces out of the many your documentation was chopped into. The model writes an answer from only those pieces, never seeing the rest. The person reads the answer. The failure this project is about happens between step two and step three, when the pieces that were picked look right and are missing the one fact the answer needs. 1 2 3 4 Someone asks a question A real user, in your help widget Retrieval picks a few pieces Your pages, chopped into pieces. The five in color are the ones that looked closest The model writes from those five only It never sees the rest of your docs The person reads it And believes it, which is the point This whole project is about one thing going wrong here
Two consequences worth holding on to. The model can only be as good as the pieces it was handed, and nobody in this chain ever checks whether those pieces actually contained the answer.

A company points an AI assistant at its documentation. Three things can happen.

It works. The answer is in the docs, retrieval finds it, the assistant answers correctly.

It finds nothing. No article is close, so the assistant declines. Annoying, but safe, and above all visible. The user complains, a ticket gets filed, a search with no results gets logged, and the content team learns that something is missing.

It finds something close but incomplete. An article looks relevant, gets retrieved, and is missing one fact the answer depends on. The assistant answers confidently and wrong. Nobody sees it, because the assistant resolved the request and the deflection metric counted it as a success. The article looks healthy forever.

That third case is easier to recognize when you watch one happen. This is a real one, produced by ordinary similarity search over the same 60 pages, which is what almost every documentation assistant does today (Part 5), and scored against the reply the asker actually got.

Discussion 13455: "Fetching a lot of time series data"

Someone reports that a large JSON response takes about half a second to build, and asks how to speed it up.

What retrieval returned

Extra Data Types / Example

Server-Sent Events / Raw Data

Additional Responses / Combining information

Stream Data / No Annotation

Server-Sent Events / Stream SSE with FastAPI

Five sections, all plausibly about returning data. None of them is about where the half second is going.

What the assistant said

"Let Pydantic serialize. Declare the return type (or response_model) instead of building the payload yourself. The docs state Pydantic serializes on the Rust side and you get much higher performance."

Fluent, specific, sourced to the retrieved text, and offered without a hedge.

What the maintainer said

That the timing figure the asker quoted is measured before the first byte arrives, so it already includes the round trip, and the server is spending that time computing the response rather than serializing it.

Different problem. Different fix.

This is the failure. Nobody filed a complaint, because the asker got an answer. Retrieval succeeded, in the sense that it returned five relevant looking sections. The assistant was not hallucinating, in the sense that everything it said is in those sections. The deflection metric recorded a success. And the advice points away from the actual cause.

The third case is not a rarer version of the second. It is substantially more dangerous, and that has been measured.

What Google Research measured

Across several models and datasets, answers were scored by whether the retrieved context was sufficient to answer the question. Given no supporting context at all, models answered incorrectly about 10% of the time, mostly by abstaining instead, which is the second outcome above. Given insufficient context, they answered incorrectly about 66% of the time.

Partial information does not make a model more cautious. It makes it confident. Sufficient Context: A New Lens on Retrieval Augmented Generation Systems, arXiv 2411.06037.

That is the whole premise. An article that is 90% of an answer is worse than an article that is 0% of one, and it is the 90% article that no process in a content team is currently looking for.

Part 2

Why nobody sees the third one

Content teams do not prioritize at random. They prioritize from signals, and there are four of them.

Every routine prioritization signal surfaces the safe failure and misses the dangerous one.
Signal Fires when the assistant declines Fires when the assistant is confidently wrong
Customer complaints Yes. The user did not get an answer and says so No. The user got an answer, believed it, and left
Escalations to a human Yes. The conversation gets handed over No. The assistant resolved it
Searches returning nothing Yes. That is exactly what the log records No. Retrieval returned five confident results
Deflection rate Counts against you, correctly Counts for you. The failure improves the metric

This is structural rather than an oversight in anyone's tooling. Every one of those signals is a report of dissatisfaction, and nobody is dissatisfied at the moment a confident wrong answer happens. The person who finds out is a customer, three days later, when the thing they were told does not work.

A signal that does not arrive cannot be collected. It has to be manufactured, before deployment, out of the documentation itself. That is the rest of this.

Part 3

What I built

A pipeline that maps a documentation set as a knowledge graph, crosses it against questions real users actually asked, and produces a ranked list of the articles most likely to make an assistant answer confidently and wrong, naming the specific missing fact in each case.

Here is the shape it hunts for, before any of the machinery. This is one real finding.

How a near miss is read off the graph Three checks on one Concept. A Question points at it by ASKS_ABOUT, present. A Section points at it by REQUIRES, present. Any Section at all pointing at it by DEFINES, absent across all 570 sections. The finding is that third, absent edge, and the first two are what make the absence matter. QUESTIONS 6 threads asked about this SECTION Stream Data / Files and Async CONCEPT event loop ASKS_ABOUT REQUIRES NO SECTION, OUT OF 570 nothing DEFINES it DEFINES absent
A near miss is one absence, made to matter by two things that are present. Take away the question and it is a gap nobody has hit yet. Take away the dependency and it is a word the documentation simply does not use. Both of the arrows that are present carry the exact sentence they were read off, and the file and line where it lives.

The corpus, and why this one

60 pages of the FastAPI documentation, taken in the documentation's own reading order: all 51 tutorial pages, then the first 9 advanced pages, pinned to one commit.

The corpus is not the interesting decision. The demand signal is, meaning the evidence that somebody actually wanted the missing thing. The same repository runs a GitHub Discussions board with roughly 3,900 answered questions, so one source supplies the documentation, the questions people actually asked about it, and the correct answers, written by people who know. That last part is what removes the need to hand label anything.

The decision the whole project rests on

If a question was answered in a discussion and that answer is not in the documentation, then the gap is real and the missing content already exists in somebody's reply. That is the project's ground truth, meaning the known-correct answer everything else gets scored against. Free, verifiable by anyone, and no annotation budget.

The cost is that it only sees gaps somebody bothered to ask about in public, which is a narrower set than a real team's search log.

The graph

Five node types and seven edge types. Deliberately small: this is a schema designed around one question, not a general ontology.

The graph schema: five node types and seven edge types Documentation on the left: a Doc contains Sections, and a Section links back to a Doc. Demand on the right: a Question is answered by an Answer. Both sides meet at the Concept in the middle. Sections reach Concepts by three different edges, DEFINES, REQUIRES and MENTIONS, and Questions reach the same Concepts by ASKS_ABOUT. The full counts are in the table below. THE DOCUMENTATION WHAT PEOPLE ASKED Doc 60 Section 570 Concept 967 where the two sides meet Question 300 Answer 300 CONTAINS 570 LINKS_TO 50 DEFINES 914 REQUIRES 1,611 MENTIONS 994 ASKS_ABOUT 960 ANSWERED_BY 300
The whole schema. The three teal edges are the ones every finding is made of, and the Concept in the middle is the only place the documentation and the questions touch. Counts are from data/manifest.json.
The three edges that matter are DEFINES, REQUIRES and ASKS_ABOUT. Everything else is scaffolding around them.
Edge From, to What it claims Count
DEFINES Section, Concept This section explains what the thing is or how to use it. A reader who knew nothing could learn it here 914
REQUIRES Section, Concept This section's instructions assume the reader already knows the thing. Following them without it would fail 1,611
MENTIONS Section, Concept Named in passing, neither explained nor depended upon 994
ASKS_ABOUT Question, Concept Answering this question requires understanding this concept 960
CONTAINS Doc, Section Structural 570
ANSWERED_BY Question, Answer GitHub marked this reply as the accepted answer 300
LINKS_TO Section, Doc An internal documentation link 50

48 times the extractor produced two relations for the same section and concept, usually "explains" alongside "assumes". Where that happens the finders treat the concept as explained and drop it, which loses findings rather than inventing them.

Why this shape rather than another

The whole design turns on the difference between explaining a thing and assuming it. Every other graph over documentation I looked at models "this page is about that thing", which cannot express the finding, because being about a thing and teaching it are different claims.

Two other shapes were available for the same job. This graph wins on what it can express and does not win on retrieval, which Part 5 measures and finds no difference on.

One of the two is what nearly everyone actually builds, so it is worth four sentences. A vector index takes each of those chunks from Part 1 and turns it into an embedding: a few hundred numbers standing for what the chunk is about, arranged so that chunks about similar things end up with similar numbers. Your question is turned into numbers the same way, and retrieval hands back whichever chunks sit closest to it. Nothing in that arrangement knows what a chunk says, only what it resembles, and that one fact is what the first column below is losing on.

Only the middle column was built and measured here. The other two are argued, and the third was rejected on its documented behavior rather than on a benchmark I ran.
A flat vector index This typed graph An untyped entity graph
What it stores One embedding per chunk, and nothing about how chunks relate Typed, evidenced relations between sections, concepts and questions Entities and one generic association, usually with a merged description per entity
Can it express an absence No. Similarity has no way to say "nothing here teaches this". The nearest chunk is always returned, however far away it is Yes. "No Section DEFINES this Concept" is a query, and it is the whole finding Partly. It can find an entity with few links, but not one that is depended on and never explained, because it has no edge type for the difference
Can you audit a result No. A similarity score is not a reason, and there is no sentence to point at Yes. Every edge carries the sentence, the file and the lines No. Merging several sources into one description destroys the spans before you can read them
Cost to build Minutes. One embedding pass, about 80MB of model, no judgment anywhere High. 79 model batches for the documentation and 30 for the questions, then a completeness sweep over every concept nothing defines, then an independent check of every finding it reports Moderate. One extraction pass, no evidence to validate, which is exactly why there is less to check
Cost to keep true Re-embed the changed pages Re-extract them, and every relation is a judgment that can be wrong in a way an embedding cannot Re-extract, with less to get wrong
Measured retrieval quality 5 confidently wrong answers over 300 questions 3 over the same 300, which the sample cannot resolve Not measured here

The graph earned its cost as a diagnosis, not as a retriever.

That is the trade in one line. A vector index is close to free and can never tell you what your documentation fails to say, because every question gets its five nearest chunks whether or not any of them contain the answer. This graph can tell you, and it cost a great deal more to build, and most of that cost was not the graph. It was the extraction and the checking that the graph makes it possible to demand, which is also what caught 24 wrong findings before anybody read them.

The one rule everything else hangs off

No edge without evidence.

Every edge carries the verbatim source span that justifies it, plus the file path and the line range where that span lives. There are no exceptions, including for the structural edges where the span is obvious. It is the difference between "the graph says so" and "here is the sentence the graph read", and without it none of the findings are actionable, because nobody acts on a finding they cannot check.

It is also enforced rather than requested. A model asked for a verbatim span will sometimes paraphrase, so no span is believed. Each one is searched for in the file its edge claims it came from, and an edge whose span is not found there is dropped, not downgraded to low confidence. The permitted normalizations are Unicode form, whitespace, quote and dash shapes, and case, and that is the complete list.

What the enforcement actually caught

Of 3,509 documentation edges proposed across 79 batches, zero were dropped for an unfindable span. That is not a sign the check was unnecessary. It is a sign the instruction worked, and it is only knowable because the check exists.

The same check on question spans dropped 13% on the first attempt, for a reason worth knowing: GitHub serves a discussion body twice, once as markdown and once as the plain text it renders to. The mapper read the plain text and the file on disk holds the markdown, so a span containing a backtick could not be found. Matching the two like with like recovered 124 of 125.

The three findings

That shape is the first of three. Each is a shape in the graph, and each carries a proof path of real arrows leading to real lines in real files.

Type The shape The proof
F1 Near miss A Concept that Questions ask about, that a Section REQUIRES, and that no Section in the corpus DEFINES The asking span, the depending span, and the empty define set with the count of sections checked. Confirmed by the answer on the thread, quoted verbatim
F2 Orphan concept A Concept with REQUIRES or MENTIONS in-degree above zero, DEFINES in-degree of zero, and at least two sections leaning on it The in-edge list and the empty define set
F3 Retrieval collision Two Sections a single Question maps to with comparable concept overlap, where only one carries the answer The competing sections, their overlap scores, and each one's similarity to what the maintainer actually replied

F3 is what the demonstration in Part 1 was. When two sections cover the same ground and only one carries the fact, retrieval has no way to prefer the right one, so the assistant answers from whichever scored higher and the failure looks like a success. The fix is usually not new writing, it is a pointer from the section that gets retrieved to the section that has the answer. It is also the commonest finding here, 99 of 153, and the least verified.

Everything is ranked by demand, meaning the number of distinct discussion threads touching the concept. A gap nobody asks about is real and is not worth writing this quarter, and a tool that cannot tell the difference produces a backlog instead of a priority.

Part 4

The fourteen it found

Over 60 pages, 570 sections, 967 concepts and 300 answered threads. 5,399 edges, every one carrying its evidence span.
Type Findings Threads reached What that means here
F1 Near miss 14 20 Two of them were asked about by more than one person, and one thread raised two of these gaps at once, so fourteen findings reach twenty threads. The rest were asked once, which is what a long tail looks like
F2 Orphan concept 40 0 Zero by construction: any orphan somebody asked about is promoted to F1, so F2 is precisely the tail nobody has asked about yet. These are near misses with the question missing. The dependency and the absence are already in place, so the only thing standing between an orphan and a confident wrong answer is somebody finally asking
F3 Retrieval collision 99 116 12 of them caught more than one question. Each one is a question, so demand is never zero

The near misses, in full

There are only fourteen, so here they all are rather than a top ten, with the fact the documentation never supplies. The three marked as not holding up were rejected by the independent check in Part 6 and are still listed, because a table that quietly drops its own failures is not evidence of anything.

Every row links to the finding in the explorer, where the proof path and the evidence spans are one click away, along with the confirming answer on the thirteen that have one.
Threads Concept What the documentation never says Held up
6 event loop how to run a coroutine on the main thread's event loop from a sync path operation function, which FastAPI runs on a threadpool worker Yes
3 APIRoute which parts of APIRoute are safe to subclass, given that overriding its constructor is not a supported extension point Yes
1 EmailStr how to declare an optional EmailStr form field that accepts an empty submission, which browsers send as an empty string Yes
1 exc.errors() when WebSocketRequestValidationError applies, which is only during the connection stage before accept Yes
1 port which layer returns a 502 with an empty body while uvicorn is already listening on the port Yes
1 session.exec which select session.exec requires, the one from sqlmodel rather than the one from sqlalchemy Yes
1 host how to check whether a server bound to 0.0.0.0 is actually reachable from a client Yes
1 jwt.decode which jwt.decode arguments handle a token carrying an aud claim, the audience value or the verify_aud option Yes
1 Literal what Enum gives predefined parameter values that typing.Literal does not, and when each one fits Yes
1 ValidationError how to return a validation style error for one body field from inside a path operation function, and what shape each entry in the detail list must take Yes
1 virtual environment which Python version a virtual environment for running FastAPI's own test suite must use Yes
1 path operation how to attach several named OpenAPI examples to a response declared on an APIRouter No
1 fastapi.encoders how to give jsonable_encoder a rule for a type it does not recognize No
1 TokenData why the tutorial declares a TokenData model rather than passing the decoded username straight through No

These are the shape a content team can act on. Not "improve the security docs", but "the OAuth2 page never says which jwt.decode argument handles an audience claim, and here is the thread where somebody had to be told". That is a work item with a reader attached to it.

Fourteen is a small number, and that is the point

What is worth saying about the fourteen is what they have in common: almost all of them are demand of one. One person asked, once.

A gap nobody has asked about twice is still a gap, and it is not this quarter's work.

That is why demand is on every finding rather than being used as a filter. Two of the fourteen have real recurring demand, and a content lead with a week to spend should spend it on those two. The other twelve are a backlog, ranked correctly below them, and three of those twelve are the ones that did not hold up. A tool that could not tell those apart would hand over 153 equally urgent tasks, which is the same as handing over nothing.

Part 5

Does the graph make a better assistant?

Everything so far uses the graph to diagnose documentation, to find the holes. There is a second and completely separate thing you can do with it: use it live, to choose which pieces the assistant reads when somebody asks a question. That is worth testing, because if it works it is a much bigger idea than a report.

So both were built and both answered all 300 questions.

Two ways of choosing five pieces of documentation for the same question. Everything after the choosing is identical, so any difference in the answers comes from the choosing.
Path How it picks the five
A, ordinary similarity search The method almost every documentation assistant uses today, and the thing to beat. Every piece is turned into numbers, the question is turned into numbers the same way, and the five pieces whose numbers sit closest to the question's come back
B, graph retrieval Start from the sections that explain what the question asks about. Then follow one arrow further: for anything those sections themselves assume, pull in the section that explains that. Three slots for the first kind, two reserved for the second

Path B's reserved slots are the whole experiment. A section that answers your question is easy to find by similarity, because it sounds like your question. The thing that section quietly assumes is not, because a prerequisite never sounds like the question that needs it. Only the graph knows the two are connected, and that connection is what is being tested.

The first version of this experiment did not test anything

It filled all five slots by similarity to the question first and appended the prerequisites afterwards, so the prerequisites were only ever reached when a question produced fewer than five starting sections. That was 70 questions of 300. On the other 213 the traversal ran and its results were thrown away before the assistant saw them.

Reserving the slots fixes it, and the mechanism now reaches the assistant on 278 questions of 300. The numbers below are from that corrected run.

What a prerequisite looks like in practice

Discussion 13828 asks why declaring the same request body on both a dependency and the path operation validates it twice.

Similarity search finds five sections that talk about request bodies. The graph finds the page on dependencies, and then, because that page leans on a class it does not itself explain, pulls in the two sections that do explain it. Those two are marked.
Path A picked Path B picked
Body Multiple Parameters / Multiple body params and query Request Body
Get Current User / Create a user model Classes as Dependencies / Shortcut
Path Operation Configuration / Response description Classes as Dependencies / Type annotation vs Depends
Body Multiple Parameters / Multiple body parameters Classes as Dependencies / Classes as dependencies
Declare Request Example Data / Body with multiple examples Classes as Dependencies / Use it

The step that fetched them, copied out of the results file

sec:tutorial/dependencies/classes-as-dependencies#shortcut REQUIRES concept:commonqueryparams -> DEFINES -> sec:tutorial/dependencies/classes-as-dependencies#use-it

Read it as: the Shortcut section assumes you know what CommonQueryParams is, the Use it section is where that is explained, so fetch it too.

Path A declined this question. Path B answered it correctly, using the section the arrow fetched. That is the mechanism working exactly as intended, and it happened often enough to look promising and not often enough to show up in the totals.

The result

Every answer was scored against the reply the asker actually got, into one of three buckets.

Bucket What it means
Correct Matches the maintainer on the substance
Confidently wrong Presents an answer that contradicts the maintainer or misses the fact the answer turns on, with no hedge. This is the failure the whole project is about
Declined Says outright it cannot answer from what it was given
300 questions, both paths, 598 of 600 answers scored. One question came back without a verdict on either path.
Path Correct Confidently wrong Declined
A, similarity search 24 5 270
B, graph retrieval 23 3 273

One disclosure before the numbers. The scorer was not blind: every block it was handed is headed SCORE ID: q:13029::vector or SCORE ID: q:13029::graph, so the prompt named the path that produced the answer, and the similarity answer came first in all 300 pairs. That is a defect in the setup, it is on the record in data/work/compare/score/in/, and if it favored anything it favored the graph.

The graph did not make the assistant more right. 23 correct against 24. Both paths answered the same 300 questions, so the informative cases are the 17 where exactly one of them got it right, and that split is 9 to 8. A coin would do that.

It did make the assistant more cautious. Confidently wrong answers went from 5 to 3, and declines went up. Following the arrows swapped nine correct answers for declines, converted three of the five confident errors into a decline or a correct answer, and picked up three new correct answers along the way.

It did not make the assistant know more. It made it admit ignorance more often.

Which is arguably the trade this whole essay argues for, since a decline is the safe failure and a confident wrong answer is the dangerous one. It is also far too small to claim. Three fewer errors out of 300 is the kind of difference that appears and disappears between runs, and the honest summary of Part 5 is that graph retrieval did not beat ordinary similarity search here on any measure that the sample can resolve.

How small a difference could this have caught

Saying an experiment is "too small" is not useful unless you say how small. Only the 17 questions where the two paths disagreed carry any information, and with 17 of those, the least lopsided split that would have counted as a real difference is 14 to 3. The observed split was 9 to 8.

So this design could only have detected an effect several times larger than anything that showed up. It has not shown that the two methods are equally good. It has shown that if there is a difference, it is small, and that this corpus cannot resolve it.

Why the sample cannot resolve it, which is the interesting part

Both paths declined about 90% of the questions, so the entire comparison rests on the 30 or so each one actually answered. That is not the retrieval. That is the question set, and the reason is the most interesting thing the project found by accident.

282 of the 300 askers had already read the documentation and said so

FastAPI's Questions form asks people to confirm it before posting, line by line: "I searched the FastAPI documentation, with the integrated search", and "I already read and followed all the tutorial in the docs and didn't find an answer." 282 of the 300 threads carry it.

Excellent for finding gaps. The demand signal is drawn from exactly the population the documentation failed.

Useless for benchmarking answerability. The sample is selected for being unanswerable from the docs, so it cannot fairly measure how well anything answers from the docs.

Which makes a 90% decline rate close to correct. An assistant that confidently answered most of these, from the same tutorial the asker had just read, would be doing the exact thing this project exists to find.

I did check whether my own instructions caused the abstaining rather than the corpus, by rerunning three batches with the line about what was being measured removed: 11 declines of 15 instead of 12 of 15, which is far too small a probe to settle it and is reported here as the weak check it is.

So where does that leave it

The graph earns its cost as a diagnosis and has not earned it as a retriever. The honest next step is not to keep tuning this until the number moves, which is how people end up believing things that are not true. It is a different question set: real search-log queries, which are not pre-filtered against the documentation, and enough of them that the answered subset is in the hundreds rather than the thirties. Until that runs, the finding stands as a null with a known cause.

Part 6

Checking every finding

Fourteen near misses is a small enough population to check all of them, so all of them were checked rather than a sample. Each one went to three independent readers, given the corpus, told to be skeptical, and told to go and search rather than judge from memory. A finding counts as valid only when no section of the corpus explains the concept and the answer on the thread supplies a fact the corpus does not contain.

Eleven of fourteen is 78.6%, and fourteen is a small number, so the interval sits beside it: the range of true rates that would not be surprising if you saw 11 of 14, here anywhere from about half to about nine in ten. Every near miss was checked, so the rate has no sampling error: it is the rate for this corpus and this run. The interval is reported anyway, as the uncertainty about the method rather than about the draw, and it is wide because the population is small.
Checked Held up Rate Wilson 95% Readers each Agreement
14 11 78.6% 0.52 to 0.92 3 Unanimous on all 14

Three did not hold up: path operation, fastapi.encoders and TokenData. They are still in the table in Part 4 and still in the explorer, marked as not holding up, because a list that quietly drops its own failures is not evidence of anything.

The path operation case is worth its own paragraph

All three readers agreed the corpus defines "path operation", and it does: First Steps gives it a titled step, then defines "Path" in one subsection and "Operation" in the next. No single heading-scoped chunk defines it, and the parent that ties the two halves together is two lines long.

The graph is section-level because retrieval is chunk-level. So the page teaches it and no retrievable piece of the page does, which is a real retrieval problem wearing the costume of a false positive. It is counted against the tool above, because that is the honest call under the rule as written.

A gap in FastAPI, or a gap in my index?

Everything above measures the findings against the 60 indexed pages. "These 60 pages never explain it" and "FastAPI never explains it" are different claims, and only the first one follows from a check run inside the corpus. A reader hears the second.

So the other 95 English documentation pages were fetched and all fourteen near misses put in front of three more readers, told to search only outside the corpus and to report either a definition with its file and line, or nothing.

The fourteen near misses against both questions Of the eleven that held up against the 60 indexed pages, eight are absent from all 155 English pages and are real gaps in FastAPI's documentation. Three are documented on a page outside the index: APIRoute, host and virtual environment. Of the three that did not hold up, one is also documented outside. Absent from all 155 pages Documented outside the 60 Held up against the 60 Did not hold up 8 real gaps in FastAPI's documentation 3 APIRoute, host, virtual environment 2 path operation, fastapi.encoders 1 TokenData
Three independent readers over the 95 pages outside the index, majority verdict, recorded in data/corpus_boundary.json. Only the top left cell is a documentation gap in the sense a reader would assume.

The top right cell is the one that costs something. APIRoute is the second most asked about finding on the page, and FastAPI documents it properly in a how-to guide that is not in the 60. So does host, and virtual environments get a page of their own. A content team handed those three would go and write something that already exists.

Eight of the fourteen are gaps. Three are my index pretending to be a gap.

This is the concrete version of the thing Part 7 states abstractly, and it cuts both ways. Measured as a claim about FastAPI, precision, meaning how many of the gaps this tool reported are real, drops from 11 of 14 to 8 of 14. Measured as a claim about an assistant that indexes only the tutorial, all eleven stand, because from that assistant's side a page it was never given and a page nobody wrote are the same thing. Which number is right depends on which question you came with, and the honest thing is to publish both rather than pick the flattering one. It also names the cheapest improvement available to this project, which is not a better extractor: it is indexing the other 95 pages.

No pass or fail gate is drawn, because at fourteen items a threshold would be decided by two or three of them. RefusalBench human validated 180 items and scored 93.1%, whose own interval runs from 0.88 to 0.96, so even a 90% gate could not separate work done by a funded team with expert annotators from failure. The number is reported with its interval instead.

What this does and does not establish

It establishes that at the observed rate, a reported near miss is a genuine absence in this corpus, confirmed by an answer written outside it, and that three independent readers agree on every one of the fourteen.

It establishes nothing about the other 139 findings. The explorer ships 153, and the 78.6% covers only the 14 near misses. Every orphan concept and every retrieval collision carries validated: null in data/findings.json and is labeled "Not independently checked" on its card, which is the honest label and not a reassuring one.

How much of the output was checked Of 153 findings, 11 were checked and held up, 3 were checked and did not, and 139 were never checked. The unchecked portion is every orphan concept and every retrieval collision. 11 held up 3 did not 139 never checked: every orphan concept and every retrieval collision
The validated fraction of what the explorer ships. Checking the other two types needs a different test from the one in this part, because an orphan concept has no thread answer to check against.

It says nothing about recall, the other half of that pair: of the gaps really out there, how many the pipeline found. How many it never noticed is not measured here and could not be, because measuring it would need the hand labeling this design was built to avoid. Every guard in the pipeline also pushes the same way: the span validator, the scope rule and the definition sweep each drop findings rather than add them, so they buy precision with recall. Fourteen near misses across 60 pages is a floor, not an estimate.

Part 7

What this does not do

  • It does not measure recall, and cannot. Every number here is about precision: of the gaps reported, how many are real. How many real gaps were never noticed is unmeasured, and measuring it would need the hand labeling this whole design exists to avoid. Every fix in Part 6 traded recall for precision, so fourteen near misses over 60 pages is a floor and not an estimate.
  • The retrieval comparison is underpowered. Both paths declined about 90% of the questions, so the comparison rests on around thirty answers each, and only 17 of those disagree. A null on that base is weak evidence of no difference, not evidence of no difference.
  • The demand signal is pre-filtered against the documentation. 282 of the 300 threads carry FastAPI's Questions template, in which the asker confirms they already read the tutorial and did not find the answer. That is excellent for finding gaps and useless for benchmarking answerability, because the sample is selected for being unanswerable from the docs. It is the price of the free ground truth in Part 3. A real team's search log would contain the answerable questions too.
  • Extraction quality bounds everything downstream. Every relation in the graph is a judgment a model made about one sentence, and three of the fourteen findings those judgments produced did not survive an independent check. That rate is measured and published, and whatever is still wrong in the extraction is still in these counts.
  • A corpus boundary looks exactly like a documentation gap. This one is now measured rather than argued. Three of the eleven near misses that held up are documented on a FastAPI page outside the 60, including the second most asked about finding on the page (Part 6). From a retriever's side a page you did not index and a page nobody wrote are the same thing, which is a real property of the problem, but it is not an excuse: the cheapest improvement available here is indexing the other 95 pages, and I have not done it.
  • One corpus, and a developer one. Everything here is FastAPI, so the method generalizes in principle and is demonstrated in exactly one place. A consumer help center is a different writing problem with no code samples to extract concepts from.
  • A sample, not an audit. 60 of 155 English pages and 300 of roughly 3,900 answered threads, and those 300 are not simply the most recent: a thread was skipped unless its accepted answer ran to 120 characters, because an answer of "+1" carries no fact that could be missing from the documentation. 67 were skipped to reach 300. The counts are counts within that slice.
  • No claim about a deployed assistant. This measures documentation. It does not measure escalations, deflection, or anything that happened to a real user.

Part 8

Prior work

None of the technique here is new, and it is worth being specific about who did what, because a portfolio project that quietly implies otherwise is worse than one that cites properly.

  • Sufficient Context, Google Research arXiv 2411.06037

    The 10% against 66% result, and the framing of sufficiency as the thing to measure rather than relevance. This is the method behind the near-miss finding. I did not come up with it and the finding type is an application of it. Read it.

  • Ragas open source

    Already builds a knowledge graph from a corpus, for the purpose of generating a test set. The graph-over-documentation idea is theirs and others', not mine. What is different here is the question asked of the graph and who the answer is addressed to.

  • Giskard open source

    Ships an out-of-scope question generator, which attacks a neighboring problem: finding questions your assistant should refuse rather than articles that will make it answer wrongly.

  • BRINK, UAEval4RAG, RefusalBench, AbstentionBench benchmarks

    All cover measuring whether a model abstains when it should. The technique of evaluating abstention is well populated, which is exactly why this project does not claim it.

So what is the contribution

Routing, not method. Every piece of prior work above produces a number about a system and hands it to whoever owns the system. Nobody who can fix a documentation gap can act on "your pipeline scores 0.71 on sufficiency".

This one is addressed to the person who owns the content, in their units: this article, this missing fact, this many people asked, and here is the sentence in your own file that shows the article depends on something you never wrote. That is a work item. A benchmark score is not.

How much of the work this actually covers Seven steps from noticing a documentation problem to shipping a fix: notice, locate, prioritize, assign, write, review, ship. This project covers the first two and part of the third. The remaining four and a half are done by a person somewhere else. THIS PROJECT Notice manufactured Locate article and fact Prioritize demand only Assign Write Review Ship SOMEBODY ELSE, IN ANOTHER TOOL
Two and a half boxes of seven. Worth drawing because the claim just made is a claim about routing, and this is how far the routing actually goes.

So the honest scope of this thing is a report, not a workflow. It manufactures a signal that did not exist, points at an article and names a missing fact, and orders the list by how many people asked. Then it stops. There is no export, no ticket, no owner and no done state, and a content lead who acted on it would be retyping findings into whatever system they actually work in.

That is the right place to have stopped for a demonstration, and it is the wrong place to stop for a product. If I were taking this further the next thing I would build is not more finding types. It is the export, because a finding that has to be retyped is a finding that competes with everything else in an inbox, and Part 4 already shows the yield is two urgent items per 60 pages. At that volume the tool has to cost almost nothing to act on, or the arithmetic never works.

Whether that repackaging is worth anything is a fair question, and this repository does not try to answer it with adoption numbers, because there are none and inventing a traction story would undercut everything else on this page.

The tool

Try it

The explorer holds all 153 findings against the real corpus. Every card carries a "How this was computed" toggle that opens the graph path, the verbatim sentence behind each hop, and a link to the exact lines of the exact file at the pinned commit.

That toggle is the point of the whole thing. A finding you cannot audit is a finding nobody acts on.