Three minutes
Summary
- The problem. Three things can happen when an assistant is asked a question: it answers, it declines, or it finds an article that is nearly right and answers confidently and wrong. Every signal a content team has (complaints, escalations, empty searches, the deflection rate) surfaces the decline and hides the confident wrong answer, because the person who was misled left satisfied. Google Research measured the danger: handed no context, models are wrong about 10% of the time; handed insufficient context, about 66% (Part 1, Part 2).
- The thesis. Since no signal arrives, the signal has to be manufactured from the documentation itself, before deployment. A map of the documentation as boxes and arrows, one box per section and per idea, an arrow wherever a section explains an idea or quietly assumes it, can express an absence: an idea that sections depend on, that users ask about, and that no section explains. Ordinary similarity search cannot express that, because it always returns the nearest piece however far away it is (Part 3).
- The method. 60 pages of the FastAPI documentation were mapped, with no arrow allowed to exist without the verbatim sentence that justifies it, and crossed with 300 answered questions from the project's own discussion board, whose accepted answers served as the known-correct answers. Every one of the fourteen near misses went to three independent readers told to search rather than judge from memory, and the bar was set in advance: none of the 60 pages explains the idea, and the thread's answer supplies a fact the pages lack. The same map was then tried a second way, choosing what the assistant reads, against ordinary similarity search over the same 300 questions, which is the control (Part 3, Part 5, Part 6).
- The results. 153 findings in all, 14 of them near misses. Eleven of the fourteen held up, unanimously; measured against all 155 pages of FastAPI's documentation rather than the 60 indexed, eight did, because three are documented on pages outside the index. As a way of choosing what the assistant reads, the map did not beat similarity search: 23 correct answers against 24, 3 confidently wrong against 5, on a comparison that could only have detected a much larger difference (Part 4, Part 5, Part 6).
- What it means. The map earns its cost as a diagnosis and has not earned it as a retriever. The null on retrieval has a known cause: 282 of the 300 questions came from people who said they had already read the documentation and not found the answer, which makes them excellent for finding gaps and useless for judging answering. Both precision numbers are published, because a gap-finding tool is judged on its false positives, and picking the flattering one would be the same failure this project is about (Part 5, Part 6, Part 7).
- Cautions, and how to check. Recall, the share of real gaps found, is unmeasured and cannot be measured without the hand labeling this design avoids, so fourteen is a floor. The scorer of the retrieval comparison was not blind to which path produced each answer, which is on the record and, if anything, favored the map. Only the 14 near misses were checked; the other 139 findings are labeled as unchecked in the explorer. Every number on this page is tested against the data files, and every finding carries the sentence, file and line it was read from (Part 6, Part 7).
Part 1
The three outcomes
First, what actually happens when someone asks
Everything below depends on one mechanism, so it is worth thirty seconds. When a company points an AI assistant at its documentation, the assistant does not read your documentation. It cannot: there is far too much of it to fit in a single question.
Instead your pages are chopped into small pieces ahead of time, usually a few paragraphs each. Those pieces are called chunks, and where the cuts fall matters more than it sounds. When somebody asks a question, separate software picks the handful of pieces that look closest to the question. That step is called retrieval. Only those few pieces are handed to the model, and the model writes its answer from them and nothing else.
A company points an AI assistant at its documentation. Three things can happen.
It works. The answer is in the docs, retrieval finds it, the assistant answers correctly.
It finds nothing. No article is close, so the assistant declines. Annoying, but safe, and above all visible. The user complains, a ticket gets filed, a search with no results gets logged, and the content team learns that something is missing.
It finds something close but incomplete. An article looks relevant, gets retrieved, and is missing one fact the answer depends on. The assistant answers confidently and wrong. Nobody sees it, because the assistant resolved the request and the deflection metric counted it as a success. The article looks healthy forever.
That third case is easier to recognize when you watch one happen. This is a real one, produced by ordinary similarity search over the same 60 pages, which is what almost every documentation assistant does today (Part 5), and scored against the reply the asker actually got.
Discussion 13455: "Fetching a lot of time series data"
Someone reports that a large JSON response takes about half a second to build, and asks how to speed it up.
Extra Data Types / Example
Server-Sent Events / Raw Data
Additional Responses / Combining information
Stream Data / No Annotation
Server-Sent Events / Stream SSE with FastAPI
Five sections, all plausibly about returning data. None of them is about where the half second is going.
"Let Pydantic serialize. Declare the return type (or response_model) instead of building the payload yourself. The docs state Pydantic serializes on the Rust side and you get much higher performance."
Fluent, specific, sourced to the retrieved text, and offered without a hedge.
That the timing figure the asker quoted is measured before the first byte arrives, so it already includes the round trip, and the server is spending that time computing the response rather than serializing it.
Different problem. Different fix.
The third case is not a rarer version of the second. It is substantially more dangerous, and that has been measured.
What Google Research measured
Across several models and datasets, answers were scored by whether the retrieved context was sufficient to answer the question. Given no supporting context at all, models answered incorrectly about 10% of the time, mostly by abstaining instead, which is the second outcome above. Given insufficient context, they answered incorrectly about 66% of the time.
Partial information does not make a model more cautious. It makes it confident. Sufficient Context: A New Lens on Retrieval Augmented Generation Systems, arXiv 2411.06037.
That is the whole premise. An article that is 90% of an answer is worse than an article that is 0% of one, and it is the 90% article that no process in a content team is currently looking for.
Part 2
Why nobody sees the third one
Content teams do not prioritize at random. They prioritize from signals, and there are four of them.
| Signal | Fires when the assistant declines | Fires when the assistant is confidently wrong |
|---|---|---|
| Customer complaints | Yes. The user did not get an answer and says so | No. The user got an answer, believed it, and left |
| Escalations to a human | Yes. The conversation gets handed over | No. The assistant resolved it |
| Searches returning nothing | Yes. That is exactly what the log records | No. Retrieval returned five confident results |
| Deflection rate | Counts against you, correctly | Counts for you. The failure improves the metric |
This is structural rather than an oversight in anyone's tooling. Every one of those signals is a report of dissatisfaction, and nobody is dissatisfied at the moment a confident wrong answer happens. The person who finds out is a customer, three days later, when the thing they were told does not work.
A signal that does not arrive cannot be collected. It has to be manufactured, before deployment, out of the documentation itself. That is the rest of this.
Part 3
What I built
A pipeline that maps a documentation set as a knowledge graph, crosses it against questions real users actually asked, and produces a ranked list of the articles most likely to make an assistant answer confidently and wrong, naming the specific missing fact in each case.
Here is the shape it hunts for, before any of the machinery. This is one real finding.
The corpus, and why this one
60 pages of the FastAPI documentation, taken in the documentation's own reading order: all 51 tutorial pages, then the first 9 advanced pages, pinned to one commit.
The corpus is not the interesting decision. The demand signal is, meaning the evidence that somebody actually wanted the missing thing. The same repository runs a GitHub Discussions board with roughly 3,900 answered questions, so one source supplies the documentation, the questions people actually asked about it, and the correct answers, written by people who know. That last part is what removes the need to hand label anything.
The decision the whole project rests on
If a question was answered in a discussion and that answer is not in the documentation, then the gap is real and the missing content already exists in somebody's reply. That is the project's ground truth, meaning the known-correct answer everything else gets scored against. Free, verifiable by anyone, and no annotation budget.
The cost is that it only sees gaps somebody bothered to ask about in public, which is a narrower set than a real team's search log.
The graph
Five node types and seven edge types. Deliberately small: this is a schema designed around one question, not a general ontology.
data/manifest.json.
| Edge | From, to | What it claims | Count |
|---|---|---|---|
| DEFINES | Section, Concept | This section explains what the thing is or how to use it. A reader who knew nothing could learn it here | 914 |
| REQUIRES | Section, Concept | This section's instructions assume the reader already knows the thing. Following them without it would fail | 1,611 |
| MENTIONS | Section, Concept | Named in passing, neither explained nor depended upon | 994 |
| ASKS_ABOUT | Question, Concept | Answering this question requires understanding this concept | 960 |
| CONTAINS | Doc, Section | Structural | 570 |
| ANSWERED_BY | Question, Answer | GitHub marked this reply as the accepted answer | 300 |
| LINKS_TO | Section, Doc | An internal documentation link | 50 |
48 times the extractor produced two relations for the same section and concept, usually "explains" alongside "assumes". Where that happens the finders treat the concept as explained and drop it, which loses findings rather than inventing them.
Why this shape rather than another
The whole design turns on the difference between explaining a thing and assuming it. Every other graph over documentation I looked at models "this page is about that thing", which cannot express the finding, because being about a thing and teaching it are different claims.
Two other shapes were available for the same job. This graph wins on what it can express and does not win on retrieval, which Part 5 measures and finds no difference on.
One of the two is what nearly everyone actually builds, so it is worth four sentences. A vector index takes each of those chunks from Part 1 and turns it into an embedding: a few hundred numbers standing for what the chunk is about, arranged so that chunks about similar things end up with similar numbers. Your question is turned into numbers the same way, and retrieval hands back whichever chunks sit closest to it. Nothing in that arrangement knows what a chunk says, only what it resembles, and that one fact is what the first column below is losing on.
| A flat vector index | This typed graph | An untyped entity graph | |
|---|---|---|---|
| What it stores | One embedding per chunk, and nothing about how chunks relate | Typed, evidenced relations between sections, concepts and questions | Entities and one generic association, usually with a merged description per entity |
| Can it express an absence | No. Similarity has no way to say "nothing here teaches this". The nearest chunk is always returned, however far away it is | Yes. "No Section DEFINES this Concept" is a query, and it is the whole finding | Partly. It can find an entity with few links, but not one that is depended on and never explained, because it has no edge type for the difference |
| Can you audit a result | No. A similarity score is not a reason, and there is no sentence to point at | Yes. Every edge carries the sentence, the file and the lines | No. Merging several sources into one description destroys the spans before you can read them |
| Cost to build | Minutes. One embedding pass, about 80MB of model, no judgment anywhere | High. 79 model batches for the documentation and 30 for the questions, then a completeness sweep over every concept nothing defines, then an independent check of every finding it reports | Moderate. One extraction pass, no evidence to validate, which is exactly why there is less to check |
| Cost to keep true | Re-embed the changed pages | Re-extract them, and every relation is a judgment that can be wrong in a way an embedding cannot | Re-extract, with less to get wrong |
| Measured retrieval quality | 5 confidently wrong answers over 300 questions | 3 over the same 300, which the sample cannot resolve | Not measured here |
The graph earned its cost as a diagnosis, not as a retriever.
That is the trade in one line. A vector index is close to free and can never tell you what your documentation fails to say, because every question gets its five nearest chunks whether or not any of them contain the answer. This graph can tell you, and it cost a great deal more to build, and most of that cost was not the graph. It was the extraction and the checking that the graph makes it possible to demand, which is also what caught 24 wrong findings before anybody read them.
The one rule everything else hangs off
No edge without evidence.
Every edge carries the verbatim source span that justifies it, plus the file path and the line range where that span lives. There are no exceptions, including for the structural edges where the span is obvious. It is the difference between "the graph says so" and "here is the sentence the graph read", and without it none of the findings are actionable, because nobody acts on a finding they cannot check.
It is also enforced rather than requested. A model asked for a verbatim span will sometimes paraphrase, so no span is believed. Each one is searched for in the file its edge claims it came from, and an edge whose span is not found there is dropped, not downgraded to low confidence. The permitted normalizations are Unicode form, whitespace, quote and dash shapes, and case, and that is the complete list.
What the enforcement actually caught
Of 3,509 documentation edges proposed across 79 batches, zero were dropped for an unfindable span. That is not a sign the check was unnecessary. It is a sign the instruction worked, and it is only knowable because the check exists.
The same check on question spans dropped 13% on the first attempt, for a reason worth knowing: GitHub serves a discussion body twice, once as markdown and once as the plain text it renders to. The mapper read the plain text and the file on disk holds the markdown, so a span containing a backtick could not be found. Matching the two like with like recovered 124 of 125.
The three findings
That shape is the first of three. Each is a shape in the graph, and each carries a proof path of real arrows leading to real lines in real files.
| Type | The shape | The proof |
|---|---|---|
| F1 Near miss | A Concept that Questions ask about, that a Section REQUIRES, and that no Section in the corpus DEFINES | The asking span, the depending span, and the empty define set with the count of sections checked. Confirmed by the answer on the thread, quoted verbatim |
| F2 Orphan concept | A Concept with REQUIRES or MENTIONS in-degree above zero, DEFINES in-degree of zero, and at least two sections leaning on it | The in-edge list and the empty define set |
| F3 Retrieval collision | Two Sections a single Question maps to with comparable concept overlap, where only one carries the answer | The competing sections, their overlap scores, and each one's similarity to what the maintainer actually replied |
F3 is what the demonstration in Part 1 was. When two sections cover the same ground and only one carries the fact, retrieval has no way to prefer the right one, so the assistant answers from whichever scored higher and the failure looks like a success. The fix is usually not new writing, it is a pointer from the section that gets retrieved to the section that has the answer. It is also the commonest finding here, 99 of 153, and the least verified.
Everything is ranked by demand, meaning the number of distinct discussion threads touching the concept. A gap nobody asks about is real and is not worth writing this quarter, and a tool that cannot tell the difference produces a backlog instead of a priority.
Part 4
The fourteen it found
| Type | Findings | Threads reached | What that means here |
|---|---|---|---|
| F1 Near miss | 14 | 20 | Two of them were asked about by more than one person, and one thread raised two of these gaps at once, so fourteen findings reach twenty threads. The rest were asked once, which is what a long tail looks like |
| F2 Orphan concept | 40 | 0 | Zero by construction: any orphan somebody asked about is promoted to F1, so F2 is precisely the tail nobody has asked about yet. These are near misses with the question missing. The dependency and the absence are already in place, so the only thing standing between an orphan and a confident wrong answer is somebody finally asking |
| F3 Retrieval collision | 99 | 116 | 12 of them caught more than one question. Each one is a question, so demand is never zero |
The near misses, in full
There are only fourteen, so here they all are rather than a top ten, with the fact the documentation never supplies. The three marked as not holding up were rejected by the independent check in Part 6 and are still listed, because a table that quietly drops its own failures is not evidence of anything.
| Threads | Concept | What the documentation never says | Held up |
|---|---|---|---|
| 6 | event loop | how to run a coroutine on the main thread's event loop from a sync path operation function, which FastAPI runs on a threadpool worker | Yes |
| 3 | APIRoute |
which parts of APIRoute are safe to subclass, given that overriding its constructor is not a supported extension point | Yes |
| 1 | EmailStr |
how to declare an optional EmailStr form field that accepts an empty submission, which browsers send as an empty string | Yes |
| 1 | exc.errors() |
when WebSocketRequestValidationError applies, which is only during the connection stage before accept | Yes |
| 1 | port | which layer returns a 502 with an empty body while uvicorn is already listening on the port | Yes |
| 1 | session.exec |
which select session.exec requires, the one from sqlmodel rather than the one from sqlalchemy | Yes |
| 1 | host | how to check whether a server bound to 0.0.0.0 is actually reachable from a client | Yes |
| 1 | jwt.decode |
which jwt.decode arguments handle a token carrying an aud claim, the audience value or the verify_aud option | Yes |
| 1 | Literal |
what Enum gives predefined parameter values that typing.Literal does not, and when each one fits | Yes |
| 1 | ValidationError |
how to return a validation style error for one body field from inside a path operation function, and what shape each entry in the detail list must take | Yes |
| 1 | virtual environment | which Python version a virtual environment for running FastAPI's own test suite must use | Yes |
| 1 | path operation | how to attach several named OpenAPI examples to a response declared on an APIRouter | No |
| 1 | fastapi.encoders |
how to give jsonable_encoder a rule for a type it does not recognize | No |
| 1 | TokenData |
why the tutorial declares a TokenData model rather than passing the decoded username straight through | No |
These are the shape a content team can act on. Not "improve the
security docs", but "the OAuth2 page never says which
jwt.decode argument handles an audience claim, and here
is the thread where somebody had to be told". That is a work item
with a reader attached to it.
Fourteen is a small number, and that is the point
What is worth saying about the fourteen is what they have in common: almost all of them are demand of one. One person asked, once.
A gap nobody has asked about twice is still a gap, and it is not this quarter's work.
That is why demand is on every finding rather than being used as a filter. Two of the fourteen have real recurring demand, and a content lead with a week to spend should spend it on those two. The other twelve are a backlog, ranked correctly below them, and three of those twelve are the ones that did not hold up. A tool that could not tell those apart would hand over 153 equally urgent tasks, which is the same as handing over nothing.
Part 5
Does the graph make a better assistant?
Everything so far uses the graph to diagnose documentation, to find the holes. There is a second and completely separate thing you can do with it: use it live, to choose which pieces the assistant reads when somebody asks a question. That is worth testing, because if it works it is a much bigger idea than a report.
So both were built and both answered all 300 questions.
| Path | How it picks the five |
|---|---|
| A, ordinary similarity search | The method almost every documentation assistant uses today, and the thing to beat. Every piece is turned into numbers, the question is turned into numbers the same way, and the five pieces whose numbers sit closest to the question's come back |
| B, graph retrieval | Start from the sections that explain what the question asks about. Then follow one arrow further: for anything those sections themselves assume, pull in the section that explains that. Three slots for the first kind, two reserved for the second |
Path B's reserved slots are the whole experiment. A section that answers your question is easy to find by similarity, because it sounds like your question. The thing that section quietly assumes is not, because a prerequisite never sounds like the question that needs it. Only the graph knows the two are connected, and that connection is what is being tested.
The first version of this experiment did not test anything
It filled all five slots by similarity to the question first and appended the prerequisites afterwards, so the prerequisites were only ever reached when a question produced fewer than five starting sections. That was 70 questions of 300. On the other 213 the traversal ran and its results were thrown away before the assistant saw them.
Reserving the slots fixes it, and the mechanism now reaches the assistant on 278 questions of 300. The numbers below are from that corrected run.
What a prerequisite looks like in practice
Discussion 13828 asks why declaring the same request body on both a dependency and the path operation validates it twice.
| Path A picked | Path B picked |
|---|---|
| Body Multiple Parameters / Multiple body params and query | Request Body |
| Get Current User / Create a user model | Classes as Dependencies / Shortcut |
| Path Operation Configuration / Response description | Classes as Dependencies / Type annotation vs Depends |
| Body Multiple Parameters / Multiple body parameters | Classes as Dependencies / Classes as dependencies |
| Declare Request Example Data / Body with multiple examples | Classes as Dependencies / Use it |
The step that fetched them, copied out of the results file
sec:tutorial/dependencies/classes-as-dependencies#shortcut
REQUIRES concept:commonqueryparams -> DEFINES ->
sec:tutorial/dependencies/classes-as-dependencies#use-it
Read it as: the Shortcut section assumes you know what CommonQueryParams is, the Use it section is where that is explained, so fetch it too.
Path A declined this question. Path B answered it correctly, using the section the arrow fetched. That is the mechanism working exactly as intended, and it happened often enough to look promising and not often enough to show up in the totals.
The result
Every answer was scored against the reply the asker actually got, into one of three buckets.
| Bucket | What it means |
|---|---|
| Correct | Matches the maintainer on the substance |
| Confidently wrong | Presents an answer that contradicts the maintainer or misses the fact the answer turns on, with no hedge. This is the failure the whole project is about |
| Declined | Says outright it cannot answer from what it was given |
| Path | Correct | Confidently wrong | Declined |
|---|---|---|---|
| A, similarity search | 24 | 5 | 270 |
| B, graph retrieval | 23 | 3 | 273 |
One disclosure before the numbers. The scorer was not blind: every
block it was handed is headed
SCORE ID: q:13029::vector or
SCORE ID: q:13029::graph, so the prompt named the path
that produced the answer, and the similarity answer came first in all
300 pairs. That is a defect in the setup, it is on the record in
data/work/compare/score/in/, and if it favored anything
it favored the graph.
The graph did not make the assistant more right. 23 correct against 24. Both paths answered the same 300 questions, so the informative cases are the 17 where exactly one of them got it right, and that split is 9 to 8. A coin would do that.
It did make the assistant more cautious. Confidently wrong answers went from 5 to 3, and declines went up. Following the arrows swapped nine correct answers for declines, converted three of the five confident errors into a decline or a correct answer, and picked up three new correct answers along the way.
It did not make the assistant know more. It made it admit ignorance more often.
Which is arguably the trade this whole essay argues for, since a decline is the safe failure and a confident wrong answer is the dangerous one. It is also far too small to claim. Three fewer errors out of 300 is the kind of difference that appears and disappears between runs, and the honest summary of Part 5 is that graph retrieval did not beat ordinary similarity search here on any measure that the sample can resolve.
How small a difference could this have caught
Saying an experiment is "too small" is not useful unless you say how small. Only the 17 questions where the two paths disagreed carry any information, and with 17 of those, the least lopsided split that would have counted as a real difference is 14 to 3. The observed split was 9 to 8.
So this design could only have detected an effect several times larger than anything that showed up. It has not shown that the two methods are equally good. It has shown that if there is a difference, it is small, and that this corpus cannot resolve it.
Why the sample cannot resolve it, which is the interesting part
Both paths declined about 90% of the questions, so the entire comparison rests on the 30 or so each one actually answered. That is not the retrieval. That is the question set, and the reason is the most interesting thing the project found by accident.
282 of the 300 askers had already read the documentation and said so
FastAPI's Questions form asks people to confirm it before posting, line by line: "I searched the FastAPI documentation, with the integrated search", and "I already read and followed all the tutorial in the docs and didn't find an answer." 282 of the 300 threads carry it.
Excellent for finding gaps. The demand signal is drawn from exactly the population the documentation failed.
Useless for benchmarking answerability. The sample is selected for being unanswerable from the docs, so it cannot fairly measure how well anything answers from the docs.
Which makes a 90% decline rate close to correct. An assistant that confidently answered most of these, from the same tutorial the asker had just read, would be doing the exact thing this project exists to find.
I did check whether my own instructions caused the abstaining rather than the corpus, by rerunning three batches with the line about what was being measured removed: 11 declines of 15 instead of 12 of 15, which is far too small a probe to settle it and is reported here as the weak check it is.
So where does that leave it
The graph earns its cost as a diagnosis and has not earned it as a retriever. The honest next step is not to keep tuning this until the number moves, which is how people end up believing things that are not true. It is a different question set: real search-log queries, which are not pre-filtered against the documentation, and enough of them that the answered subset is in the hundreds rather than the thirties. Until that runs, the finding stands as a null with a known cause.
Part 6
Checking every finding
Fourteen near misses is a small enough population to check all of them, so all of them were checked rather than a sample. Each one went to three independent readers, given the corpus, told to be skeptical, and told to go and search rather than judge from memory. A finding counts as valid only when no section of the corpus explains the concept and the answer on the thread supplies a fact the corpus does not contain.
| Checked | Held up | Rate | Wilson 95% | Readers each | Agreement |
|---|---|---|---|---|---|
| 14 | 11 | 78.6% | 0.52 to 0.92 | 3 | Unanimous on all 14 |
Three did not hold up: path operation, fastapi.encoders and TokenData. They are still in the table in Part 4 and still in the explorer, marked as not holding up, because a list that quietly drops its own failures is not evidence of anything.
The path operation case is worth its own paragraph
All three readers agreed the corpus defines "path operation", and it does: First Steps gives it a titled step, then defines "Path" in one subsection and "Operation" in the next. No single heading-scoped chunk defines it, and the parent that ties the two halves together is two lines long.
The graph is section-level because retrieval is chunk-level. So the page teaches it and no retrievable piece of the page does, which is a real retrieval problem wearing the costume of a false positive. It is counted against the tool above, because that is the honest call under the rule as written.
A gap in FastAPI, or a gap in my index?
Everything above measures the findings against the 60 indexed pages. "These 60 pages never explain it" and "FastAPI never explains it" are different claims, and only the first one follows from a check run inside the corpus. A reader hears the second.
So the other 95 English documentation pages were fetched and all fourteen near misses put in front of three more readers, told to search only outside the corpus and to report either a definition with its file and line, or nothing.
data/corpus_boundary.json.
Only the top left cell is a documentation gap in the sense a reader
would assume.
The top right cell is the one that costs something.
APIRoute is the second most asked about finding on the
page, and FastAPI documents it properly in a how-to guide that is not
in the 60. So does host, and virtual environments get a
page of their own. A content team handed those three would go and
write something that already exists.
Eight of the fourteen are gaps. Three are my index pretending to be a gap.
This is the concrete version of the thing Part 7 states abstractly, and it cuts both ways. Measured as a claim about FastAPI, precision, meaning how many of the gaps this tool reported are real, drops from 11 of 14 to 8 of 14. Measured as a claim about an assistant that indexes only the tutorial, all eleven stand, because from that assistant's side a page it was never given and a page nobody wrote are the same thing. Which number is right depends on which question you came with, and the honest thing is to publish both rather than pick the flattering one. It also names the cheapest improvement available to this project, which is not a better extractor: it is indexing the other 95 pages.
No pass or fail gate is drawn, because at fourteen items a threshold would be decided by two or three of them. RefusalBench human validated 180 items and scored 93.1%, whose own interval runs from 0.88 to 0.96, so even a 90% gate could not separate work done by a funded team with expert annotators from failure. The number is reported with its interval instead.
What this does and does not establish
It establishes that at the observed rate, a reported near miss is a genuine absence in this corpus, confirmed by an answer written outside it, and that three independent readers agree on every one of the fourteen.
It establishes nothing about the other 139 findings. The explorer
ships 153, and the 78.6% covers only the 14 near misses. Every orphan
concept and every retrieval collision carries
validated: null in data/findings.json and is
labeled "Not independently checked" on its card, which is the honest
label and not a reassuring one.
It says nothing about recall, the other half of that pair: of the gaps really out there, how many the pipeline found. How many it never noticed is not measured here and could not be, because measuring it would need the hand labeling this design was built to avoid. Every guard in the pipeline also pushes the same way: the span validator, the scope rule and the definition sweep each drop findings rather than add them, so they buy precision with recall. Fourteen near misses across 60 pages is a floor, not an estimate.
Part 7
What this does not do
- It does not measure recall, and cannot. Every number here is about precision: of the gaps reported, how many are real. How many real gaps were never noticed is unmeasured, and measuring it would need the hand labeling this whole design exists to avoid. Every fix in Part 6 traded recall for precision, so fourteen near misses over 60 pages is a floor and not an estimate.
- The retrieval comparison is underpowered. Both paths declined about 90% of the questions, so the comparison rests on around thirty answers each, and only 17 of those disagree. A null on that base is weak evidence of no difference, not evidence of no difference.
- The demand signal is pre-filtered against the documentation. 282 of the 300 threads carry FastAPI's Questions template, in which the asker confirms they already read the tutorial and did not find the answer. That is excellent for finding gaps and useless for benchmarking answerability, because the sample is selected for being unanswerable from the docs. It is the price of the free ground truth in Part 3. A real team's search log would contain the answerable questions too.
- Extraction quality bounds everything downstream. Every relation in the graph is a judgment a model made about one sentence, and three of the fourteen findings those judgments produced did not survive an independent check. That rate is measured and published, and whatever is still wrong in the extraction is still in these counts.
- A corpus boundary looks exactly like a documentation gap. This one is now measured rather than argued. Three of the eleven near misses that held up are documented on a FastAPI page outside the 60, including the second most asked about finding on the page (Part 6). From a retriever's side a page you did not index and a page nobody wrote are the same thing, which is a real property of the problem, but it is not an excuse: the cheapest improvement available here is indexing the other 95 pages, and I have not done it.
- One corpus, and a developer one. Everything here is FastAPI, so the method generalizes in principle and is demonstrated in exactly one place. A consumer help center is a different writing problem with no code samples to extract concepts from.
- A sample, not an audit. 60 of 155 English pages and 300 of roughly 3,900 answered threads, and those 300 are not simply the most recent: a thread was skipped unless its accepted answer ran to 120 characters, because an answer of "+1" carries no fact that could be missing from the documentation. 67 were skipped to reach 300. The counts are counts within that slice.
- No claim about a deployed assistant. This measures documentation. It does not measure escalations, deflection, or anything that happened to a real user.
Part 8
Prior work
None of the technique here is new, and it is worth being specific about who did what, because a portfolio project that quietly implies otherwise is worse than one that cites properly.
-
Sufficient Context, Google Research
arXiv 2411.06037
The 10% against 66% result, and the framing of sufficiency as the thing to measure rather than relevance. This is the method behind the near-miss finding. I did not come up with it and the finding type is an application of it. Read it.
-
Ragas
open source
Already builds a knowledge graph from a corpus, for the purpose of generating a test set. The graph-over-documentation idea is theirs and others', not mine. What is different here is the question asked of the graph and who the answer is addressed to.
-
Giskard
open source
Ships an out-of-scope question generator, which attacks a neighboring problem: finding questions your assistant should refuse rather than articles that will make it answer wrongly.
-
BRINK, UAEval4RAG, RefusalBench, AbstentionBench
benchmarks
All cover measuring whether a model abstains when it should. The technique of evaluating abstention is well populated, which is exactly why this project does not claim it.
So what is the contribution
Routing, not method. Every piece of prior work above produces a number about a system and hands it to whoever owns the system. Nobody who can fix a documentation gap can act on "your pipeline scores 0.71 on sufficiency".
This one is addressed to the person who owns the content, in their units: this article, this missing fact, this many people asked, and here is the sentence in your own file that shows the article depends on something you never wrote. That is a work item. A benchmark score is not.
So the honest scope of this thing is a report, not a workflow. It manufactures a signal that did not exist, points at an article and names a missing fact, and orders the list by how many people asked. Then it stops. There is no export, no ticket, no owner and no done state, and a content lead who acted on it would be retyping findings into whatever system they actually work in.
That is the right place to have stopped for a demonstration, and it is the wrong place to stop for a product. If I were taking this further the next thing I would build is not more finding types. It is the export, because a finding that has to be retyped is a finding that competes with everything else in an inbox, and Part 4 already shows the yield is two urgent items per 60 pages. At that volume the tool has to cost almost nothing to act on, or the arithmetic never works.
Whether that repackaging is worth anything is a fair question, and this repository does not try to answer it with adoption numbers, because there are none and inventing a traction story would undercut everything else on this page.
The tool
Try it
The explorer holds all 153 findings against the real corpus. Every card carries a "How this was computed" toggle that opens the graph path, the verbatim sentence behind each hop, and a link to the exact lines of the exact file at the pinned commit.
That toggle is the point of the whole thing. A finding you cannot audit is a finding nobody acts on.