AI Property Maintenance Advisor

Part 1

The problem

The property is an eight bedroom lake rental with a hot tub, two heating and air conditioning (HVAC) systems, a septic system, and a dock. The owner lives ninety miles away. When something breaks, the report arrives as a sentence from a guest or a cleaner: the hot tub is not heating, the upstairs is warm. That sentence is everything the technician gets. It does not carry the model, the age of the unit, the code on the panel, or what was tried the last time this happened. So the first visit is often diagnostic rather than corrective, and the part comes on the second one.

The job: when equipment fails at a property I am not at, help me tell whoever is going to fix it what this thing is and what the documentation says about this symptom, so they arrive with the right part instead of finding out what they need.

Note the shape of that job. It is not "diagnose the fault." Nobody standing ninety miles away, and no tool, should be doing that. It is "carry the documentation to the person who can."

Part 2

What I actually know

The demand signal I found pointed somewhere slightly different: local forums are full of owners asking who to call, not what is wrong. I noted it, kept building the narrower thing, and left the larger question open rather than quietly redefining the product to match the evidence I happened to find.

Part 3

What v1 proved

Every decision in both versions follows from one: the tool is allowed to say it does not know, and that is a success state, not an error. A maintenance tool that returns a confident, wrong, plausible looking repair instruction sends someone to open an equipment bay on live electrical, or to order a part that was never the problem. So the product has a NO RELIABLE ANSWER state, with its own layout showing what was searched, what was found, and why that was not enough.

A tool that is right most of the time and silent about which times is not a tool a technician can use.

v1 worked like this: photograph the plate, confirm what was read, get a brief. It made two calls to Claude Opus 5, the highest priced of the three Claude models priced in the build, using Anthropic's own built in web search capped at 8 searches, and kept no database. Five decisions shaped it, and all carry into v2.

What the v1 run showed

Six acceptance cases were defined in the spec before the build. All six passed on content; the speed criterion did not. The four v1 briefs remain published as the evidence for this table: hot tub, code on panel, hot tub, vague symptom, air conditioner and the invented model.

On the criterion v1 missed. Reducing how much reasoning each brief gets would have made the sixty seconds. I did not make that change, because the same setting buys the refusal behavior. v1 had not earned the right to say whether the criterion was wrong or the product too slow.

Part 4

Where v1 stopped

Running v1 showed three limits that no prompt could fix, because each is about what the tool remembers and knows, not about how well the model writes.

• It forgot everything. Every brief ran 7 to 8 live searches and read 42,000 to 91,000 tokens of input (a token is a word fragment, the unit models are billed in). A second question about the same hot tub paid the full price again. v1 logged every lookup but never stored what it verified.
• It knew nothing about the property. The brief has a "this has happened before" field. It was empty in all 4 recorded v1 briefs, because v1 had no service history to fill it from.
• It had no path for aging equipment. When a part is discontinued, the useful answer is a documented successor model, and v1 had nowhere to put one.

Cost made the first limit sharper. v1 never computed its own cost; its logged usage, priced at current list prices, puts a v1 brief at an estimated $0.35 to $0.71 (Part 6). Version two sets out to remember what it has verified, to know each property, and to spend less per brief, without moving any of the five decisions above out of code. Two items from v1's cut list come back for that reason: several properties, and warranty tracking.

Part 5

Why LangGraph and LangChain

LangGraph is a library that runs an AI workflow as a graph of steps. Each step reads and writes a shared record of the run, and arrows between the steps, called edges, decide what runs next. LangChain is a library of building blocks for AI apps: connectors to model providers, tools, and a ready made agent loop in which the model calls a tool, reads the result, and decides what to do next.

v1 was a fixed sequence of steps, and plain code was the right tool for it. v2 adds four things a fixed sequence cannot express cleanly: routing (answer from memory, top up with a few searches, or research in full), a bounded retry loop, a pause for a person to confirm the unit, and saved state for each property. Those four are a real reason to use a graph runtime, not a way to show off a framework.

The split: LangGraph runs the flow, LangChain runs one step

LangChain's ready made agent, create_agent, was considered for the whole flow and rejected: it cannot hold a confirmation pause that survives the program being closed and started again, the validation retry loop, or state kept per property. So LangGraph runs the whole flow, and create_agent runs only the research step inside it, where a model and tool loop is exactly what is needed.

The tradeoffs actually paid

• The names move fast. Many functions were renamed or moved after the 1.0 releases, and the documentation and the source code disagree in three places. Every name was checked against the current release, the source was taken as correct, and v2 relies on none of the three disputed behaviors.
• No stock fake model could drive the agent. A fake model returns scripted replies so tests cost nothing. None of LangChain's supports tool calling (the model asking the code to run a search or fetch), and one silently loops back to the start of its script. v2 wrote its own, which fails loudly. Because it never checks the form the way the real API does, one paid check covered that gap.
• The structured output ceiling. Anthropic limits a form to 24 optional and 16 either or fields, and LangChain passes the form through unchanged. A draft modeled on v1's lenient form would have failed on the first paid call, so the form is now flat, with every field required.
• Defaults that had to be overridden. The agent's loop limit defaults to 9,999 steps, so the middleware caps are the only real guard; LangChain's own search limit leaves no reason in the run's record, so v2 wrote two small middleware pieces of its own. The Claude connector defaults to the model's maximum output, which on Haiku 4.5 would have reserved $0.32 against a $0.15 cap, so every call sets its own output limit, and neither connector has a timeout until one is set.
• A rewrite in a second language. v1 is TypeScript and v2 is Python, because Python has the more complete LangGraph ecosystem (an inference, not a measurement). The price was porting v1's validator; the differential test pays it back (Part 8).
• Model lifetime. Anthropic listed Haiku 4.5 as retiring no sooner than 15 October 2026, and the live command refuses a model's runs from that date. So cheap mode moved to Claude Haiku 5.5, listed as retiring no sooner than 7 October 2027, and every live Claude result the page reports as current was measured again on it, with Haiku 4.5's figures kept only for comparison (Part 8).

Alternatives rejected

• A vector store beside the graph. It finds text by similarity of meaning. In my earlier project on documentation gaps, vector search scored 24 correct against 23 for graph retrieval, a gap too small to justify a third way to look things up.
• A hosted graph database. At the size this build was planned for (about 50 nodes, an estimate), a Python library that holds the graph in memory and saves it as a plain file does the same job, with no account and nothing to pay.
• Building the corpus up front. v1 already rejected it. v2 grows memory only from lookups that really happened, so early questions pay for full research.

Part 6

Two APIs, each for its job

An API is a service the code calls over the internet and pays for by use. v1 used one API for everything: Claude Opus 5 read the plate, ran Anthropic's own web search, and wrote the brief. v2 splits the work by what each step actually needs. Since 30 September a third service, TypeSafe's Jev, also checks each step to try first in a brief for safety (Part 9), and the product recall check asks a free government service, with no key and no cost (Part 7).

Why the split is cheaper

Per million tokens, Claude Haiku 5.5 costs $0.10 in and $0.50 out on prompts up to 100,000 tokens ($0.50 and $2.50 above that), Sonnet 5 costs $2 and $10, and Opus 5 costs $5 and $25 (Anthropic's pricing page, checked 13 October 2026); Anthropic's server search adds $0.01 per search. Haiku 4.5, which the earlier live runs used, cost $1 and $5. So on shorter prompts the model price alone is a fiftieth of Opus 5's, and the search line, an estimated $0.07 to $0.08 per v1 brief (7 to 8 searches at $0.01), falls to $0 on Tavily's free tier. The $0 holds only on the free tier: at Tavily's pay as you go rate of $0.008 a credit, a worst case first pass of 8 credits would cost about $0.064 (arithmetic, not measured). So at volume most of the saving comes from the model and from memory skipping search.

Runs marked "no plate photo" were given the model directly, so their cost includes no plate reading; the other hot tub runs include it. On shorter prompts Haiku 5.5 costs a tenth of Haiku 4.5's price per token, but per run it cost about a sixth.

The hard caps are $0.15 per cheap mode brief and $5 for the whole build. Every live run on build d24ed7c475dcd813 stayed under its cap: the highest, one of the ten first lookups, cost $0.0098. The runs on that build cost $0.1301 and 71 Tavily credits in total. The whole build has spent $1.5532 of its $5 and 249 of its 300 Tavily credits, a total that counts every live run on every build, the safety step work of 29 and 30 September, the Claude comparison of 15 October (Part 9) and the two MCP briefs of 16 October (Part 7) included. On credits, a first pass uses at most 8 (5 searches and 3 fetches) and the one smaller retry up to 2 more, so a brief uses at most 10, and 1,000 free credits cover about 100 research briefs a month at the worst case (arithmetic, not measured).

The second payoff: the rules stay in code

In v1, Anthropic's built in search returned its citations in a way that did not work together with structured output, so v1 could not have code build the source list; it relied on an instruction to the model not to pad the bibliography. In v2, the search and fetch wrappers hand code the raw results. Code builds the source list only from searches and fetches that actually succeeded, the model can point to a source only by its number, and because code holds the raw page text it can check that every quote appears word for word on its page and that a cited error code appears there as a whole word. The instruction became a rule enforced in code.

How LangChain and LangGraph carry the split

• LangChain puts both providers in one loop. The agent runs Claude's tool calling over Tavily's search and fetch, and one configuration file names the model for each step.
• Middleware counts both bills in one place. It counts Tavily searches and fetches against their limits and reserves each Claude call in a spending ledger before the call is made.
• LangGraph decides whether either API is called at all. Routing skips search when memory holds enough verified facts and sends a top up of 2 searches when it holds some. The confirmation pause sits before any research spend, so the only paid call before the owner confirms is the plate reading, which cost $0.0005 and $0.0006 live on build d24ed7c475dcd813.

The tradeoffs. Citation mapping, the page evidence checks and the credit count all became v2's own code, which is more to maintain than one provider, and it is the part that keeps the source list honest. The second tradeoff is quality: Haiku was picked because it is cheapest, and a cheaper model can write a less complete brief while every rule passes. So "cheaper" and "as good" are separate claims, and this page supports only the first (Part 9).

Part 7

What v2 lets owners and technicians do

Each feature is described by what the person gets and why it matters on the day something breaks. Where a live run shows it working, the demo case that replays it is named. The recorded v2 demo replays eight live runs of 14 October 2026 on build d24ed7c475dcd813. Four of the eight (cases 3, 4, 5 and 6) end with no reliable answer, and the demo shows them as refusals, the success state when the documents do not support an answer, though for the three real unit cases (3, 4 and 5) each also counts as a miss in Part 8. It shows warranty status on case 8, but not the replacement sweep, the product recall check or the MCP server.

Part 8

How v2 was tested

Replay runs the whole pipeline on recorded model replies and recorded search results, at no cost and with the network blocked. It proves the rules, the routing, the ledger and the page renderer, but not how the model behaves. Small live batches on Claude Haiku 5.5 and Tavily, run on 13 and 14 October 2026 on build d24ed7c475dcd813, test the model; Claude Haiku 4.5's results of 30 September appear only where the comparison is the point. The bars were written into the requirements on 17 September 2026, before any test ran, and the bar for first lookups before the ten new runs.

The rules stayed in code

• 14 of v1's 17 guardrail behaviors are ported. The other 3 tested parsing a block of text the model returned, which structured output removed, and are marked obsolete.
• v1's and v2's validators agree on the decision, the reason and the cleaned brief for all 180 differential test briefs: 16 from v1's guardrail tests and 164 new edge cases. The test needs v1's private source, so it skips on a fresh copy of the repository.
• A refusal or a budget stop clears candidates, steps, upgrade options and maintenance items; tier labels can be lowered by code, never raised; an observed code still narrows the answer.
• A stored fact needs a word for word evidence sentence from its page; a fact that fails is dropped, not downgraded.
• The MCP server keeps them too: the same recorded case through the command line and through its tools writes the same run record, and in offline tests against fake paid services a live call refuses every missing, wrong, reused, foreign or changed confirmation, with nothing sent and no ledger row (Part 7).
• On 17 October 2026, on the code of build 841df66ad9669ef8, all 3,696 offline tests passed with the network blocked. A full run the same day caught all 1,692 planted bugs. A planted bug is a deliberate small break in the code, used to check that the tests notice it.

Criteria, bars and results

Times are each run record's processing time; the repeat's evaluation record, timed from start to finish, gives 9.9 s.

What the newer model changed

Cheap mode moved from Claude Haiku 4.5 to Claude Haiku 5.5 because the older model could retire from 15 October 2026 (Part 5), and research effort was set to high. The same ten first lookups, five on the hot tub's code and five on the air conditioner's fan, compare the two.

Every refusal came from the model at the writing step, where Haiku 5.5 declined when its only sources were the maker's general pages or a dealer's code table cut off before the action. The tool's own instructions to the writer (rule 6) ask for documentation of this exact model or its documented family and forbid general knowledge of the equipment category. Haiku 4.5's air conditioner answers, all five here and both in its demo, cited Trane's general troubleshooting pages, written for any unit, which that rule does not allow, so its 10 of 10 includes answers the rule forbids. Haiku 5.5 is not consistent either: its three air conditioner answers cite the same kind of page, and two of its refusals came on similar sources. No code check enforces the rule; only the model applies it. I shipped Haiku 5.5 at 7 of 10 and kept the rule as written, and both results are reported as measured. Haiku 5.5 was slower than Haiku 4.5 in every set of timed runs.

What the live numbers can and cannot say. Three refusals out of three is consistent with a true refusal rate as low as about 37% (one sided, 95% confidence), and three repeat questions and ten first lookups are small samples. Only the hot tub code repeat shows memory replacing search; the other two repeats ended with no reliable answer, one held to 2 searches by the top up limit and one with nothing in memory to use. The speed and cost figures sit beside v1's (57 to 117 seconds, and an estimated $0.35 to $0.71), but v2 changed the model, the search provider, the search cap and the pipeline at once, and the move to Haiku 5.5 changed the model and its research effort together, so they do not say which change made the difference.

Part 9

Design decisions, and the safety check

Four design choices shape how memory and replay behave. Each is there for a reason an owner or a technician would recognize. How they came about is recorded in the project's decision log (DECISION-LOG.md in the source).

The safety check, by a person's reading

Jev still catches the most safety steps by a person's reading, but with 26 safety steps its lead over the word lists, and over Claude asked the same question, sits inside overlapping 95% intervals, and about half of its flags are steps that person would not call safety steps. The person is me: on 15 October 2026 I labeled the 175 held out steps blind, and the comparison with Claude was registered before it ran. Every other safety result on this page, the live check included, is scored against the labels of three model readers, and by their labels Jev's lead over the published word list is clear. Both sets of results follow: the check and its test against the readers first, then my labels, the Claude comparison and a paired test, which compares two checks step by step on the same steps, counting only the steps where they disagree.

Why safety steps needed a check in code

The instructions ask the writing model to flag each safety step, and code sorts flagged steps to the top. Code can reorder flagged steps but cannot judge which steps need a flag: deciding what counts as a safety step means either trusting a judgment or writing a rule that judges the step's wording. Nor can code add a step, because every step must cite a source. So a step the model leaves unflagged stays unflagged unless something checks it, and testing such a check was the first item in Part 10.

A safety check in code, tested before it shipped

On 29 and 30 September I tested three ways to catch a safety step the writing model leaves unflagged, each able to add a safety flag but never remove one or add a step: a word rule that flags any step naming certain words (the published list was breaker, power and heat; a tuned list, chosen on the tune half described below, drops the bare word heat but keeps heater and heating, and adds wire, panel, door, disconnect, electrical, jumper and box); Jev, a judgment model from TypeSafe that reads each step and answers how likely it is to be a safety step, as a probability from 0 to 1; and Jev together with the tuned word rule. The test set was 345 steps, taken from maker and dealer documentation and from the tool's own earlier briefs, and labeled before anything was scored by three model readers working blind, who agreed on 338 of them. The set was split into a tune half, where each check's settings were chosen, and a held out half of 175 steps, 31 of them safety steps, which no setting ever saw. The settings were locked in a file before the held out half was scored, once, and the rule for choosing what ships was written before any result: of the tuned word rule, Jev, and Jev together with the tuned word rule, drop any whose precision (the share of its flags that were real safety steps) is below 0.60, then take the highest recall (the share of real safety steps it flagged), then the higher precision. Separately, on 29 September, before the live check, the instruction to the writing model was revised to spell out which steps count as safety steps: power or gas switched off, on or reset; working near anything that can be live, hot or carrying gas; and running or testing a heater.

The held out half has no steps the tool wrote, so the writing model's own flags are not in this table; the ten live lookups below measure the writing model. With 31 safety steps the sample is modest. A 95% interval is the range the true rate probably falls in, given how few steps were tested. By the readers' labels, Jev's recall (0.838 to 0.994) sits clear of the published word rule's (0.469 to 0.789) and barely overlaps the top of the tuned rule's (0.534 to 0.839), and the precision intervals of all four checks overlap. The tuned word rule plus Jev flags every step Jev flags; its extra flags caught no safety step Jev missed, which is why its precision is lower.

What shipped, and the live check. Jev and the tuned word rule plus Jev tied on recall, and Jev alone had the higher precision, so Jev ships, flagging a step when it answers 0.51 or higher. Then the ten new first lookups ran live on Claude Haiku 5.5 (build d24ed7c475dcd813), five with the hot tub's FLO question and five with the air conditioner's fan question. The 7 that answered held 18 steps, 11 of them safety steps by the same three readers, and all 11 were flagged, with 3 false flags (11 of 14). The writing model, under a revised instruction that spells out which steps count, flagged 10 of the 11 by itself; Jev flagged 10 of the 11 with 1 false flag, and the one flag Jev raised that the writer left off was the safety step the writer missed. The tuned word rule, which ships only as the backup for a failed Jev call, would also have flagged all 11 (11 of 14). There is no clean figure for the writer before the revised instruction, so no before and after comparison is reported. The ten lookups cost $0.0750 and 35 credits.

What these numbers cannot say. The labels in the table above came from three model readers, not from people; a person's labels, below, give a less certain result. The tuned word list did not hold up: it scored precision 0.955 on the half it was tuned on and 0.647 on the held out half, below the published list, because words such as "door" and "box" found harmless steps in pages it had not seen. The live check is small, 11 safety steps from two questions, 10 of them on the hot tub and mostly at a breaker, so the held out half is the stronger evidence for the check itself. With Jev alone shipped, the tuned word rule now backs up any step whose Jev call fails: it can add a flag to that step, never remove one, and the brief still carries the line telling the reader to treat any step at a breaker, a panel, a heater or a gas valve as a safety step. The backup is weaker than the check it stands in for: held out, the tuned word rule caught 22 of 31 safety steps (recall 0.710) at precision 0.647, against 30 of 31 (0.968) at 0.732 for Jev. It was picked over the published list, 20 of 31 at 0.714, on 1 October, after the held out scores were known. The published list was not one of the choice rule's candidates, but the pick follows the rule's order: both lists have precision 0.60 or higher, and the tuned list has the higher recall. Build d24ed7c475dcd813 includes the backup, but every Jev call on it answered, so the backup raised no flag live and is shown working only by the offline tests. The demo shows Jev's check at work on case 1: the writing model flagged 3 of the brief's 4 steps, including turning the power off at the breaker and back on, and Jev agreed with all three (0.98, 0.77 and 0.96) and added no flag of its own.

Checked against a person's labels

On 15 October 2026 I labeled all 175 held out steps myself, on a local labeling page that showed each step with the instructions, worked examples and hazard choices the model readers had, and no reader's vote or check's flag. My answers were sealed before anything was scored (Appendix A). I agreed with the readers' majority on 162 of the 175 steps (92.6%), a Cohen's kappa of 0.728 (agreement corrected for the agreement chance alone would give, where 1 is perfect and 0 is no better than chance). I called 26 steps safety steps; the readers called 31. The same checks, with nothing retuned, scored as follows against my labels.

The rows match the readers' table above, in the same order, with two differences: the tuned word rule plus Jev, which did not ship, was not scored against my labels, and Claude, which ran on 15 October, is new; its figures by the readers' labels are in the next section.

Jev flags the most safety steps, 22 of 26, but its interval overlaps those of both word lists and of Claude, and its precision fell from 0.732 by the readers' labels to 0.537 by mine: about half of its 41 flags are steps I would not call safety steps. The 4 safety steps Jev missed by my labels are the ones I called safety steps and all three readers did not, so Jev missed only where the readers and I parted. By my labels the tuned word list, which backs up a failed Jev call, caught no more than the published list and flagged more steps that were not safety steps; it was chosen on the readers' labels, and nothing here changed it. After seeing the 13 steps where the readers and I disagreed, I changed my mind on 3, and 2 of those now match the readers; the changes are kept apart, and every number above uses the sealed blind labels.

Claude, asked the same question

The second comparison asks whether a general model does Jev's job as well. Before it ran, its settings were fixed in code: Claude Haiku 5.5, the model cheap mode already uses, asked Jev's exact question about each of the same 175 steps, one call per step, with the settings cheap mode uses to classify a vague symptom, including effort (how much the model reasons before answering) set to low. A "true" answer is a flag, with no threshold to tune. It ran once, on 15 October 2026 on build 179b4295c98a3143 (ledger run claude-judge-152437a0a5ed): all 175 steps were answered, none declined, for $0.0182.

By my labels Claude caught 19 of 26 at precision 0.576 (table above), inside the same overlapping intervals as Jev. Every safety step Jev missed by my labels, Claude missed too, along with 3 more. By the readers' labels Claude caught 28 of 31 (recall 0.903) at precision 0.848, against 30 of 31 at 0.732 for Jev. Claude and Jev agreed on 163 of the 175 steps (93.1%), a kappa of 0.795. By the ledger, Claude cost $0.000104 a step against $0.000028 for Jev, about 3.7 times as much, and took a median 1.08 seconds a step (90th percentile 1.67 seconds).

A paired test on the same steps

Overlapping intervals judge each check by its own total. A paired test is sharper: it compares two checks step by step on the same steps, counting only the steps where they disagree, here the safety steps one caught and the other missed, and the false flags (flags on steps that were not safety steps) one raised and the other did not. Its p value is the chance of a split at least this lopsided if the two checks were equally good, so the smaller it is, the harder the split is to put down to chance. A common bar is a p value under 0.05. I chose this test after the results above were known; unlike the Claude comparison, it was not set before the run. It reads only the recorded answers and cost nothing.

By my labels, Jev caught 3 safety steps that Claude missed, and Claude none that Jev missed (p value 0.250). Among the 149 steps I did not call safety steps, Jev raised 7 false flags that Claude did not, and Claude 2 that Jev did not (p value 0.180). Against each word list, Jev caught 4 safety steps the list missed and the list none that Jev missed (p value 0.125 for each list). On false flags the lists do better: Jev raised 17 that the published list did not, against 8 the other way (p value 0.108), and 12 that the tuned list did not, against 9 (p value 0.664). By the readers' labels, Jev caught 3 that Claude missed and Claude 1 that Jev missed (p value 0.625), and Jev raised 7 false flags that Claude did not against 1 the other way (p value 0.070). Jev caught 11 safety steps the published word list missed, against 1 the other way (p value 0.006), and 8 that the tuned list missed, against none (p value 0.008).

So by my labels the paired test agrees with the intervals: every split on catches that involves Jev leans toward Jev, but with 26 safety steps none is under that bar, nor are Jev's extra false flags against Claude or the word lists. By the readers' labels, Claude's edge on false flags, 7 to 1, comes close to that bar but does not clear it. The splits under that bar are all by the readers' labels: Jev's leads over the two word lists, and Claude's over the published list, 9 safety steps to 1 (p value 0.021). They need care: the scorer reports 24 such comparisons (every pair of the four checks, by both sets of labels, for catches and for false flags), with no correction for their number, and with that many tries chance alone can produce a lopsided split; a common correction, dividing the 0.05 bar by the 24 comparisons, leaves none of them under it. How the test was run is in Appendix A.

What these comparisons cannot say. One person labeled the steps, and I designed the tool; I am not a technician, and a technician might draw the line elsewhere. With 26 safety steps every interval is wide, and by my labels the paired test, chosen after the results were known, shows no check beating another on safety steps caught or on false flags. The readers' labels came from Claude models, which may favor Claude: by their labels Claude's precision is above Jev's (0.848 against 0.732), and by mine the two are close (0.576 against 0.537), the pattern such a bias would produce, though it does not prove one. Claude ran at the cheap classifier setting, effort low, so a higher effort or a larger Claude model might score differently; neither was tried. Nothing was retuned, and Jev still ships, flagging at 0.51.

Rules that stand

• Speed is reported, not scored.
• A budget stop never counts as a refusal.
• The retry is one smaller second pass, taken only if affordable.
• Misses are published as misses, and each live result is reported only from the build it ran on.

Part 10

What I would do next

1. Decide how safety steps get flagged.Done Jev now checks every step to try first: by the model readers' labels it flagged 30 of 31 held out safety steps against 20 for the published word rule and 22 for the tuned one, and all 11 safety steps in ten new live lookups were flagged, the writer catching 10 and Jev the eleventh (Part 9).
2. Put one brief in front of one technician and ask a single question: did this change what you would bring. Neither version has answered it, and it is the only question that turns process quality into evidence.
3. Check the safety labels against a person's.Done I labeled the 175 held out steps blind on 15 October 2026 and agreed with the readers on 162. By my labels Jev caught 22 of 26 safety steps, but its lead over both word lists and over Claude asked the same question is inside overlapping intervals, and about half its flags were steps I would not call safety steps (Part 9). Two checks would settle more: a technician's labels, since I am not one, and a larger labeled set, since with 26 safety steps even a paired test, which compares two checks only on the steps where they disagree, cannot separate them: Jev's widest lead by my labels, 4 safety steps to none over each word list, has a p value of 0.125, and the published list's lead on false flags, 8 to Jev's 17, one of 0.108.
4. Enforce the source rule in code. Both models answered from the maker's general pages, which the writer's rule does not allow, and only the model applies that rule. A code check that a cited page names this model or its family would make it hold on a bad day.
5. Keep the steps, not just the meaning. Memory keeps a code's meaning, so both answers from memory listed no steps to try first. Storing each documented step with its quote would let a repeat answer carry them.
6. Match product recalls to serial ranges. The check matches model names and tells the owner to check their serial number; reading serial ranges from free text is error prone, so it waits.
7. Run v2 on a real property. The seed data is 2 properties and 7 appliances, all labeled synthetic. Three use real equipment models (the hot tub, the air conditioner and the water heater), and four have invented makers and model names.
8. Measure speed and quality with one thing changed at a time, rather than inferring either from a comparison that changed four.

Part 11

What I cut

v1's spec named these as out of scope before its build started, which is the only time a cut list is honest. Two came back in v2, for the reason given in Part 4.

• Accounts, and anything resembling a funnel. Still cut. This is a validation instrument, and email capture would say nothing about the question.
• Several properties.Back in v2 Memory and service history only pay off per property.
• Warranty tracking and prevention scheduling.Back in v2 Coverage status is now computed in code from each component's term, and maintenance due from the maker's intervals.
• Integration with existing operations tools. Still cut. That is the business, not the test.
• The cleaner capture workflow. Still cut. The right long term entry point, and a second unproven assumption stacked on the first.
• A name and any branding. Still cut, so nothing rides on presentation.

Out of v2's scope from the start:

• Paid databases, a hosted graph database, vector stores and hosted servers. None buys anything at this size (Part 5).
• Repair versus replace dollar amounts. The tool never estimates a price a source does not give.
• A live demo. The recorded v2 demo replays live runs of 14 October 2026 verbatim and makes no API call, so it is free to visit and free to host.
• A mobile capture flow, and full mode live.
• ENERGY STAR efficiency ratings (the U.S. government's efficiency label) and links to technician feedback. Deferred by the second set of requirements.

Appendix A

Method made visible

v1: reviewing my own build adversarially

Before shipping v1 I reviewed the build against its own spec across three lenses: does it match the specification, does it use the platform correctly, and what breaks. A separate pass tried to refute every finding. Thirty five claims, thirty one confirmed. The ones that mattered were an intermittent failure on exactly the search heavy cases the product exists for, output budgets that would have cut a brief off mid sentence, a bug that corrupted citations when reassembling the response, and a script injection hole in the shared brief, the file handed to a third party.

v2: gates, replay first, and a typed "proceed"

• The build ran in phases. Replay phases came first, at no cost; live phases ran only after a typed "proceed", with the planned calls and spend printed first.
• The ledger reserves each call before it is made, against a $5 cap for the whole build.
• Every library name was checked against the current release before use.
• Planted bugs test the tests: a full run on 17 October 2026, on the code of build 841df66ad9669ef8, caught all 1,692.
• Every run record carries a fingerprint of the code that produced it, so each live v2 result on this page traces to its build: the live evaluation and the demo to build d24ed7c475dcd813, the product recall check to build ad11e6e15a55d30a, Haiku 4.5's comparison figures to builds da738a1559ced359 and 9a64835e86a5a6c8, the labeled set scores to Jev calls on 30 September under the locked settings and model jev-1.13.0, the Claude comparison to build 179b4295c98a3143, the live MCP brief run by a test program to build 17427277685ed433, and the live Claude Desktop brief to build 841df66ad9669ef8. The whole build spend is the one total that also counts runs on other builds.
• The MCP server was built and tested offline first, in replay and against fake versions of the paid services. Its first real connection ran in replay, at no cost. Only then did one live brief run, which I approved in advance at about $0.01. No person was in that run: a test program played the owner, answering "proceed" to each of its two preflights only because each was within limits set beforehand (a typical cost of at most $0.02 and a worst case of at most the $0.15 cap), and confirming the unit the plate shows. Later the same day, Claude Code called the tools in a free replay session. Last, one live brief ran in Claude Desktop, which I approved in advance and confirmed myself, typing "proceed" after reading its preflight; it ended with no reliable answer.
• The safety check was scored on a labeled set built before any scoring, with its settings locked and its choice rule written before the held out half was scored once.
• The check of those labels against a person's ran blind on 15 October 2026. A local labeling page showed me each held out step with the instructions, worked examples and hazard choices the model readers had, and no reader's vote or check's flag; an answer could be changed until sealing, and the latest one stands. The answers were then sealed by a hash (a fingerprint of a file's contents that changes if one byte does) before any score existed. Once they were sealed, the labeling page would not start again, and the scorers check that the labels still match the seal. The 3 answers I changed after seeing the disagreements are kept in a separate file and scored nowhere. The locked settings file and Jev's settings were not touched, and nothing was retuned. Eleven of my yes answers named only "other" as the hazard, which the protocol counts as no, exactly as it counts the readers' votes.
• The Claude comparison was registered before it ran: its settings were written down and fixed in code (settings fingerprint 658f072743accb13), with Claude Haiku 5.5 asked Jev's exact question and criteria, one call per step, effort low, a single true or false answer, and no thinking, sampling or fallback setting; a true answer is a flag, with no threshold and no tuning; a decline would count as not flagged and be reported apart; one run only, retried only on a lost connection; scored against my sealed labels and the readers' majority, beside Jev and both word lists. It was planned at about $0.02 and capped at $0.15, and cost $0.0182. Its run record names build 179b4295c98a3143, and its spend is ledger run claude-judge-152437a0a5ed, both of 15 October 2026.
• The paired test was added on 16 October 2026, after every result above was known, and was not registered before any run. It is McNemar's exact test, the standard paired test for two checks' yes or no answers on the same steps, run by the Claude comparison's scorer on the recorded answers only, with no call and no cost. The scorer reports all 24 comparisons in its scores file, with the steps behind every split, and makes no correction for their number. Nothing in the shipped check changed.

What I got wrong

• v1: I pinned a dependency version from memory instead of checking, and the build failed on the first compile.
• v1: one of my own tests was wrong. I checked the actual behavior in a browser before changing the assertion.
• v2: the price check once accepted "$12" from a page that said "$129", caught before any live run.

Honest limits of this write up

• No technician has used either version.
• The recorded v2 demo replays live runs of 14 October 2026 with no API call; four of its eight cases end with no reliable answer, as recorded. The four v1 briefs linked in Part 3 remain the record of v1.
• The v2 property records and warranty rows are synthetic. The property, the owner and the commercial detail are genericized; the equipment and the retrieved documentation are real.
• The product recall check matches model names, not serial ranges, and has run live once.
• Cost comparisons with v1 are estimates computed from v1's logged usage, not measurements of v1's bill.
• The safety step labels came from three model readers and one person, me; no technician has labeled them, and the readers were Claude models, which may favor Claude in the comparison.
• The MCP server has run two live briefs: one through the official library's own client driven by a test program, with no person in it and no search or Jev call, and one in Claude Desktop, which researched and ended with no reliable answer, so no live brief through an assistant has yet carried steps or safety flags. Claude Code has called its tools only in replay, no form has yet reached a person outside the tests, and an assistant's retelling of a brief is outside the tool's control.

Appendix B

Terms used on this page

• API: a service the code calls over the internet, paid for by use.
• LangGraph: a library that runs an AI workflow as a graph of steps; each step reads and writes a shared record of the run, and edges decide what runs next.
• LangChain: a library of building blocks for AI apps: model connectors, tools, and a ready made agent loop.
• Checkpoint: a saved snapshot of the run, so a run can stop and continue later.
• Interrupt: a deliberate pause that waits for a person's answer.
• Middleware: code that wraps every model call or tool call and can check or stop it before it happens.
• Schema: the exact shape an answer must have: which fields, of which types.
• Validator: the code that checks each brief against the rules and clears or rejects what fails.
• Tool calling: the model asking the code to run a tool, here a search or a page fetch, and reading the result.
• Structured output: the model answers in a fixed, typed form instead of free text.
• Knowledge graph: facts stored as things and the links between them, each here carrying the exact sentence that supports it.
• Replay: running the whole pipeline on recorded model replies and search results, at no cost.
• Ledger: the running record of spend, checked before every paid call.
• Planted bug: a deliberate small break in the code, used to check that the tests notice it.
• Token: a word fragment; models are billed per million tokens.
• Build: a single frozen version of the code, identified by a fingerprint stored in every run record. The live evaluation and the demo ran on build d24ed7c475dcd813, the product recall check on build ad11e6e15a55d30a, the Claude comparison on build 179b4295c98a3143, the first MCP brief on build 17427277685ed433 and the Claude Desktop brief on build 841df66ad9669ef8; Haiku 4.5's comparison figures come from builds da738a1559ced359 and 9a64835e86a5a6c8; "the whole build" spend is the project's total.
• Research effort: a setting for how much the model thinks and how long it keeps researching; high for the research step here.
• Warranty status: whether each warranty component is active on the run's date, worked out by code from its term and the install or purchase date.
• Successor model: the model a maker names as replacing a discontinued one.
• CPSC: the U.S. Consumer Product Safety Commission, which runs a free public service listing product recalls.
• Product recall match: a product recall that prints the unit's exact model number; a similar model never counts.
• HVAC: heating, ventilation and air conditioning equipment.
• Jev: a judgment model from TypeSafe that answers a yes or no question as a probability from 0 to 1; here, whether a step is a safety step.
• Recall: the share of real safety steps a check flagged.
• Precision: the share of a check's flags that were real safety steps.
• Held out half: the half of a labeled set kept aside from every setting choice, so its score is a fair test.
• Blind labels: labels given without seeing anyone else's labels or any check's flags.
• Cohen's kappa: agreement between two labelers corrected for the agreement chance alone would give; 1 is perfect agreement and 0 is no better than chance.
• 95% interval: the range a true rate probably falls in, given how few cases were tested; when two checks' intervals overlap, the intervals alone have not shown which is better.
• Paired test: a test that compares two checks step by step on the same steps, counting only the steps where they disagree; here, McNemar's exact test.
• p value: the chance of a split at least as lopsided as the one seen, if the two checks were equally good; the smaller it is, the harder the split is to put down to chance, and when many comparisons are run, a small one can still turn up by chance.
• Hash: a short fingerprint of a file's contents that changes if one byte changes; sealing the labels by hash shows they were not edited after scoring began.
• MCP: the Model Context Protocol, an open standard that lets an AI assistant, such as Claude Desktop or Claude Code, call outside tools; here, the advisor's four tools, served from the owner's own computer.
• Preflight: what a command or tool shows before a call that costs money or goes to an outside service: the planned calls and, for a paid call, their typical and worst case cost, the run cap and the spend so far. Nothing goes out until the answer is exactly "proceed".