It is easy to feel productive because more code is being produced.

But when you inspect outcomes, the improvement may be smaller than expected.
Why?

Because AI increases implementation speed but product work isn’t just processing requests. User has a problem, ship obvious fix, repeat. There is more to building good software than this.

Good product judgment involves finding one abstraction that solves many problems. Deciding which solution will stay elegant after the next ten requests arrive.

Elegance is hard to model. Depends on many not-so-explicit factors.

AI doesn't help directly with that skill. Still depends on experience, code mental models and discussions.

Dax says he used to spend 95% of energy thinking and 5% doing.

Now maybe it is 96% thinking and 4% doing.

The joke carries the point: AI speeds up implementation, improves execution, but the hard judgment work remains. AI does not automatically speed up the work of finding good abstractions.

So, the takeaway advice is: Generate code fast, but slow down enough to build foundations. Then again move faster from a better base.

I wrote earlier about how Vercel advocated for ruthless judgement when shipping AI-assisted code.

Dax’s insight takes it one step further. We need to ask:

• Before fixing: what foundations need re-thinking due to these issues?

• Before shipping another fix: did we strengthen the foundation or add another layer on top? Did we solve the class of problem or just the latest instance? Did we find the deeper structure behind it?
  

The title is dramatic. The idea is more interesting than the title.

At first glance, it seems to oversell the idea of the chat "logs" - all the inputs, outputs, tool calls etc that accumulate during a conversation with Agent.

Yet another graph memory paper bolted on to the LLM.

But that's not the point. What it says is that:

• the activegraph is the single source of truth of everything that happened.

• the harness, the imperative loop around the LLM isn't in there explicitly.

• you can trace from any observation back to exactly what caused it, full details

• if the conversation bumps into an error state, you can create alternative futures by branching and tweaking the prompt without replaying entire history. This isn’t natural to do now.

So what drives the agent?
Implicit reactive events.
tool call event -> append to state
output created -> update memory.
prompt updated -> fork and re-run

A declarative set of harness rules sit outside. There is no explicit orchestrator for your Agent - it is in the BIOS of activegraph. Works through implicit reactive events that follow the harness rules.

What's (or could be) cool about this?

• Implicit reactive events run the system. Not an easy-to-complicate imperative harness loop. Any fix automatically triggers a deterministic cascade of correct downstream behaviors, rewriting the entire suffix of the graph purely through reactive dataflow.

• Behaviors and rules are treated as data inside the log, you can append a behavior.updated or prompt.patched event mid-run.

• enables creating different "futures" with branching. and comparing the futures. Notice a bug? fork and run with the new prompt, reusing all the history.

• all the provenance, what happened after what, is preserved. enables explicit, precise debugging.
  

Clearly, this creates a humungous graph. How does a human visualize it? The graph isn't a UI feature for humans to stare at; it's a data substrate for the agent.

Ask a query to the active graph: how did we arrive at fact X. and it slices through the graph to render the relevant history for you.

Because every object carries strict provenance, if a final report says, “We recommend avoiding this investment due to regulatory risk,” a UI wouldn’t show you the 700-node operational graph. It would isolate and render a single linear path showing only the nodes directly connected to that claim:

[User Goal] ➔ [Planner Behavior] ➔ [Research Question] ➔ [Document Source] ➔ [LLM Extraction] ➔ [Final Risk Claim]

By filtering out the other 95% of the graph that didn’t contribute to this specific conclusion, the visualization becomes incredibly clean and auditable.

But the big graph needs compaction. Can't keep appending. If a run hits 1,000,000 events, replaying it from scratch every time would make the system crawl. This graph bloating is a key bottleneck and work in progress.

There is a GitHub implementation too. Not as popular as BabyAGI yet.

But I like the direction:

Traceability, Branching, and Memory are not three separate features here. They emerge from the same graph. Code (rules) and data (memory) live together.

The legal AI product market is already crowded. Major names: LexisNexis Protégé / Lexis+ AI, Thomson Reuters CoCounsel / Westlaw, Harvey, vLex Vincent AI / Clio, Bloomberg Law, Spellbook, Legora, DraftWise, Luminance, Kira, and Icertis.

In spite of the crowded market, legal AI is far from being solved. Stanford/Yale researchers found that RAG-based legal research tools reduced hallucinations versus general chatbots, but still hallucinated 17% to 33% of the time across LexisNexis and Thomson Reuters tools.

On popular forums like hackernews, the sentiment is split between “huge productivity tool” and “dangerous unless verified.” AI can lead them to relevant law faster, but every citation must be checked in Westlaw/Lexis before filing.

Reddit forums also confirm repeatedly that AI helps with quick direction, summaries, and first-pass research, but lawyers still must verify every source. One thread notes enterprise tools drawing from actual statutes/case law can be useful, but “it can take some massaging” and “All responses have to be verified 24/7”.

Interestingly, small law offices turn to AI for relief from overwork. Law firms of all sizes use AI to draft court filings. From this Stanford cyberlaw article:

As the 404 Media article observed, “the legal industry is under great pressure to use AI.” To overworked attorneys at small law offices, these tools must seem like a godsend.

What makes Legal AI so hard?

First, the query complexity is mind-boggling.

See the slide below from a Harvey presentation. Typical queries involve both semantic and keyword components, domain-specific vocabulary, spanning many specialized legal corpora, and may have many parts to them.

This needs a full-blown, iterative Agentic search with well-tuned keyword and semantic search engines, filters applied correctly and at right place to scale search, and planner knowing correctly which legal sources to localize to. Building such a powerful Agentic search system is technically quite challenging.

Why do we need Agents for Legal RAG

Now, query complexity isn’t the only reason we need sophisticated Agents. Legal work has a broad span: besides QA, also includes research planning, drafting, review, and iterative refinement, and considerable human-in-the-loop operation:

• Planner agents break a legal research question into sub-questions.

  • Users can review and edit those sub-questions before the query is sent to search Agents for final answer generation.

  • Having human-in-the-loop enables about 20% improvement in results.

• Reflection agents are used for drafting: critique the draft, find missing clauses or legal issues, then revise.

• Therefore, the Agentic system must know when to retrieve, reason, critique, validate, and escalate.

The second issue is far bigger.

Legal work often has no single ground-truth answer, especially drafting and complex reasoning.

A case/citation retrieved by the Agent can be relevant but legally unusable.

Why?

• It may be overruled, from a lower court, jurisdictionally weak, stale, or not citable.

• Pure vector/semantic search does not understand legal authority hierarchy.

• “Accuracy” alone is a weak metric because an answer can look correct but miss authority, citation quality, comprehensiveness, or hallucination risk.

This makes the Agent’s task doubly hard. Finding plausible citations that satisfy the complex query requirements is not enough.

Standard Agentic RAG optimizes for relevance.
Legal users need authority, citability, completeness, and defensibility.

Can Knowledge Graphs (KGs) help with this issue?

• After vector search finds contextually relevant content, a knowledge graph may be used to improve the quality of the filtered content, separate relevant from the legally useful

• A KG adds legal relationships. It represents cases, statutes, courts, jurisdictions, citations, overruling, distinguishing, following, and hierarchy as connected nodes/edges.

• GraphRAG retrieves semantically, then validates structurally. First use vector/keyword search to find candidate material; then traverse the graph to check whether those materials are legally valid and authoritative.

  • It filters weak matches. A semantically relevant but overruled or non-citable case can be demoted or removed.

  • It boosts stronger authority. Binding precedent, higher-court decisions, current statutes, and jurisdiction-matched sources rank higher than merely similar text.

  • It improves context completeness. The graph can pull related controlling law, parent/child doctrines, cited authorities, exceptions, and contrary cases that vector search may miss.

  • It makes citations defensible.

Knowledge Graphs clearly are a silver bullet in this space. However, their usefulness depends on finding all applicable precedents, decisions, exceptions, stale citations, court hierarchy, jurisdiction fit, and so on. Building a comprehensive knowledge graph is computationally intensive and requires careful construction of the graph and relationships. Missing, incomplete retrieved information can be severely risky for the legal professionals.

How do we evaluate the quality of retrieved citations?

Conventional Agentic search relies on automated evals: metrics like accuracy, faithfulness, recall, precision, etc. are used on curated datasets to evaluate.

However, accuracy isn’t well-defined in Legal QA. Legal RAG needs a “usefulness score” that includes multiple facets: answer relevance, legal authority, citation accuracy, comprehensiveness, current validity, jurisdiction fit. hallucination rate, practitioner usefulness.

In essence, Legal AI must be evaluated like a lawyer would evaluate work, not like a simple QA benchmark. LLM-as-judge can help triage, but final hallucination checks should be grounded in retrieved sources and legal metadata. This makes building Legal AI benchmarks even harder.

Since RAG took the limelight, the legal domain has been considered one of its most promising applications. Yet despite a crowded market of Legal AI products, the problem is far from solved. To summarize:

Standard RAG is only the first layer for legal AI.

• Legal queries are complex.

• Standard Agentic RAG optimizes for relevance; legal users need authority, citability, completeness, and defensibility. The results need not only be relevant but also legally usable.

• GraphRAG can exploit deeper legal metadata (courts, jurisdictions, overrulings, …) to improve results but building comprehensive graphs is expensive.

• Need a suite of metrics to evaluate usefulness of citation quality. Useful benchmarks are harder/expensive to create.

• Finally, human verification is essential for evaluating usefulness. How to minimize the human effort remains to be a challenge.

What makes Legal AI so hard is a mix of hard queries, relevance being a poor proxy for legal usefulness, absence of realistic benchmarks and difficult evaluation. In law, any relevant source is not necessarily a useful or citable one, and judging output quality often requires expert human review.

Today’s products solve parts of this problem, but many still need careful prompting, verification, and user training. The next generation of Legal AI will need to move beyond finding plausible answers toward producing work that is authoritative, needs less verification, and is genuinely useful to law professionals.

Agents with terminals are suddenly everywhere. Devin, Cursor, Manus, Claude Code, Codex, and several newer agent systems all point in the same direction: give the model a workspace, give it tools, let it act.

At first glance this sounds like a leap forward in AI capability. But if you watch what these code agents actually do, their actions look quite primitive.

They list files. They search text. They read snippets. They run tests. They inspect errors. They edit a file and try again.

ls # list
rg "deleteDocument|delete_document" # grep
sed -n '1,160p' src/routes/documents.ts # read
npm test -- documents # test

This isn’t trivial. The terminal gives the agent a way to interact with the codebase through stable, named operations, bash commands. It does not have to hold the entire repository in context. It can ask the repo questions.

That makes code agents feel less like a single giant model “understanding” a project, and more like a fast junior developer with a very patient shell: look around, search, read, try something, observe the result.

They use the boring but extremely powerful and scalable UNIX tools to compose together an insightful trail that leads to code understanding and the final answer.

Localization: Where Does the Answer Live?

Before an agent can fix, explain, or modify code, it has to localize the concern:
Where in this repo does the answer live?

A user asks:

How does this app send email?

The answer may be spread across:

mailer.py
send_email()
send_welcome_email()
settings.SMTP_HOST
users/views.py
auth/views.py

This is the localization problem: turn a human question into a searchable handle, the seed keywords. That might be a domain word, UI label, error message, route, function name, or config key.

The seed does not need to be final. It only needs to reveal the next one. Code search is not just lookup. It is translation from the user’s vocabulary into the repository’s vocabulary.

If you prefer, I created a short Code Search Demo video describing this example in more detail

Tracing Through Names

To answer the above question, the agent starts with a broad concern, then narrows it into the specific files and functions where the behavior lives.

It might start wide, with the powerful tool recursive grep (ripgrep, rg):

rg -l "email" # all file mentions of email

result:

src/notifications/mailer.py
src/users/views.py
config/settings.py

mailer.py looks to be the key. The next question verifies that. What functions it defines:

rg -n "def " src/notifications/mailer.py

4:  def send_email(to, subject, body):
19: def send_welcome_email(user):
26: def send_password_reset(user, token):

Now the search has found a likely base function send_email:

email -> mailer.py -> send_email()

Reading just that function reveals the next names:

rg -A 8 "def send_email" src/notifications/mailer.py # read send_email 

with smtplib.SMTP(settings.SMTP_HOST) as s:
    s.send_message(msg)

Now the search terms are no longer guesses. They came from the code: smtplib, SMTP_HOST, send_email.

rg -n "send_email\\(" --type py
rg -C 1 "SMTP" config/settings.py #verify settings.py contains SMTP settings

The trace becomes:

email
-> mailer.py
-> send_email()
-> smtplib + SMTP_HOST
-> signup + password reset callers

After a few searches, the answer is localized and grounded:

The app emailer uses Python’s smtplib, configured through SMTP env vars, and is triggered by signup and password reset.

This is the key idea. ripgrep does not understand email. It helps the agent follow the names that the program already uses to connect the feature together.

The Agentic Search Loop

The email example has the basic code-agent loop:

seed term -> search -> inspect -> discover better term -> search again

A result might reveal a filename, function name, import, route, error string, or test. Any of those can become the next query.

Good seed terms usually come from the terms closest to the user:

domain words: email, invoice, document
UI labels: "Send invite", "Delete document"
routes: /documents/:id, POST /login
errors: 403, forbidden, invalid token
config: SMTP_HOST, DATABASE_URL
tests: password reset, rejects empty name

Each step narrows the question: from rough domain language, to repo vocabulary, to the files where behavior actually lives.

Tests add another source of names. They are search amplifiers. A failing test might mention the exact route, expected status code, fixture, or error string:

test failure -> expected 403 -> search "403" -> find policy check

Finally, how do we know the loop is converging? The match set gets smaller. The same files keep recurring. The names in code start lining up with the user’s question.

ripgrep: The Detective Workhorse

ripgrep is popular with code agents for a simple reason: it is cheap enough to use repeatedly and scales to large files.

That matters because agentic search is many small queries, not one big query.

rg -l "email"
rg -n "def " src/notifications/mailer.py
rg -A 8 "def send_email" src/notifications/mailer.py
rg -n "send_email\\(" --type py
rg -C 1 "SMTP" config/settings.py

Each variant answers a slightly different localization question.

rg -l gives the shape of the match set: which files even matter? rg -n gives line numbers, so the agent can jump to the right place. -A, -B, and -C reveal nearby context without reading the whole file. Type filters and globs keep the search inside the relevant part of the repo. Respecting .gitignore usually keeps irrelevant artifacts out of the way.

This makes ripgrep useful in two ways at once.

• First, it is localized content search: find symbols, strings, comments, routes, errors, and config keys.

• Second, it navigates repo structure: which directories mention this? Which file types? Which tests? Which modules keep recurring?

ripgrep is not semantic search. It will not know that send mail and dispatch notification are related unless the code gives it a bridge. Codebases have plenty of bridges: function names, imports, tests, constants, routes, and error messages. In fact, Claude Code abandoned vector embedding based semantic search in favor of keyword based explicit, agentic search.

Why is exact search so powerful? In a repo, names point to dependencies: functions have callers, routes have handlers, modules are imported, errors have branches, and tests have expectations.

In practice, several more optimizations are needed to make code agents scale. Repeated ripgrep queries eat up a lot of tokens and do repeated work. How do we cache? A clever solution is to build a code-map of your entire repository and query it to localize before making explicit ripgrep calls. For example, the code-review-graph tool provides a MCP server to query the local code map.

Beyond Code: Named Interfaces for Agents

Code is friendly to agents because it already has named, inspectable surfaces. The filesystem gives the agent a few reliable moves:

list what exists
search for a name
read the nearby context
run the thing
inspect the result

Interestingly, this search pattern is starting to show up outside code. Agent memory can be written to files. Plans, summaries, and observations can live in a workspace and be retrieved later with the same moves: ls to see what exists, cat or sed to read state, rg to recover something relevant.

But not everything agents need is naturally a file tree. Databases, APIs, conversations, logs, dashboards, and long-term memory all have structure that may not fit cleanly into normal files. Can we extend the same boring search pattern to them?

The design question here is whether they should expose filesystem-like surfaces:

• What is ls for a database? Tables, rows, schemas, recent queries?

• What is grep for agent memory? Content search, metadata search, embeddings, or all three?

• Find some early ideas in the article below.

The lesson from code agents is that agents are effective when the knowledge base is exposed through named operations that allow inspection and localization.

The future of Agentic Search isn’t about giving models bigger prompts or more tools. It is more about giving every knowledge base a surface that allows the agent to localize and inspect at multiple resolutions.

If an agent can read, write, edit, and search files, it can do a surprising amount of useful work. It can inspect the current state of the world. It can modify that state. It can keep versions through Git. It can recover when something goes wrong. It can use commands like `ls`, `cat`, `grep`, `find`, `head`, and `wc` without needing a new API for every small action.

That is a strong primitive.

But the ordinary filesystem has limits.

It is not where most data lives. Your context is spread across Postgres, Mongo, Redis, Elasticsearch, S3, Slack, Google Drive, remote server files, docs systems, logs, tickets, traces, and many other places.

Moving back data from a specialized index back to ordinary filesystem may not be feasible.

Also, normal filesystems are not great at many things agents need: indexing, metadata, filtering, and efficient search over large collections.

Databases already solve many of these problems.

So maybe the answer is not to force everything into files. The better move is to expose databases through the same interface as a standard filesystem.

Database as virtual filesystem

The agent keeps the simple interface. The backend gets to remain specialized.

`ls` can list folders, tables, collections, channels, documents, or query namespaces.

`cat` can read a file, row, document, message, page, or stored context item.

`find` can traverse the virtual hierarchy.

`grep` can search content, ideally by using the backend’s native index instead of downloading everything and scanning locally.

In summary: keep the filesystem API as the agent-facing operating surface, but let the system underneath route those commands to the right database, index, object store, or SaaS tool.

Several recent efforts point in this direction.

Virtual Filesystems for Databases

LangSmith Agent Builder treats agent memory as files: AGENTS.md, skills, tools.json, and knowledge files. The interesting part is that these do not have to live on a literal filesystem. They can be stored in Postgres while still being exposed to the agent as files. The agent gets a familiar workspace. The system gets database-backed persistence and management.

Mintlify built ChromaFs, a virtual filesystem over Chroma. Their docs assistant could use commands like ls, cat, grep, and find over indexed documentation. This made the docs corpus feel like a filesystem to the agent, while avoiding the cost and latency of spinning up a real sandbox for every session.

Leonie Monigatti implmented one with Elasticsearch. The virtual filesystem supports the core Bash commands, but the backing store is an Elasticsearch index. The useful detail is how grep works: first use Elasticsearch regex for coarse search, then do more exact filtering in memory. That is the kind of translation layer these systems need.

Both above rely on just-bash by Vercel Labs, a TypeScript reimplementation of bash that supports grep, cat, ls, find, and cd. just-bash exposes a pluggable IFileSystem interface, so it handles all the parsing, piping, and flag logic while (for Mintlify) ChromaFs translates every filesystem call into a Chroma query.

Mirage pushes the idea further. It tries to expose many data sources as one universal filesystem workspace: S3, Google Drive, Slack, Postgres, Mongo, Redis, remote files, and more. The agent can run familiar Bash-like commands across all of them.

A virtual Bash interface over popular DBs is powerful. The agent does not need a different interaction pattern for every data source. It can inspect heterogeneous systems through one modest interface, without changing its workflow.

What’s the catch?

What does grep mean on S3? Does it download files, stream them, use an index, or run remotely?

What does cat mean on a huge database table? (Mirage turns rows into JSON format)

• What does ls mean over Slack?

• Are permissions inherited correctly from the source system?

• Are search semantics the same across Elasticsearch, Chroma, Postgres, object storage, and Google Drive?

This is where the abstraction gets tricky.

A universal Bash interface for agents is welcome news. The common set of Bash commands: ls, cat, grep, find, head, and wc are boring commands, but boring is often what makes an interface useful. Models already understand them. Developers understand them. Workflows compose around them.

But the hard part is making the abstraction cheap, faithful and predictable enough for agents to rely on it.

If that happens, then we will have extended society of Agents.

• Talking seamlessly to many databases, indices, object stores, and SaaS tools

• Exposed through the simple, modest filesystem interface, which agents already know how to use.

Some of these fixes may help. But unless you know what failure you are fixing, this is mostly expensive guessing. When RAG quality is unstable, the temptation is to keep adding fixes. The better first question is: what exactly is broken?

Most engineers can now recite the common RAG fixes I mentioned above. Unfortunately, they see these fixes as quick knobs to experiment with. Trying these one after another can lead to months of wasted effort, debugging fatigue, and deep frustration.

What’s the solution then? The harder part is knowing which trick matches which failure. A reranker, for example, helps when the right chunk is already retrieved but buried too low. It does not solve the same problem as better chunking or HyDE. Identifying the right failure mode quickly is far more important than the specific fix you try.

Failure Modes of Agentic Search?

One useful way to think about RAG is how to setup bridges between what the user asks and where the answer lives in your documents. A bad answer usually means one of those bridges failed: the system did not retrieve the right evidence, retrieved it but ranked it poorly, retrieved without paying heed to the surrounding context, or could not distinguish the correct chunk from a similar but wrong one.

These are different failure modes and each mode comes with its own bag of fixes. That is why I prefer grouping RAG fixes by failure mode instead of treating them as a checklist.

The first failure mode is Missing Context.

Sometimes the retriever finds the right paragraph, but the generated answer still feels incomplete or brittle. The paragraph may mention the answer, but not the section logic, assumptions, definitions, or surrounding details needed to use it correctly.

That is the problem structure-aware chunking tries to solve. Chunking by document structure instead of fixed token count keeps paragraphs closer to the section they belong to. Metadata such as section name, fragment title, date, or source gives the model additional placement context. And retrieving small chunks but passing a larger surrounding window to the LLM helps the system zoom out before generating.

The goal is not simply better chunking. The goal is to restore enough context for the answer to be correct and make sense.

Second failure mode: Weak query-document Bridge.

Sometimes the answer exists in the corpus, but the user’s query does not connect cleanly to the document text. This happens with short queries, vague queries, synonyms, domain-specific phrasing, and tables.

A couple of different fixes work here, depending on why the query failed to connect.

Hybrid search helps because keyword search and semantic search catch different kinds of relevance. HyDE helps when the original query is too thin, by creating a richer retrieval target. Multi-query retrieval helps by searching from several phrasings or intent guesses. For tables, summaries often work better than raw embeddings because they turn rows and numbers into language the query can match.

Again, the goal is not to try HyDE or use hybrid search. The goal is to make the user’s question easier to connect to the right evidence.

The third failure mode is Bad Ranking.

Sometimes the right chunk is retrieved, but it appears too low in the list. To a user, that might mean it is never seen. To an LLM, it means the useful evidence may be diluted by earlier noisy chunks.

That is where rerankers help. They do not necessarily expand what you retrieve; they improve the order of what you already found. The goal is to put the best evidence at the top.

Fourth failure mode: Weak Generalization to new queries.

A RAG pipeline can look good on easy examples and still fail in production. The real test is whether it can choose correctly when several chunks look similar, but only one is actually right.

That is why hard negatives or hard distractors matter. They test whether the retriever and the overall pipeline can distinguish close matches from correct matches. That enables generalization to unseen production queries.

So instead of asking, “Should we add HyDE?”, “Should we use hybrid search?”, or “Should we change chunk size?”, ask a more basic question first:

What kind of failure are we seeing?

• Is it missing context?

• A weak query-document bridge?

• Bad ranking?

• Low generalization?

• Agent’s plan is incomplete

• …

In practice, failure modes rarely show up cleanly. They are often entangled, and one failure can look like another.

A missing-context problem may look like a weak bridge problem.
A ranking problem may look like a generation problem.
A weak query-document match may look like “the model is hallucinating.”

So diagnosis takes effort and practice. You have to inspect execution logs, review retrieved chunks, look at failed queries, compare near-misses, and study the data closely.

I’ll be writing more on this topic. Subscribe to this series, and follow my YouTube Agentic Search series for deeper walkthroughs.

Related Posts:

OffNote Labs Newsletter

(LLM-IR #1) Why your GPT + Vector Search RAG demo won't make it to production?

One of the most popular use cases made feasible by ChatGPT / OpenAI APIs is the ability to chat with custom documents, company web pages, video transcripts and so on. Here is how the popular embed-and-match recipe works…

Read more

3 years ago · 8 likes · Nishant Sinha

Token-maxxing is in!

“Employees at Meta Platforms who want to show off their AI superuser chops are competing on an internal leaderboard for status as a “Session Immortal”— or, even better, “Token Legend.”

The resistance is structural, not emotional. Why is this reaction ununsual? Three core reasons:
1. AI changes the work itself. You move from being a hands-on developer to a code manager.
2. AI’s output is unpredictable. We are biased to expect consistency and determinism from software. Non-determinism creates distrust.
3. The stakes feel existential. "this will make you more productive" sounds like "this will make half of you unnecessary". why will I stay useful is not clarified.

Orgs succeeding at adoption treat it as a 𝐜𝐡𝐚𝐧𝐠𝐞 𝐦𝐚𝐧𝐚𝐠𝐞𝐦𝐞𝐧𝐭 𝐢𝐧𝐢𝐭𝐢𝐚𝐭𝐢𝐯𝐞:
- involving workers in design,
- providing role-specific training, and
- measuring outcomes instead of logins.

Really nice investigation on this issue here.

I did a quick short on this topic sometime back. Getting the hang of it!
Watch more videos by me on AI Engineering here — Do subscribe!

Add hybrid search, hierarchy, contextualization, graphs, rerankers, agent loops, or switch to something like PageIndex and say: this is vectorless, don’t need vector embeddings anymore! LLMs will handle long documents directly.

This is where the discussion gets muddled.

Most RAG debate compares implementations (how), not retrieval strategies (what/why). We keep enumerating methods instead of forming a domain understanding.

• There is naive RAG, advanced RAG, GraphRAG, HippoRAG, PageIndex, Hindsight, and so on.

• Next, we fall back on weak empirical takes or social media commentary: RAG is dead, GraphRAG is better, or you rarely need GraphRAG. All you need is Memory.

It is easy to add another X-Rag to the list. Happens on a daily basis.
Why? Lack of clarity on what any of these methods is trying to achieve.

For example, how does the default chunk-embed-match method compare to the newer pageindex method? The latter claims they are vectorless - is that the key differentiator?

Not knowing the what and why of a retrieval method pushes the engineer to keep switching methods and retuning knobs. There are hundreds of those knobs floating around. Different ways to chunk, embed, fine-tune, rerank, filter, graph ANN configurations, graph representations, ingestion configs, memory formats and so on.

I’ve seen teams waste months tweaking configurations before they realize what is actually wrong with their search engine.

One of the core retrieval problems is simpler than the jargon makes it sound: localization.

Localization Is the Core Retrieval Problem

Localization is the art of narrowing down from a huge corpus to a much smaller subset that is likely to contain the answer.

Say you have a million documents. In comes a query. The retrieval system cannot brute-force compare the query with the whole corpus in one shot. Many reasons why: queries span many attributes, keywords vs metadata, could be multi-hop and so on.

The system needs to localize: reduce that million to a few tens or hundreds of candidate documents, sections, or chunks that plausibly contain the answer to the query. Localization must be efficient. Once done, examine the localized candidates to extract the final answer (say, by reasoning using LLM).

Once you think this way, many retrieval methods become easier to compare.

Not “vector vs vectorless.”
Not “old RAG vs new RAG.”
Not “LLM-based vs classical.”

A vector-full approach localizes by

• hashing (embedding) each document into a small vector, hashing the query too,

• scoring query vs all documents at scale,

• then filtering the top-K results.

• In short: hash -> score -> filter. Each stage determines how efficient overall localization is.

A vectorless approach like PageIndex localizes differently.

• It builds a hierarchy of sections, tagged with summaries of what lies below. Intuitively, a table of contents, tagged with per item summary.

• The query is matched against that structure, iteratively top-down or one-shot

• and the engine narrows toward the sections most likely to answer it.

• once localized, reason through the full sub-documents, or redo localization.

The first approach compares the query against everything in one flat sweep. The second one uses structure to progressively narrow the search space. The retrieval policy is tree-guided rather than pool-wide.

Once you see this, a lot of noisy debate becomes easier to parse. Embeddings are an implementation choice. LLM reasoning is an implementation choice. Summaries are an implementation choice. Even “vectorless” is just a description of one mechanism. Those are all answers to how.

It was never about vectors or LLM magic. What we really care about is efficient localization.

In fact, the key question is:

Which localization strategy works best for your problem?

More broadly, it is not just about Search. Localization is core to keeping context compact/relevant inside Agents (harness engineering).

We’re perfectly content if classic grep is enough to zero in to the right code parts.

Avoid Arguments about How Before You Know the Objective

A lot of retrieval discussion goes in circles because the objective is never made explicit.

Engineers switch methods without a clear model of what changed. They compare branded systems instead of comparing retrieval policies.

That is what makes search feel more mysterious than it is.

If you can clearly state how your system localizes to relevant documents, you are already thinking at a more useful level than most RAG discourse.

Too much discussion lives in the how. The interesting part is usually the what.

Summary of Key Ideas

• The paper’s main claim is that Claude Code itself is architecturally simple at the center: a basic loop that calls the model, runs tools, and repeats. The real sophistication sits in the surrounding harness.

• That harness is what the paper focuses on: permissions, context management, extensibility, subagent orchestration, and durable session persistence.

• The authors frame Claude Code as a system optimized around five values: human control, safety/security, reliable execution, capability amplification, and context adaptability.

• Claude Code uses a layered safety model rather than trusting user approvals alone: deny-first rules, multiple permission modes, an ML-based auto-mode classifier, hook interception, and optional shell sandboxing.

• Context management is treated as a first-class bottleneck. The paper argues Claude Code uses a multi-step compaction pipeline plus lazy loading of instructions and tools so the context window stays usable over long sessions.

• Extensibility is split into four distinct mechanisms, each with a different role: MCP for tool integration, plugins for packaging, skills for lightweight instruction shaping, and hooks for lifecycle interception.

• Subagents are presented as isolated workers rather than just extra prompts in the same thread. They can run with separate configs, tools, and even separate worktrees.

• Persistence is append-oriented: transcripts are stored as JSONL, can be resumed or forked, and are designed to outlive the active context window.

My takes on what stands out most:

• The strongest surprise is how much of the product is “scaffolding,” not model logic. The paper keeps coming back to the idea that the core loop is simple and most of the engineering effort is in the control system around it.

• The permission system is more layered than most people would guess. The paper identifies seven safety layers, not just “ask user before running command.”

• User approval prompts are considered behaviorally unreliable because users approve about 93% of them. So the design response was not “ask better,” but “rely less on prompts and more on boundaries.”

• Context management is not one summary step, but a five-stage compaction pipeline plus lazy-loading of instructions, rules, and tool definitions.

• CLAUDE.md is apparently not just a static instruction file. The paper describes a hierarchy of memory files, path-scoped lazy loading, and even the detail that this content is injected as user-context rather than system-prompt content. A fully modular pipeline to create CLAUDE.md.

• A particularly interesting design choice: resumed or forked sessions do not restore previously granted session permissions. The system treats each session as a fresh trust boundary, even at the cost of friction.

• The persistence model is unusually transparent. Instead of opaque state, the paper says Claude Code uses append-only JSONL transcripts and sidechain transcript files for subagents.

• The paper points to one sharp edge: under adversarial/performance pressure, some safety guarantees can weaken together because multiple layers share latency and cost constraints. It cites a case where commands with more than 50 subcommands may collapse into a single generic approval path.

Anthropic never wrote a detailed architecture document - all we know, about the model and its harness, is from their piece-wise communications. So, this unofficial structured writeup fills in a crucial gap. Read the full paper here.

How do you turn a vibe coder into a senior engineer?

𝐎𝐩𝐭𝐢𝐦𝐢𝐳𝐞 𝐟𝐨𝐫 𝐫𝐞𝐥𝐢𝐚𝐛𝐢𝐥𝐢𝐭𝐲, 𝐧𝐨𝐭 𝐣𝐮𝐬𝐭 𝐯𝐞𝐥𝐨𝐜𝐢𝐭𝐲.

Most teams are still trying to solve this with more 𝘮𝘪𝘤𝘳𝘰-prompting.

Scaling a vibe-coded prototype without understanding its foundations is like building the second floor of a house without checking if the first floor has load-bearing walls.

That is the wrong approach. The right layer is to setup defaults: encode judgment once, get it on every task.

There are a couple of approaches floating around that do this. I’ll discuss two of them in this post.

Addy Osmani’s Agent Skills repository. Emulate a senior engineer’s workflow through a sequence of /slash commands. The (ex-) vibe coder follows this sequence deliberately during coding or creates a meta-command that loop over these atomic steps.

This is a pretty comprehensive set of commands with detailed prompts that cover all stages of development from spec’ing to shipping.

The repo forrestchang/andrej-karpathy-skills tackles the painpoint at a more abstract layer. AI moves fast at cost of silent, expensive mistakes: how to constrain its persona into a more deliberate agent.

Targets four key frustrating scenarios:

• wrong assumptions that ship to production

• overengineered code and bloated diffs

• edits that spill into unrelated files

• weak verification, so humans become full-time babysitters

The big idea is 𝐧𝐨𝐭 to write better prompts or enable easy manual guidance. Instead, set behavior-level constraints so the model acts like a careful engineer by default.The repo focuses on a persistent CLAUDE.md style/persona layer.

Instead of manually steering with /slash commands on each task, this sets coarser-level defaults: ask clarifying questions, avoid overengineering, keep edits scoped, and verify before saying done.

Vibe coding opened a huge new frontier for 10x output and faster iteration, but it also introduced a tax: constant babysitting and heavy review.

The practical middle path is not to slow AI down, but to channel it with clear engineering defaults and human checkpoints.

Let the developer stay in control of intent, boundaries, and final quality, while AI contributes speed, exploration, and bursts of implementation leverage.

That is how you keep the upside of velocity without paying for it later in reliability debt.

I wrote earlier about operational risks associated with AI written code. Ruthless judgement during /ship and desire to own what they ship are hallmarks of an AI-assisted Senior Engineer.

Some notes:

• There is a sharp drop in skill usage even when curated skills are available. This suggests that agents struggle to recognize relevant skills from their names and descriptions alone, and refining skill metadata may help agents better select which skills to load.

• What is Skill Refinement: the process of transforming retrieved skills into more useful forms

  • the agent reflects on which skills were useful and which were misleading

  • (more costly, better perf) agent can even merge and synthesize across multiple retrieved skills, extracting the relevant portions from each and combining them into a single coherent skill while discarding irrelevant content.

  • Refinement is assisted by the meta-skill skill-creator from Claude

Key lesson: A better packaging layer does not remove the decision problem.

The core problems are here to stay:
- How does your Agent pick the right one from hundreds of Skills or Tools?
- Even when the right Skills are retrieved, does the Agent deploy them correctly?

Like most of his other quick tweets, this too went viral.

In summary: how to use Agents to store, manipulate, retrieve from a unified (personal) knowledge Wiki. Given only a raw dump of documents of your interest, changing over time. An auto-updated, human-readable index, of all your knowledge. Do this with minimal human effort.

Here’s the high-level plan:

• Dump all your documents into a folder.

• Agent turns that into a Knowledge Wiki - a map for your knowledge, some call it a second brain. This is collection of markdown files, organized by concepts, relations between documents, cite original wherever required.

• Query the full Wiki using plain English, again through an Agent.

• The Wiki evolves as you add more documents.

• Easily navigate through Wiki using Obsidian. That serves as the Data IDE. View raw data and make manual edits if required.
  
  You can ask it (all sorts of) things like:
  - "Which other docs reference this concept?"
  - "What connects these two concepts?"
  - "What are the most important insights in this (sub-)project?"
  - and more..
  
  All curation is Agent-automated. No vector database. No config files. Fully end user enabled.

It turns out that several researchers/influencers already have created and maintain personal versions of this Wiki concept. For example, Lex Fridman, @omarsar0 and others.

Karpathy put the requirements (PRD) out, which were mostly implicit till this point. More details below.

If you’ve followed the whole retrieval ecosystem, RAG, vector databases, knowledge graphs, and the frameworks around them, it might seem surprising that a much simpler approach, just dumping files into a workspace and letting the agent grep through them, works at all.

However, this approach isn’t quite out of the blue. We could have seen this coming:

• PageIndex - vectorless RAG

• Openclaw - filesystem + .md as database. read more here.

• many other recent ..

All of above rely on

• store all data as .md files, stored with some hierarchy

• building or reusing external indices (directory path or content-summary based) for all the data transformed

• Openclaw and others update the .md files with task reflections, updates etc

Karpathy’s Proposal

Karpathy’s proposal generalizes this to the full docs lifecyle:

• create a (road) map to organize the documents. This map or index is itself a collection of .md files - the Wiki.

• update the map as new docs arrive

• generic-query these documents (no additional infra other than Agent and tools)

Building a personalized Wiki, which adapts to new knowledge is, in many ways, the holy grail for the end-user.

As much as this excites me, I can already see several blockers to making it reliable, scalable, and genuinely useful in practice.

• Can Agents keep data clean and organized as we keep throwing more documents at it?

• What topic clusters will it choose? Do they align with what human expects? What is the granularity of concepts? Too many concepts will disorient both human and search.

• Will the global overview lose fine-grained details in the articles? Even make it harder to retrieve back the details?

• How do we query at scale? One way to scale this is nested retrieval as in PageIndex - lookup top level index, then go down level by level to the original docs. At some point, however, you need to trust that the LLM can perform faithful reasoning over a large-enough context. That may not hold. More scalable options include using qmd for searching markdown files.

• What hurts most with Agent vibe coding is that humans lose the mental map of their code base. Are humans ok losing the map for their personal KB too?

In my opinion, we will have vibe-success building such Wikis over tens or even hundreds of documents. But then, we will run into data management issues (overlapping, incomplete, conflicting data) and consequently, poor query results, seeding disillusionment with personal knowledge bases again.

Potential workarounds:

• Keeping human in the loop for curating, re-organizing the data. This essential to enable persisted, long-term learning, but leads to increased human effort.

• File+Grep-based Agentic search has a ceiling. Implement iterative agentic search with advanced bridges.

• Pull back parts of the knowledge base into weights of the LLM by fine-tuning. I’ve strong reservations here, mainly because it reverses the nice modularity we have between the controller LLM and the data.

A few open source efforts (this with a lot of community discussions) are already on way to create these autonomous Personal Knowledge Wikis. Excited to see how the space evolves and how far we can go with minimal human-in-the-loop !

First, a quick look at the CLI agents and “super-agents”. They are converging on the same basic primitives (read in detail here):
- hierarchical object storage
- filesystem as memory
- terminal as the universal interface
- git for versioning and conflict handling

This matters because it simplifies the surface area of agent systems. Instead of building everything around a dedicated protocol layer with explicit service metadata, agents can increasingly work through the interfaces that software systems already expose: files, shells, processes, and repositories.

This stack is simpler than the earlier “everything through MCP” vision.
But the wrong conclusion is that MCP’s bottlenecks have disappeared.
They have not. They have moved!

𝐅𝐢𝐫𝐬𝐭, 𝐌𝐂𝐏’𝐬 𝐞𝐱𝐩𝐥𝐢𝐜𝐢𝐭 𝐛𝐥𝐨𝐚𝐭 𝐢𝐬 𝐛𝐞𝐜𝐨𝐦𝐢𝐧𝐠 𝐢𝐦𝐩𝐥𝐢𝐜𝐢𝐭 𝐂𝐋𝐈 𝐛𝐥𝐨𝐚𝐭.
Earlier, the clutter was visible in eager-loaded service metadata and tool registries.
Now the same sprawl can show up as a growing universe of CLI commands, flags, and external service wrappers.

The burden of managing that complexity has shifted to (or stays with) the agent.

𝐒𝐞𝐜𝐨𝐧𝐝, 𝐭𝐡𝐞 𝐟𝐢𝐥𝐞𝐬𝐲𝐬𝐭𝐞𝐦 𝐢𝐬 𝐛𝐞𝐢𝐧𝐠 𝐚𝐬𝐤𝐞𝐝 𝐭𝐨 𝐚𝐜𝐭 𝐥𝐢𝐤𝐞 𝐚 𝐝𝐚𝐭𝐚𝐛𝐚𝐬𝐞.
You can get surprisingly far with files, grep, hierarchy and conventions.

The filesystem is a powerful substrate, and in many cases it is enough. Files are easy to inspect, easy to version, and easy to integrate into existing developer workflows. But the filesystem is not a database. You can get far with grep and hierarchical naming conventions, but friction appears quickly when you need richer retrieval, structured relationships, scalable writes or consistent shared state.

So yes, the stack is getting simpler.
But simplification at the interface layer does not remove coordination complexity. 𝐈𝐭 𝐣𝐮𝐬𝐭 𝐫𝐞𝐥𝐨𝐜𝐚𝐭𝐞𝐬 𝐢𝐭.

The key lesson is not that MCP lost and CLI won. When one interface layer gets simpler, the burden usually reappears somewhere else: tool discovery, state management, retrieval, or conflict handling.

The important question is no longer “Do you use MCP or CLI?”
It is: 𝐰𝐡𝐞𝐫𝐞 𝐢𝐬 𝐲𝐨𝐮𝐫 𝐬𝐭𝐚𝐭𝐞, 𝐦𝐞𝐦𝐨𝐫𝐲, 𝐚𝐧𝐝 𝐭𝐨𝐨𝐥-𝐬𝐞𝐥𝐞𝐜𝐭𝐢𝐨𝐧 𝐜𝐨𝐦𝐩𝐥𝐞𝐱𝐢𝐭𝐲 𝐚𝐜𝐭𝐮𝐚𝐥𝐥𝐲 𝐥𝐢𝐯𝐢𝐧𝐠 𝐧𝐨𝐰?

Key takewaway: Understand where the coordination layer has moved in your system.

The issue is not that the code looks bad. It often looks surprisingly polished, sometimes more polished than what many engineers would write under time pressure.

The problem is that a clean PR is not the same as safe-to-ship code.

Models can generate convincing implementations without understanding the implicit constraints around your system:

• infrastructure limits

• runtime dependencies

• operational quirks

• failure patterns that only show up in production

That creates a dangerous gap between looks correct and is safe.

The right stance is to leverage AI, not rely on it.

AI can help teams move faster through grunt work.
But engineering judgment now shifts from writing every line to ruthlessly reviewing what gets shipped.

If you’re using AI coding tools in real systems, this is the psychological and technical battle that matters most. Hard learned lessons from using agents in production.

I think we are finally zeroing in on what the top engineering skills are for the upcoming cobuild-with-AI generation.

In contrast, RL approaches learn from rewards provided at the end of the run. Less supervision - better generalization. However, each trace has too little information/supervision. We need many runs / rollouts to separate relevant actions from noise.

Is there a middle ground - efficiency + generalization?

PivotRL is one in the several recent approaches to find a middle ground between RL and SFT.

• (filter) narrow down to key moments in the traces - pivots. Actions at the pivot points vary highly across runs.

• (reuse). transfer rewards across traces with equivalent intermediate actions.

Also, see article on on-policy distillation with dense supervision from Thinking Machines. Dense supervision tries to find a similar sweet spot.

Yet another middle ground is Near-Future Policy Optimization (NFPO), which uses a checkpoint from later in the same training run as your trajectory source. Why? External teacher trajectories (from a stronger model) are high quality - but far from your current policy. The model struggles to absorb them. Past replay trajectories are close to your current policy but you’ve already learned from them.

NPO finds a middle path:

Great overview of different related techniques (Mar 2026)

This is now standard practice. For a long time, developers tried to generate code directly from top-level specifications, and the results were often poor. It is very difficult to make one-shot generated code reliably match a high-level spec.

How did we solve this?

We introduced a formal planning phase: an artifact that translates intent into a lower-level structure that can guide implementation. That works much better. The lesson is that autonomous code generation can work, but only with careful scaffolding.

There is a useful analogy here to compiler design. A compiler translates code into lower-level instructions automatically, at scale, and with high reliability. Its advantage comes from using multiple phases. Each phase analyzes the whole program at one level of abstraction and then converts it into the next intermediate form. Trying to collapse all of that into a single pass would be algorithmically harder and much more difficult to debug. Multiple smaller phases keep the process tractable and clean.

Agents do not automate coding.
They automate (first-phase) draft generation.

Agent-driven code generation should work the same way. We need multiple intermediate specifications of increasing granularity that eventually lead to low-level code, while still keeping tight control over what gets generated.

Over time, more of our intent can be pushed into specifications and guardrails, giving us an Agent workflow with its own sequence of “compile” phases.

For example, how Kumo AI team escaped low level Pytorch/XGBoost library errors by working at higher layer.

Coding agents are only as good as the abstractions they work with.

We put this to the test using Kumo's Research Agent to build an account-level fraud model — and the results were eye-opening.

When coding agents work with low-level frameworks, mistakes are almost inevitable.

PyTorch errors: double class imbalance, loss averaged by data size instead of sample size
XGBoost errors: label granularity issues, label leakage

These aren't careless mistakes — they're a symptom of working with the wrong abstractions. Low-level frameworks weren't designed for the way agents reason about problems.

With Kumo's Research Agent, the same task produced zero errors — and required just 55 lines of code instead of 1,014. The resulting model was also 10% more accurate than an expert-built XGBoost model in production.
The lesson? The abstraction layer matters as much as the agent itself. Kumo gives coding agents the right foundation to reason about relational data — so they stop making the kinds of mistakes that quietly undermine your models.
We're seeing this play out across fraud detection, recommendations, risk scoring, and more. The teams winning with AI aren't just picking the best agent — they're pairing it with the right platform.

Yet another instance of this - build better GPU abstractions to allow Agents to optimize GPU programs better. here

Do you agree? Will be interesting to see how this hypothesis pans out in the next few years.

Model Access Layers (APIs / Infra Gateways)

👉 “Where developers get models”

• OpenRouter

• Ollama

• Vercel

  Thanks for reading Engineering Agents! Subscribe for free to receive new posts and support my work.

What they do

• Aggregate or host models (OpenAI, Anthropic, OSS, etc.)

• Provide unified APIs, routing, pricing abstraction

• Sit between model providers and apps

Mental model

Like AWS for models — infra layer, not end-user apps

Developer Tools / Coding Environments

👉 “Where developers use models while building”

• Trae

• opencode

• kilocode

What they do

• AI inside IDE / CLI / dev workflows

• Help generate, edit, debug code

• Tight feedback loop with developers

Mental model

Like VS Code + Copilot layer — high engagement, sticky usage

3) User-Facing Apps / Aggregators

👉 “Where end users discover and use models”

• Yupp

• Zo

What they do

• Provide chat interfaces / multi-model access

• Often compare models or unify UX

• Distribution = eyeballs + usage

Mental model

Like ChatGPT-style surfaces, but multi-model

The real gap is not prompt creativity; it is the engineering discipline to make prompted systems observable and safe to change.

Two ideas matter most.

First, make all behavior observable. A prompt is not just text; it is executable business logic. That means every run should be traced: prompt version, retrieved context, tool calls, outputs, latency, and cost. Once you can see runs clearly, silent failures stop being anecdotes and become measurable alerts you can act on.

Second, make changes safely. Prompt updates should be evaluated on a small but representative set of real production cases, with rubrics tied to actual product outcomes. The strongest setup combines LLM-as-a-judge for nuance with deterministic checks for things code can verify reliably. Over time, continuous evaluation becomes your canary system for prompts, catching regressions before users do.

The payoff is simple: you ship with confidence, improve systematically, and build a moat through measured learning rather than one-off prompt tricks.