chatbot evolution

My portfolio has a chat box on the homepage. Ask it what I've built and it answers from my projects, résumé, and repo docs.

I recently replaced the backend behind it. v1 was a hand-orchestrated pipeline of LLM calls. v2 is an eve agent with tools and its own runtime.

Looking back on that rigid llm call pipeline feels pretty silly. I realized at about the time everyone else did that flexible agentic loops were the correct way to build apps around these things.

---

The job

The chat has one job: answer questions about me using real data instead of guesses. That means it needs to reach into:

• Projects — repo metadata and doc reads from GitHub (through a short cache), plus a semantic search index over project docs that's kept in sync separately
• Résumé — experience, education, awards, skills
• Profile / persona — who I am, how I want to sound

The whole question is how you get an LLM to reach into that stuff at the right moment without hallucinating a job I never had.

v1 and v2 answer that question very differently.

---

v1: a pipeline I had to choreograph

v1 was a non-agentic, multi-LLM-call pipeline. Every chat turn went through three fixed stages:

                    ┌─────────────────────────────────────────────┐
   user message ───▶│  STAGE 1 · PLANNER   (LLM call #1)           │
                    │  claude-haiku-4-5                            │
                    │  "What should I search for? Do I even need   │
                    │   to retrieve anything?"                     │
                    │  → { queries:[...], topic, useProfile }      │
                    └───────────────────────┬─────────────────────┘
                                            │ (only if queries.length > 0)
                    ┌───────────────────────▼─────────────────────┐
                    │  STAGE 2 · RETRIEVAL  (no LLM, pure search)  │
                    │  hybrid BM25 + embeddings over projects,     │
                    │  résumé; fetch-all for profile               │
                    │  → top-K docs with scores                    │
                    └───────────────────────┬─────────────────────┘
                    ┌───────────────────────▼─────────────────────┐
                    │  STAGE 3 · ANSWER    (LLM call #2)           │
                    │  claude-sonnet-4-6                           │
                    │  retrieved docs + persona → final answer     │
                    │  → { message, uiHints, cardReasoning }       │
                    └───────────────────────┬─────────────────────┘
                                            ▼
                              SSE: stage / reasoning / token / ui / done

Two LLM calls per turn. A planner to decide what to look up, a deterministic search step in the middle, and an answer model to write the reply. The orchestration lived across a few workspace packages — chat-orchestrator, chat-llm, chat-next-api, chat-preprocess-cli — and looked roughly like this:

emitStageEvent('planner', 'start');
const plan = await runStreamingJsonResponse<PlannerLLMOutput>({
  client,
  model: plannerModel,            // claude-haiku-4-5
  systemPrompt: buildPlannerSystemPrompt(profileContext),
  userContent: buildPlannerUserContent(conversationSnippet, userText),
  schema: PlannerLLMOutputSchema,
  onTextDelta: (delta) => emitReasoning({ stage: 'planner', delta }),
});
emitStageEvent('planner', 'complete');

if (plan.queries.length > 0) {
  emitStageEvent('retrieval', 'start');
  retrieved = await executeRetrievalPlan(retrieval, plan, { minRelevanceScore });
  emitStageEvent('retrieval', 'complete');
}

emitStageEvent('answer', 'start');
const answer = await runStreamingJsonResponse<AnswerPayload>({
  client,
  model: answerModel,             // claude-sonnet-4-6
  systemPrompt: buildAnswerSystemPrompt(persona, profileContext),
  userContent: buildAnswerUserContent({ conversationSnippet, retrieved }),
  schema: AnswerPayloadSchema,
});

It worked, but it was more code than I wanted to own. The pain points:

1. The embeddings lived in my git repo. Retrieval needed vectors, so there was a separate CLI (chat-preprocess-cli) that crunched my projects and résumé into embeddings and committed them as JSON. How big? The commit that finally killed this pipeline deleted generated/projects-embeddings.json (26,205 lines) and generated/resume-embeddings.json (29,287 lines). Fifty-five thousand lines of vectors, versioned in git, that I had to remember to regenerate every time I edited a README. If I forgot, the chat answered with old facts.

2. Everything was sequential. Planner blocks retrieval blocks answer. Total latency was the sum of all three, every time, even for "hey what's your github." There was no way for the model to say "actually I don't need to look anything up" without first paying for a planner round-trip.

3. The rigid pipeline couldn't think mid-flight. The shape was fixed: look up once, then answer. If the answer model realized halfway through that it needed a different project's docs — too bad. There was no second lap.

4. Three provider harnesses. chat-llm had to paper over OpenAI's Responses API, Anthropic's Messages API (with tool_use bolted on just to coax out structured JSON), and a claude-code-CLI subprocess. Every provider streamed differently. Every provider broke differently.

It was well-tested, but I was still encoding a workflow the model could handle if I gave it tools.

---

What changed

The commit message that started the unwind:

feat(chat-vault): replace preprocess pipeline with vault file_search Delete the old /api/chat route, the chat preprocess scripts and evals, the chat-llm/chat-orchestrator/chat-next-api/chat-preprocess-cli workspace packages, generated embeddings, and obsolete debug…

The diff deleted thousands of lines.

The change was simple: stop making the app decide the whole retrieval plan up front. Give the model tools and let it decide when to use them.

Enter eve.

eve is a framework for building durable, tool-using AI agents — and what I love about it is that it hands you the one part everybody rebuilds from scratch the first time they write an agent: the loop. The model-calls-a-tool → tool-returns → model-decides-what's-next cycle, plus streaming, tool dispatch, and the durable state that lets a single turn pause and resume. I'd already hand-rolled all of that in v1's pipeline. eve meant I didn't have to write it again — only better — just to get an agent that could think mid-flight. What I actually wrote was small: an agent definition and a handful of tool stubs.

It also doesn't assume a runtime. eve models everything a run needs — the event log, the queue that wakes a paused turn, overflow storage — behind a "world" interface, and you bind that world to whatever infrastructure you have. For production I bind it to AWS: the agent ships as a Docker Lambda with a streaming Function URL, and its world is backed by DynamoDB, SQS, and S3 in a CDK construct. That's the whole deployment story — implement the world once, and eve's loop runs durably on serverless. The wiring is below.

---

v2: move the loop into an agent

In v2, /api/chat is no longer the orchestrator. It's a thin bridge. The agent loop runs in a separate service — an eve agent on its own Lambda — and the portfolio becomes the data layer the agent is allowed to call.

Here's the whole loop:

  browser
    │  POST /api/chat   (message + history)
    ▼
  /api/chat  ── thin bridge (regional streaming Lambda) ───────────────┐
    │  mint channel JWT (HS256, 5-min TTL)                              │
    ▼                                                                   │
  eve agent Lambda  (Docker + Function URL, RESPONSE_STREAM)           │
    │                                                                   │
    ├──▶ Anthropic  claude-sonnet-4-6   ◀── the agent loop lives here  │
    │         │                                                         │
    │         │  "I need data." → tool call                            │
    │         ▼                                                         │
    │   POST /api/internal/agent/tool   (mint service JWT, HS256)  ─────┘
    │         │
    │         ├─ list_projects / project_catalog_stats → admin records + cached GitHub meta
    │         ├─ list_project_docs / read_project_doc   → GitHub API (1h cache)
    │         ├─ search_project_docs                    → OpenAI vector store (synced)
    │         ├─ file_search / grep / read_file         → the vault (profile/résumé)
    │         └─ set_ui_hints                           → cards + links for the UI
    │
    └──▶ streams reasoning + tokens + UI hints back ──▶ SSE ──▶ browser

The Lambda part is where most of the infrastructure work went: long turns need durable in-flight state on top of stateless compute.

The important pieces:

The bridge is boring on purpose

/api/chat validates input, checks the kill switch / rate limit / monthly budget cap, mints a short-lived JWT, and forwards the turn to the agent. Then it just translates the agent's event stream into the SSE shape the frontend already speaks:

const client = new Client({
  host: process.env.EVE_AGENT_ORIGIN,
  headers: async () => {
    const token = `Bearer ${await mintAgentChannelJwt()}`;   // HS256, exp 5 min
    return { authorization: token, 'x-eve-authorization': token };
  },
});

const session = client.session();
const response = await session.send({ message: buildTurnMessage(input), signal });

for await (const ev of response) {
  switch (ev.type) {
    case 'reasoning.appended': cb.onThinkingText(ev.data.reasoningDelta); break;
    case 'message.appended':   cb.onToken(ev.data.messageDelta);         break;
    case 'actions.requested':  /* a tool is being called → render a step */ break;
    case 'action.result':      /* tool came back → maybe it's UI hints */  break;
  }
}

No planner. No retrieval stage. No answer stage. The bridge does not decide what happens next; it relays events.

The agent definition is small

This is the definition that replaced the three-stage pipeline:

import { defineAgent } from "eve";
import { anthropic } from "@ai-sdk/anthropic";

export default defineAgent({
  model: anthropic("claude-sonnet-4-6"),
  reasoning: "low",
  ...(process.env.EVE_WORKFLOW_WORLD
    ? { experimental: { workflow: { world: "@portfolio/eve-world-aws" } } }
    : {}),
});

One model runs the turn and decides when to reach for a tool. "Do I need to look something up?" is no longer a separate Haiku call. If the agent needs three repos' docs, it reads three. If it needs nothing, it answers. The control flow is no longer hard-coded in my app.

Tools call back into the portfolio

The useful bit: the agent does not get a copy of my data. It gets thin tools that call back to the portfolio. Each tool on the agent side is a stub that POSTs to the portfolio's internal API:

export default defineTool({
  description: "Semantic search over the knowledge vault…",
  inputSchema: z.object({ query: z.string(), maxResults: z.number().optional() }),
  execute: (input) => callTool("file_search", input),   // → POST /api/internal/agent/tool
});

Freshness is split a few ways. Repo reads (read_project_doc, list_project_docs) come from the GitHub API through the app's one-hour cache. The structured catalog (list_projects) joins my admin records with cached GitHub repo metadata. Semantic project-doc search (search_project_docs) is still an index — an OpenAI vector store — refreshed by a portfolio sync job when I save my config (or hit /api/admin/projects/sync). Nothing here is real-time. The point is that GitHub is the source of truth, not vectors committed to the repo. The win over v1 is no embeddings checked into git, and no manual regenerate step.

Service auth

Because the agent and the portfolio are now separate services talking over HTTP, I needed real auth between them. It's two symmetric HS256 JWTs, each scoped to one direction:

Short-lived, minted per request, verified on the far side. Neither secret is baked into an image or a Lambda env var; both are pulled from Secrets Manager at cold start by bootstrap.mjs before the server boots.

Durable runs on Lambda

An agent loop needs durable step state: pause for a tool, resume, survive a redeploy, and not lose the current run because Lambda froze. Lambda has no disk and no memory between invocations. So eve's "world" gets backed by AWS primitives in a CDK construct (EveWorldInfra):

• DynamoDB (single-table, pk/sk + GSIs) — the durable event log for a run: the agent's steps and tool continuations while a turn is in flight.
• SQS FIFO — when a run pauses (say, waiting on a tool), a continuation message gets queued; a small consumer Lambda pulls it and re-POSTs to the agent's flow endpoint to wake it back up. Ordered, exactly-once-ish, per-run.
• S3 — overflow storage for tool payloads too chunky for a DynamoDB item, auto-expiring after 30 days.
• Docker Lambda + Function URL with RESPONSE_STREAM so tokens actually stream out instead of buffering until the end.

const fn = new lambda.DockerImageFunction(this, 'EveAgentFunction', {
  code: lambda.DockerImageCode.fromImageAsset(repoRoot, { file: 'apps/eve-agent/Dockerfile.lambda' }),
  timeout: Duration.minutes(5),
  memorySize: 2048,
});
const url = fn.addFunctionUrl({
  authType: lambda.FunctionUrlAuthType.NONE,        // auth is the channel JWT, not IAM
  invokeMode: lambda.InvokeMode.RESPONSE_STREAM,    // real streaming
});

It is a fair amount of plumbing, but it buys the thing I needed: a run can pause for a tool and resume without losing its place, while the agent still scales to zero when idle. Cross-turn chat continuity is simpler: each request includes the prior turns as context. The eve world handles durability within a turn.

---

v1 vs v2

                    v1                              v2
              ───────────────                 ───────────────
  control     I hard-code the stages      the model decides the stages
  LLM calls   2 fixed per turn            as many as the agent needs (often 1)
  retrieval   my hybrid BM25+vectors      tools the agent calls on demand
  data        embeddings in git        GitHub-backed reads/catalog (cached);
                                          synced semantic index
  runtime     one big edge function       thin bridge + separate agent Lambda
  state       none                        durable in-flight runs (Dynamo/SQS/S3);
                                          chat history replayed each turn
  my job      orchestrator & babysitter   data layer & door policy

Did I need to go agentic? Not strictly. v1 answered questions fine. But it made the app own planning, retrieval, provider quirks, and generated embedding artifacts. v2 moves the decision-making into the agent and leaves the portfolio as the data/tool layer. It is more infrastructure, but less bespoke orchestration. That's the trade I wanted.