▋
docs/features/chat/performance.md

Chat Performance Decisions

The chat path prioritizes time to useful output, durable conversation context, and authoritative accounting. The model loop remains the dominant cost, so the design removes avoidable model rounds and parallelizes independent network work.

Turn lifecycle

sequenceDiagram
  participant UI as Browser UI
  participant API as /api/chat
  participant DB as DynamoDB
  participant Eve as Eve agent
  participant LLM as Anthropic

  UI->>API: message + conversationId
  par Independent preflight
    API->>DB: settings (5 s cache)
    API->>API: minute/hour/day rate limits
    API->>API: initialize cost client
  end
  API->>DB: monthly budget state
  API->>DB: load Eve SessionState
  API->>Eve: new message + continuation
  Eve->>LLM: model step
  opt Grounding is needed
    LLM->>Eve: tool calls
    Eve->>LLM: grounded results
  end
  Eve-->>API: streamed reasoning, tools, and answer
  API-->>UI: frame-sized SSE deltas
  API->>DB: save Eve SessionState
  API-->>UI: done
  API->>DB: atomic cost increment

Decisions

Persist Eve cursors server-side

The browser keeps only a high-entropy conversationId. The server derives a domain-separated HMAC key and stores Eve's serializable SessionState in the dedicated ChatSessions DynamoDB table for 30 days. The table stores no chat transcript. Session storage in the browser naturally starts a new conversation when the tab session ends.

Alternatives considered:

  • Resending the complete transcript on every turn grows prompt tokens and loses Eve's native continuation semantics.
  • Returning the raw continuation token to the browser exposes runtime state and couples the public request contract to Eve internals.
  • Storing full transcripts duplicates data already owned by Eve's durable world.

Session load/save failures fail open. The bridge starts a new Eve session with the visible transcript, keeping chat available while making the degraded path explicit in logs.

Infer grounded UI cards

Successful project-document reads and scoped project searches emit project cards. Vault reads emit cards from their frontmatter, and direct URLs in the final answer emit link hints. set_ui_hints remains an optional tool for a profile link by stable id or a non-inferable card. This removes a mandatory model round while preserving the generic UI contract.

Keep authoritative work off the perceived critical path

The done SSE event is sent after Eve completes and its continuation cursor is saved. Cost accounting then performs one atomic DynamoDB update before the server closes the stream. The returned values provide both current and previous threshold states, so no preliminary read is necessary. CloudWatch cost metrics use Embedded Metric Format logs and require no separate API request.

Batch at the natural boundaries

  • The three Upstash windows execute concurrently.
  • Semantic hits are capped at 1,200 characters each and exact project reads at 24,000 characters, bounding context growth for a single tool result.
  • Visible GitHub repository metadata uses bounded concurrency.
  • Eve agent startup fetches environment and repository secret documents in parallel with one Secrets Manager client.
  • The browser commits accumulated token and thinking deltas at most once per 16 ms frame.
  • The AWS Workflow streamer buffers for 25 ms and uses DynamoDB BatchWrite in groups of 25 with unprocessed-item retries.
  • Live stream polling reads chunks and the terminal meta row in one Query when the page is unbounded.

Observability

Each completed turn logs one chat.turn.completed JSON event containing:

  • preflight duration;
  • time to first agent event and first answer token;
  • total user-visible duration;
  • Eve session id and whether the turn resumed;
  • iteration/tool counts, token usage, and computed cost.

Cost logs use the PortfolioChat/Costs CloudWatch namespace with stable App/Env dimensions so the month-to-date alarm remains attached. Session and continuation records expire automatically and contain no message text.