Polling Analysis Agent

An agent that answers questions about US public-opinion polling — "In the 2021 General Social Survey, what share of US adults said they were very happy?" — and backs every number it reports with provenance: who produced the data, when it was fielded, and the sample size.

It runs a deterministic pipeline with one embedded agent loop:

[Query] → [Search] → [Extract] → [Analyze] → [Summarize] → [Answer]

Each stage does one job and is inspectable on its own, so a run is easy to follow and debug. The design favors that clarity over giving the model free rein: only the Analyze stage runs an open-ended agent loop, and it does so in a sandbox with no network access. There are two ways to use it:

• a single-shot CLI — one question per invocation;
• a multi-turn chat UI — a browser session is one conversation, so a follow-up like "what about by party?" is understood in context and answered against the same dataset without searching again.

Both are thin adapters over the same run_turn entrypoint. See AGENTS.md for the full architecture and conventions.

How the data is sourced

The agent never fetches from the open web. It is a client of the polling-data-mcp broker — a separate, read-only MCP server in its own repo (rschoenbeck/polling-data-mcp) that serves Harvard Dataverse and the General Social Survey (GSS). The broker owns the source whitelist, network egress, download caps, dataset search, and the standardization of each dataset into a Frictionless descriptor. The agent just asks it questions.

That split keeps the trust boundary clean: untrusted source data and the code that fetches it live in the broker, and a few properties hold all the way through the agent — raw survey rows are computed on inside the sandbox but never enter the model's prompt, and any text relayed from a source is treated as data, never as instructions.

Start the broker first, in its own checkout, before running the agent:

uv run polling-data-mcp --transport http --host 127.0.0.1 --port 8000

The agent reaches it at BROKER_URL (default http://127.0.0.1:8000/mcp). A plain GET /mcp returning HTTP 406 is normal for MCP streamable HTTP, not a fault.

Requirements

• Python 3.13 (pinned in .python-version)
• uv
• A dedicated, spend-capped ANTHROPIC_API_KEY (see .env.example)
• The polling-data-mcp broker reachable at BROKER_URL (see above)

Setup

uv sync                 # install deps from pyproject.toml + uv.lock
cp .env.example .env    # then fill in your ANTHROPIC_API_KEY (and BROKER_URL if not default)

Usage (Web UI) (Recommended)

A Gradio chat UI is a thin adapter over the same pipeline as the CLI:

uv run polling-agent-ui                 # serves the chat UI at http://127.0.0.1:7860
uv run polling-agent-ui --port 8080     # bind a different port (also --host)

The broker must be reachable and ANTHROPIC_API_KEY set (as with the CLI below).

A browser session is one conversation. A follow-up is understood in context (e.g. "what about by party?"), and when it stays on the same dataset it skips Search and re-reads the prior dataset from cache, so on-topic follow-ups are cheaper and faster. Clearing the chat starts fresh. Each answer renders as markdown with an expandable provenance/methodology section.

Usage (CLI)

uv run polling-agent "In the 2021 General Social Survey, what share of US adults said they were very happy?"
uv run polling-agent --refresh "..."   # force re-extraction, bypass cache
uv run polling-agent --verbose "..."   # print per-stage intermediate output

Note that the CLI only supports single-shot questions.

Model & code-execution tool

All model calls go through client.py, which selects the model id and applies a per-call max_tokens cap. The ids below were confirmed against Anthropic's docs on 2026-05-26 (recorded in config.py with the same source links):

• Model: claude-opus-4-7 (Claude Opus 4.7) — 1M-token context. Models overview
• Code-execution tool: code_execution_20260120 (tool name code_execution). Analyze runs it as a closed sandbox: the loop is offered only this tool (never a web-fetch/search tool) and the sandbox has no network access, so analysis computes on the extracted data and can never reach out for more. Code execution tool

Running in a container

Containerizing runs the agent with no host credentials, as a non-root user, with only ANTHROPIC_API_KEY injected at run time. The key is never baked into the image — .dockerignore keeps .env (and .git, .venv, data/) out of the build context.

The broker must be reachable from inside the container. On Docker Desktop the host is reachable as host.docker.internal, so point BROKER_URL at it. The broker's HTTP transport rejects requests whose Host header it doesn't recognize (DNS-rebinding protection), so start it bound to all interfaces and allow the container's gateway host:

uv run polling-data-mcp --transport http --host 0.0.0.0 --port 8000 \
  --allowed-host host.docker.internal:8000

docker build -t polling-agent .

# Credential-clean run: forward ONLY the key + the broker URL, nothing else.
export ANTHROPIC_API_KEY=sk-ant-...
docker run --rm -e ANTHROPIC_API_KEY -e BROKER_URL=http://host.docker.internal:8000/mcp \
  polling-agent "In the 2021 General Social Survey, what share of US adults said they were very happy?"

# Flags pass straight through to the CLI:
docker run --rm -e ANTHROPIC_API_KEY -e BROKER_URL=http://host.docker.internal:8000/mcp \
  polling-agent --verbose "..."

Don't mount your host home, .env, or cloud-CLI config into the container — that defeats the purpose. The SQLite store and extraction cache live on the container's ephemeral filesystem and vanish with --rm; mount a named volume at /app/data to persist them.

Egress. This process reaches only two destinations: api.anthropic.com (model calls) and BROKER_URL (the broker). A strict network-level allowlist via an egress proxy is the production hardening step and is out of scope here; at minimum, don't run the container on the host network.

Uploaded-data retention. Analyze uploads two files to Anthropic's hosted code-execution container via the Files API: the staged rows file and a codebook.json of the column titles and value legends. Hosted code execution is not Zero-Data-Retention-eligible, so uploaded data may be retained server-side per Anthropic's standard retention. The agent deletes both uploaded files after each run (success or failure), but a deployment handling sensitive data should account for this retention in its data-handling review.

Quality gates

uv run ruff format .          # format
uv run ruff check .           # lint
uv run pytest                 # unit tests (fast, no API calls)
uv run pytest -m "not eval"   # fast tests only
uv run pytest -m eval         # single-shot eval set (hits the API + a live broker)

Evaluation

The single-shot eval set (tests/eval/) runs the real pipeline once per question in tests/eval/single_shot.yaml and scores four axes per case: routing (with no prior turn, every question routes to the full pipeline), extraction (Search + Extract reach the expected dataset and row count), analysis (the model's computed value is within tolerance of the value grounded from the broker's own rows), and provenance (producer

• field dates + sample size present and matching the broker).

Every expected number is computed from real GSS rows by tests/eval/ground_truth.py, not hand-typed — re-run it against a live broker to refresh the set:

uv run python -m tests.eval.ground_truth   # rewrites single_shot.yaml (broker only, no API)
uv run pytest -m eval -s tests/eval        # run the eval; -s shows the per-axis scorecard

The eval needs a live broker and ANTHROPIC_API_KEY (it spends API budget). It is informational — no fixed pass/fail threshold; use the per-axis breakdown to find the weakest stage.

The set is 6 curated GSS cases + 1 no-match control, deliberately small because each case's Analyze step is a hosted code-execution loop (the run's dominant cost). The six span all three GSS cross-sections, both metrics (valid-percent + count), and all three variables — enough to exercise every axis once without repeating near-identical analyses. The no-match control (a question the broker has no dataset for) must short-circuit as a Search no_match rather than invent an answer.

Offline tests (-m "not eval") never hit the API or a live broker — they use the recorded broker fixtures in tests/fixtures/broker/.