hncb-fusion-deid-demo/CLAUDE.md

# CLAUDE.md

Guidance for Claude Code working in this repo. Read this first.

## What this is
A **demo** (not production) of a reversible PII de-identification round trip for
HNCB: advisor query with a real name → tokenized before it leaves for the cloud →
Amazon Bedrock AgentCore agent reasons on tokens only → identity restored on-prem
before the advisor sees the answer. Everything runs in **one AWS account**;
"on-prem" is a logical, tag-labelled zone standing in for the branch data centre.
The demo proves data-flow behaviour, not physical residency.

## Critical context
- **Fusion is a shared SaaS instance. DO NOT build, host, or Terraform it.** It is
  configured in the Amplify AI Gateway console (see `fusion/POLICY_SETUP.md`).
  Because it is SaaS, anything it calls must be a **public HTTPS endpoint with
  auth** — it cannot reach private VPC resources or use local AWS creds.
- This means the AWS side's job is to expose two endpoints Fusion will call:
  **`/tokenize`** (ingress) and **`/restore`** (egress), plus host the detector,
  vault, RAG tool, and agent. See the task backlog below.
- **Detection ≠ redaction.** The detector returns typed findings; our code owns
  minting tokens, writing the vault, and restoring. Never redact-and-discard.
- Tokens must be **random per request** and stored in the vault keyed by session.

## Architecture (maps to the customer's 8 steps)
Advisor UI → **Fusion SaaS** → [`/tokenize`: detect (Presidio) + mint + vault-write]
→ AgentCore Runtime (Bedrock model) → tool call via AgentCore Gateway → RAG Lambda
(resolve token → de-identified evidence) → agent talking points → **Fusion SaaS**
→ [`/restore`: vault lookup] → Advisor UI.

## Repo map
```
terraform/       DynamoDB (vault + customers), RAG Lambda, IAM, Presidio hosting
lambda_rag/      RAG tool (real): token resolve -> evidence package
gateway_api/     TODO: /tokenize and /restore HTTPS endpoints Fusion calls  (T1, T2)
agent/           Strands agent for AgentCore Runtime + tool schema
presidio/        detector service (typed findings) + Dockerfile
seed/            fake customer (Wang Xiaoming) + seed script
ui/              advisor UI (restored-vs-tokenized split view)
scripts/         deploy.sh, agentcore_setup.sh, teardown.sh
fusion/          POLICY_SETUP.md (SaaS console config — human does this, not you)
```

## Commands
- Deploy core infra: `bash scripts/deploy.sh` (review first)
- Seed data: `VAULT_TABLE=... CUSTOMERS_TABLE=... python seed/seed.py --with-demo-token`
- AgentCore: `bash scripts/agentcore_setup.sh` (verify against `agentcore --help`)
- Teardown (do this to stop spend): `bash scripts/teardown.sh`
- Terraform: run all `terraform` commands with `-chdir=terraform`.

## Conventions & guardrails
- Region default `us-east-1` (best Bedrock/AgentCore quotas). Keep it a variable.
- **Never commit AWS creds, tokens, or real PII.** Seed data is synthetic only.
- Python: standard library + boto3; keep handlers small and dependency-light.
- Pin `bedrock-agentcore` / `strands-agents` versions before `agentcore launch` —
  these SDKs move fast; verify signatures against installed versions, don't assume.
- Prefer Lambda Function URLs (with auth) or API Gateway for the public endpoints.
- After any infra change, remind the human to `teardown` when done recording.

## Task backlog (pick these up)
Each task: keep it demo-grade, add a note in the file, and update this list.

- **T1 — `/tokenize` endpoint** ✅ **done + live-verified** (`gateway_api/tokenize/handler.py`,
  terraform `terraform/gateway.tf`). Lambda behind a **public API Gateway HTTP API**
  (not a Function URL — this account's SCP blocks unauthenticated Function URLs),
  shared-secret auth (`tokenize_api_key`, sent as `x-api-key` / bearer, checked
  in-handler). Calls Presidio `/analyze`, resolves overlapping findings (specific
  entity wins, so the ROC ID stays `TW_ROC_ID` not `PERSON`), mints random
  `CUST_<rand>`/`TW_<rand>` tokens, writes vault rows
  `{token, type, value, session_id, expires_at [, customer_id]}` (`value` = original
  PII for /restore; `customer_id` best-effort resolved for the RAG tool), splices
  right-to-left, returns `{deidentified_prompt, session_id}`. **Verified live** in
  `ap-southeast-1`: `王小明`→`CUST_*`, `A123456789`→`TW_*`, clean prompt, vault rows
  written with resolved `customer_id`, repeat→different tokens, no/bad key→401.
  Infra changes made while deploying: Presidio task bumped to **2GB** (1GB OOM'd on
  the 603MB zh model → connection refused), **awslogs** added, **ARM64/Graviton**
  runtime (built on Apple Silicon via podman). All AWS objects tagged
  `Owner="conan hncb demo"`. *Caveats:* Presidio has no stable endpoint — its
  Fargate public IP changes per task launch, so `presidio_url` in
  `terraform/local.auto.tfvars` must be refreshed and the tokenize Lambda re-applied
  (an ALB/Cloud Map would fix this; out of scope for the demo). The default VPC was
  created by hand and tagged `maintenance=manual-cleanup-required` (Terraform doesn't
  own it, so `teardown.sh` won't remove it).
- **T2 — `/restore` endpoint** ✅ **done + live-verified** (`gateway_api/restore/handler.py`,
  terraform in `terraform/gateway.tf`). `POST /restore` on the same API Gateway,
  same shared-secret auth; IAM is read-only `dynamodb:Scan` on the vault. Input
  `{session_id, text}` → scans the vault for THIS session's tokens (scoped, so one
  session can't restore another's), swaps inline tokens back to the original PII
  (longest-token-first), and prepends the customer's real name at the envelope
  level (`（客戶：王小明）`). Returns `{final, restored}`. **Verified live** with the
  full T1→T2 round trip: tokenized prompt out → simulated token-only cloud answer →
  `王小明` + `A123456789` restored, no tokens left; no/bad key → 401; unknown
  session → text unchanged, `restored:0` (no cross-session leak). `restore_url`
  is a Terraform output.
- **T3 — reachability from SaaS** ✅ **done + live-verified** (`terraform/ecs.tf`,
  `terraform/gateway.tf`). Presidio is now private: its SG allows 5001 only from the
  tokenize Lambda's SG (no `0.0.0.0/0`). The tokenize Lambda runs **in the VPC**
  (`vpc_config` + `AWSLambdaVPCAccessExecutionRole`) and reaches Presidio on its
  **private** IP; DynamoDB is reached via a **gateway VPC endpoint** (no NAT). The
  Fargate task keeps a public IP only to pull from ECR, but all inbound is SG-locked.
  Only the API Gateway (`/tokenize` + `/restore`, shared-secret auth) is public.
  **Verified live:** Presidio's public IP `:5001` now times out from the internet;
  the full round trip still works through the private path. *Caveat update:* the
  tokenize Lambda now targets Presidio's **private** IP (`172.31.2.216` currently),
  which still changes per task launch — refresh `presidio_url` in
  `terraform/local.auto.tfvars` and re-apply `aws_lambda_function.tokenize` if the
  task restarts (Cloud Map/ALB would give a stable name; out of scope for the demo).
- **T4 — AgentCore Gateway target** ✅ **done + live-verified** (`terraform/agentcore.tf`,
  `scripts/agentcore_setup.sh`, `scripts/gateway_invoke_test.py`). Created an MCP
  Gateway (`hncb-rag-gateway`, `authorizerType=AWS_IAM` so inbound is SigV4 — no
  Cognito) with the RAG Lambda registered as a `lambda` MCP target; the Gateway
  assumes a Terraform-managed IAM role that can only `lambda:InvokeFunction` the RAG
  tool. Built the Gateway/target with `aws bedrock-agentcore-control` (Terraform
  lags here). **Verified live:** `gateway_invoke_test.py` (SigV4 MCP
  initialize→tools/list→tools/call) returned the full de-identified evidence package
  for `CUST_000123` through the Gateway; the tool shows up namespaced as
  `rag___get_customer_activity_summary`. Gotchas handled: the Gateway `inputSchema`
  is a restricted JSON-Schema subset (strip `default`/unknown keys); MCP responses
  can be SSE; tool names are target-prefixed. Gateway tagged `Owner="conan hncb demo"`;
  `teardown.sh` deletes it (not in Terraform state). MCP URL is printed by the setup
  script. *Note:* `.venv-t4/` (gitignored) holds boto3 for the invoke test.
- **T5 — Agent runtime** ✅ **done + live-verified** (`agent/agent.py`,
  `agent/requirements.txt`, `scripts/agentcore_setup.sh` T5 section). Pinned deps
  (bedrock-agentcore 1.16.0 / strands-agents 1.45.0 / mcp 1.28.1 / boto3 1.43.38),
  `agentcore configure --non-interactive` + `agentcore launch` (cloud CodeBuild,
  auto-created execution role + ECR + STM memory). **Runtime ARN:**
  `arn:aws:bedrock-agentcore:ap-southeast-1:286171702468:runtime/hncb_advisor_agent-duWZOT5Far`.
  **Verified live:** `agentcore invoke {"prompt": "...CUST_000123..."}` returns
  numbered talking points built only from the de-identified evidence — no name, no
  leaked token. Three gotchas fixed (all captured in code/scripts):
  1. In `ap-southeast-1` the model is **INFERENCE_PROFILE-only** → use
     `apac.anthropic.claude-3-5-sonnet-20241022-v2:0`, not the raw id.
  2. Our Gateway is **AWS_IAM**, so `agent.py` SigV4-signs MCP requests (httpx.Auth)
     instead of using a bearer token.
  3. The auto-created runtime **execution role can't call the Gateway** by default →
     must attach `bedrock-agentcore:InvokeGateway` on the gateway ARN (the launch
     step in `agentcore_setup.sh` shows the exact command). Runtime + memory added to
     `teardown.sh`; agent logic was de-risked locally before launch.
- **T6 — UI wiring** ✅ **done + live-verified** (`ui/index.html`, `terraform/ui.tf`,
  `gateway_api/orchestrator/`). Hosted on **S3 + CloudFront** (private bucket, OAC).
  For the Fusion-less dry run I added a thin **demo orchestrator** (`POST /demo`, a
  Lambda on the same API) that runs the full chain server-side —
  `/tokenize` → SigV4 `InvokeAgentRuntime` → `/restore` — and returns
  `{final, deidentified_prompt, agent_tokenized}` (this stands in for Fusion ONLY
  for the dry run; we still never build/host Fusion). `index.html` reads the live
  endpoint from a generated `config.js` (so no ephemeral URL is committed) and
  renders the split view. CORS enabled on the API. **UI:**
  `https://d3n89cj9w7ynf0.cloudfront.net`. **Verified live:** CloudFront serves
  `index.html` + `config.js` (200); `/demo` returns `王小明`→`CUST_*` on the cloud
  side and `王小明`-restored talking points on the advisor side. Only the browser
  render itself is left as a human eyeball check. All resources tagged.
- **T7 — E2E rehearsal** ✅ **done + live-verified** (README demo/deploy sections
  rewritten with the real commands). Ran the full round trip via `/demo`
  (`王小明`→`CUST_317499` out, agent talking points on tokens only, `王小明` restored).
  **Money shot proven:** in the cloud runtime CloudWatch trace for that session the
  token appears in **13 events** while `王小明` and `A123456789` appear in **0** —
  the Bedrock/AgentCore trace is token-only. **Teardown verified (non-destructively):**
  `terraform destroy` targets 43 resources (Fargate, CloudFront, S3, DynamoDB, all
  Lambdas, API GW, VPC endpoint, SGs, IAM) and `teardown.sh` additionally deletes the
  Gateway/targets, Runtime, Memory, and both ECR repos — i.e. every billed resource.
  Residual is free-only: the hand-made default VPC, the AgentCore SDK exec role, an
  idle CodeBuild project. The actual `teardown.sh` run is left for the human to fire
  after recording (didn't auto-destroy the live demo).

## Known caveats (don't "fix" these silently)
- zh-TW detection is demo-narrow (tuned to the scripted entities), not production recall.
- The agentic token-resolution loop (T4) is custom orchestration by design.
- If you touch `fusion/`, it's documentation for a human — do not try to automate SaaS config.