From 0d4d4dcc67d58515d577581cb6e8ebb1b4e4e67b Mon Sep 17 00:00:00 2001 From: Conan Scott Date: Tue, 7 Jul 2026 20:32:29 +1000 Subject: [PATCH] agents.md update for codex --- AGENTS.md | 168 ++++++++++++++++++++++++++++++++++++++++++++++++++++++ 1 file changed, 168 insertions(+) create mode 100644 AGENTS.md diff --git a/AGENTS.md b/AGENTS.md new file mode 100644 index 0000000..7535b4d --- /dev/null +++ b/AGENTS.md @@ -0,0 +1,168 @@ +# AGENTS.md + +Guidance for Codex working in this repo. Read this first. + +## What this is +A **demo** (not production) of a reversible PII de-identification round trip for +HNCB: advisor query with a real name → tokenized before it leaves for the cloud → +Amazon Bedrock AgentCore agent reasons on tokens only → identity restored on-prem +before the advisor sees the answer. Everything runs in **one AWS account**; +"on-prem" is a logical, tag-labelled zone standing in for the branch data centre. +The demo proves data-flow behaviour, not physical residency. + +## Critical context +- **Fusion is a shared SaaS instance. DO NOT build, host, or Terraform it.** It is + configured in the Amplify AI Gateway console (see `fusion/POLICY_SETUP.md`). + Because it is SaaS, anything it calls must be a **public HTTPS endpoint with + auth** — it cannot reach private VPC resources or use local AWS creds. +- This means the AWS side's job is to expose two endpoints Fusion will call: + **`/tokenize`** (ingress) and **`/restore`** (egress), plus host the detector, + vault, RAG tool, and agent. See the task backlog below. +- **Detection ≠ redaction.** The detector returns typed findings; our code owns + minting tokens, writing the vault, and restoring. Never redact-and-discard. +- Tokens must be **random per request** and stored in the vault keyed by session. + +## Architecture (maps to the customer's 8 steps) +Advisor UI → **Fusion SaaS** → [`/tokenize`: detect (Presidio) + mint + vault-write] +→ AgentCore Runtime (Bedrock model) → tool call via AgentCore Gateway → RAG Lambda +(resolve token → de-identified evidence) → agent talking points → **Fusion SaaS** +→ [`/restore`: vault lookup] → Advisor UI. + +## Repo map +``` +terraform/ DynamoDB (vault + customers), RAG Lambda, IAM, Presidio hosting +lambda_rag/ RAG tool (real): token resolve -> evidence package +gateway_api/ TODO: /tokenize and /restore HTTPS endpoints Fusion calls (T1, T2) +agent/ Strands agent for AgentCore Runtime + tool schema +presidio/ detector service (typed findings) + Dockerfile +seed/ fake customer (Wang Xiaoming) + seed script +ui/ advisor UI (restored-vs-tokenized split view) +scripts/ deploy.sh, agentcore_setup.sh, teardown.sh +fusion/ POLICY_SETUP.md (SaaS console config — human does this, not you) +``` + +## Commands +- Deploy core infra: `bash scripts/deploy.sh` (review first) +- Seed data: `VAULT_TABLE=... CUSTOMERS_TABLE=... python seed/seed.py --with-demo-token` +- AgentCore: `bash scripts/agentcore_setup.sh` (verify against `agentcore --help`) +- Teardown (do this to stop spend): `bash scripts/teardown.sh` +- Terraform: run all `terraform` commands with `-chdir=terraform`. + +## Conventions & guardrails +- Region default `us-east-1` (best Bedrock/AgentCore quotas). Keep it a variable. +- **Never commit AWS creds, tokens, or real PII.** Seed data is synthetic only. +- Python: standard library + boto3; keep handlers small and dependency-light. +- Pin `bedrock-agentcore` / `strands-agents` versions before `agentcore launch` — + these SDKs move fast; verify signatures against installed versions, don't assume. +- Prefer Lambda Function URLs (with auth) or API Gateway for the public endpoints. +- After any infra change, remind the human to `teardown` when done recording. + +## Task backlog (pick these up) +Each task: keep it demo-grade, add a note in the file, and update this list. + +- **T1 — `/tokenize` endpoint** ✅ **done + live-verified** (`gateway_api/tokenize/handler.py`, + terraform `terraform/gateway.tf`). Lambda behind a **public API Gateway HTTP API** + (not a Function URL — this account's SCP blocks unauthenticated Function URLs), + shared-secret auth (`tokenize_api_key`, sent as `x-api-key` / bearer, checked + in-handler). Calls Presidio `/analyze`, resolves overlapping findings (specific + entity wins, so the ROC ID stays `TW_ROC_ID` not `PERSON`), mints random + `CUST_`/`TW_` tokens, writes vault rows + `{token, type, value, session_id, expires_at [, customer_id]}` (`value` = original + PII for /restore; `customer_id` best-effort resolved for the RAG tool), splices + right-to-left, returns `{deidentified_prompt, session_id}`. **Verified live** in + `ap-southeast-1`: `王小明`→`CUST_*`, `A123456789`→`TW_*`, clean prompt, vault rows + written with resolved `customer_id`, repeat→different tokens, no/bad key→401. + Infra changes made while deploying: Presidio task bumped to **2GB** (1GB OOM'd on + the 603MB zh model → connection refused), **awslogs** added, **ARM64/Graviton** + runtime (built on Apple Silicon via podman). All AWS objects tagged + `Owner="conan hncb demo"`. *Caveats:* Presidio has no stable endpoint — its + Fargate public IP changes per task launch, so `presidio_url` in + `terraform/local.auto.tfvars` must be refreshed and the tokenize Lambda re-applied + (an ALB/Cloud Map would fix this; out of scope for the demo). The default VPC was + created by hand and tagged `maintenance=manual-cleanup-required` (Terraform doesn't + own it, so `teardown.sh` won't remove it). +- **T2 — `/restore` endpoint** ✅ **done + live-verified** (`gateway_api/restore/handler.py`, + terraform in `terraform/gateway.tf`). `POST /restore` on the same API Gateway, + same shared-secret auth; IAM is read-only `dynamodb:Scan` on the vault. Input + `{session_id, text}` → scans the vault for THIS session's tokens (scoped, so one + session can't restore another's), swaps inline tokens back to the original PII + (longest-token-first), and prepends the customer's real name at the envelope + level (`（客戶：王小明）`). Returns `{final, restored}`. **Verified live** with the + full T1→T2 round trip: tokenized prompt out → simulated token-only cloud answer → + `王小明` + `A123456789` restored, no tokens left; no/bad key → 401; unknown + session → text unchanged, `restored:0` (no cross-session leak). `restore_url` + is a Terraform output. +- **T3 — reachability from SaaS** ✅ **done + live-verified** (`terraform/ecs.tf`, + `terraform/gateway.tf`). Presidio is now private: its SG allows 5001 only from the + tokenize Lambda's SG (no `0.0.0.0/0`). The tokenize Lambda runs **in the VPC** + (`vpc_config` + `AWSLambdaVPCAccessExecutionRole`) and reaches Presidio on its + **private** IP; DynamoDB is reached via a **gateway VPC endpoint** (no NAT). The + Fargate task keeps a public IP only to pull from ECR, but all inbound is SG-locked. + Only the API Gateway (`/tokenize` + `/restore`, shared-secret auth) is public. + **Verified live:** Presidio's public IP `:5001` now times out from the internet; + the full round trip still works through the private path. *Caveat update:* the + tokenize Lambda now targets Presidio's **private** IP (`172.31.2.216` currently), + which still changes per task launch — refresh `presidio_url` in + `terraform/local.auto.tfvars` and re-apply `aws_lambda_function.tokenize` if the + task restarts (Cloud Map/ALB would give a stable name; out of scope for the demo). +- **T4 — AgentCore Gateway target** ✅ **done + live-verified** (`terraform/agentcore.tf`, + `scripts/agentcore_setup.sh`, `scripts/gateway_invoke_test.py`). Created an MCP + Gateway (`hncb-rag-gateway`, `authorizerType=AWS_IAM` so inbound is SigV4 — no + Cognito) with the RAG Lambda registered as a `lambda` MCP target; the Gateway + assumes a Terraform-managed IAM role that can only `lambda:InvokeFunction` the RAG + tool. Built the Gateway/target with `aws bedrock-agentcore-control` (Terraform + lags here). **Verified live:** `gateway_invoke_test.py` (SigV4 MCP + initialize→tools/list→tools/call) returned the full de-identified evidence package + for `CUST_000123` through the Gateway; the tool shows up namespaced as + `rag___get_customer_activity_summary`. Gotchas handled: the Gateway `inputSchema` + is a restricted JSON-Schema subset (strip `default`/unknown keys); MCP responses + can be SSE; tool names are target-prefixed. Gateway tagged `Owner="conan hncb demo"`; + `teardown.sh` deletes it (not in Terraform state). MCP URL is printed by the setup + script. *Note:* `.venv-t4/` (gitignored) holds boto3 for the invoke test. +- **T5 — Agent runtime** ✅ **done + live-verified** (`agent/agent.py`, + `agent/requirements.txt`, `scripts/agentcore_setup.sh` T5 section). Pinned deps + (bedrock-agentcore 1.16.0 / strands-agents 1.45.0 / mcp 1.28.1 / boto3 1.43.38), + `agentcore configure --non-interactive` + `agentcore launch` (cloud CodeBuild, + auto-created execution role + ECR + STM memory). **Runtime ARN:** + `arn:aws:bedrock-agentcore:ap-southeast-1:286171702468:runtime/hncb_advisor_agent-duWZOT5Far`. + **Verified live:** `agentcore invoke {"prompt": "...CUST_000123..."}` returns + numbered talking points built only from the de-identified evidence — no name, no + leaked token. Three gotchas fixed (all captured in code/scripts): + 1. In `ap-southeast-1` the model is **INFERENCE_PROFILE-only** → use + `apac.anthropic.Codex-3-5-sonnet-20241022-v2:0`, not the raw id. + 2. Our Gateway is **AWS_IAM**, so `agent.py` SigV4-signs MCP requests (httpx.Auth) + instead of using a bearer token. + 3. The auto-created runtime **execution role can't call the Gateway** by default → + must attach `bedrock-agentcore:InvokeGateway` on the gateway ARN (the launch + step in `agentcore_setup.sh` shows the exact command). Runtime + memory added to + `teardown.sh`; agent logic was de-risked locally before launch. +- **T6 — UI wiring** ✅ **done + live-verified** (`ui/index.html`, `terraform/ui.tf`, + `gateway_api/orchestrator/`). Hosted on **S3 + CloudFront** (private bucket, OAC). + For the Fusion-less dry run I added a thin **demo orchestrator** (`POST /demo`, a + Lambda on the same API) that runs the full chain server-side — + `/tokenize` → SigV4 `InvokeAgentRuntime` → `/restore` — and returns + `{final, deidentified_prompt, agent_tokenized}` (this stands in for Fusion ONLY + for the dry run; we still never build/host Fusion). `index.html` reads the live + endpoint from a generated `config.js` (so no ephemeral URL is committed) and + renders the split view. CORS enabled on the API. **UI:** + `https://d3n89cj9w7ynf0.cloudfront.net`. **Verified live:** CloudFront serves + `index.html` + `config.js` (200); `/demo` returns `王小明`→`CUST_*` on the cloud + side and `王小明`-restored talking points on the advisor side. Only the browser + render itself is left as a human eyeball check. All resources tagged. +- **T7 — E2E rehearsal** ✅ **done + live-verified** (README demo/deploy sections + rewritten with the real commands). Ran the full round trip via `/demo` + (`王小明`→`CUST_317499` out, agent talking points on tokens only, `王小明` restored). + **Money shot proven:** in the cloud runtime CloudWatch trace for that session the + token appears in **13 events** while `王小明` and `A123456789` appear in **0** — + the Bedrock/AgentCore trace is token-only. **Teardown verified (non-destructively):** + `terraform destroy` targets 43 resources (Fargate, CloudFront, S3, DynamoDB, all + Lambdas, API GW, VPC endpoint, SGs, IAM) and `teardown.sh` additionally deletes the + Gateway/targets, Runtime, Memory, and both ECR repos — i.e. every billed resource. + Residual is free-only: the hand-made default VPC, the AgentCore SDK exec role, an + idle CodeBuild project. The actual `teardown.sh` run is left for the human to fire + after recording (didn't auto-destroy the live demo). + +## Known caveats (don't "fix" these silently) +- zh-TW detection is demo-narrow (tuned to the scripted entities), not production recall. +- The agentic token-resolution loop (T4) is custom orchestration by design. +- If you touch `fusion/`, it's documentation for a human — do not try to automate SaaS config.