T8 — Multi-entity treatment: tokenize Credit Card + mask Email (partner feedback)

Status: ⬜ TODO

Why

Partner watched the demo video and gave two pieces of feedback:

Token rotation wasn't hammered home enough (highlighting the text on screen wasn't sufficient — make it undeniable).
They want more than the name de-identified — fields matching their "Azure Processing Content" table (ID/CC/Name tokenized, Email/Mobile masked, etc.).

Name (PERSON) and the Taiwan ROC ID (TW_ROC_ID) are already tokenized (T1/T2). This task adds Credit Card (tokenize) and Email (mask) and makes rotation visible. Mobile masking and Address/Amount generalization are explicitly OUT of scope here (leave for a later T9 if they ask).

Entity → treatment policy

Introduce a small policy map in the tokenize handler and branch per finding:

Entity	Treatment	Token / mask form	Notes
`PERSON`	tokenize (rotate + vault)	`tkn_NAME_<rand>`	already works; just adopt the `tkn_<TYPE>_` naming
`TW_ROC_ID`	tokenize (rotate + vault)	`tkn_ID_<rand>`	already works; adopt naming
`CREDIT_CARD`	tokenize (rotate + vault)	`tkn_CC_<rand>`	enable Presidio built-in `CreditCardRecognizer` (has Luhn) for `zh`
`EMAIL_ADDRESS`	mask (one-way, partial)	e.g. `m*.w*@gmail.com`	enable built-in `EmailRecognizer`; no vault row, not restored

Adopt tkn_<TYPE>_<rand> naming so the on-screen output matches the partner's table one-to-one.

Where

presidio/app.py — ensure CREDIT_CARD and EMAIL_ADDRESS recognizers are active for the zh analyzer (they're built-in; confirm they load alongside the zh model and the custom ROC-ID recognizer; add them to supported entities if needed).
gateway_api/tokenize/handler.py — read the treatment policy; for tokenize entities keep the current mint→vault→splice; for mask entities apply a deterministic partial mask (keep the first char of each local-part segment + the full domain for email), splice, and do not write a vault row.
gateway_api/restore/handler.py — no change needed for masked fields (they're never restored). Confirm restore still only touches tokenized tkn_* values.
gateway_api/orchestrator/ (/demo) + ui/index.html — add the rotation proof (below).
seed/customers.json — optional: add credit_card / email fields for realism; not required since detection reads from the prompt text, not the DB.

Critical gotchas (do NOT "fix" these)

Masked fields must NOT rotate. Masking is deterministic by design — the same email masks to the same string every run. Only tokenized fields (NAME/ID/CC) rotate. Do not add randomness to the mask. If the partner "notices" the email not changing, that is correct behaviour, not a bug.
Email is one-way. The advisor sees it masked in the final answer, never restored. If product later wants the real email back, that is tokenization-with-a-masked-looking-surrogate (goes through the vault) — a different path. Don't silently switch masking to reversible.

Rotation proof (the actual ask from feedback #1)

Make rotation undeniable on camera: run the identical prompt twice and show the tokenized fields differ across runs (tkn_ID_8F72… vs tkn_ID_Xq93…) while the final restored answer is byte-identical and the masked email is identical.

Easiest implementation: a ?runs=2 mode (or a second call) on /demo that returns both tokenized prompts plus the single restored answer.
Surface it in ui/index.html as a "same input → different tokens → same answer" split. Optionally show the two vault entries: different tokens, same real value.
This is far harder to miss than highlighted text.

Acceptance criteria

A prompt containing name + ROC ID + credit card + email produces three rotating tkn_* tokens (NAME/ID/CC) and one stable partial-masked email.
Running the same prompt twice yields different NAME/ID/CC tokens but an identical masked email and an identical restored final answer.
The vault contains rows only for the tokenized fields — never the email.
On-screen token names match the partner's table (tkn_ID_…, tkn_CC_…, tkn_NAME_…).

Demo prompt (partner's)

"Analyze customer Wang Xiaoming (ID No. A123456789, Credit Card 4567-1234-5678-9012, Mobile 0912-345-678, Email ming.wang@gmail.com, Address: No. 3, Songren Rd., Xinyi Dist., Taipei) — most recent transaction of NTD 58,600 — and provide a visit summary and service recommendations."

(Mobile, Address, Amount are out of scope for T8. Note internally: this prompt is contrived — a real agent would have done the research to build it — but we're giving the partner the on-screen show they asked for.)

4.7 KiB Raw Blame History