In production

AI Gateway for an Enterprise Bank

One governed gateway between a bank’s teams and every LLM.

LiteLLM
AWS Bedrock
FinOps

Screenshot of the Bedrock cost reconciliation page comparing gateway spend with Bedrock invocations — Bedrock cost reconciliation

Client: Enterprise banking client
Timeline: Aug 2026 – Present
Role: AI Gateway Platform Engineer
Outcome: Spend matched to the real AWS bill

About the project

The bank runs open-source LiteLLM as one AI gateway for its internal teams. I extended it with a separate package and admin pages, without changing the LiteLLM core, so upstream releases still merge: a tamper-evident audit log, Vietnamese PII guardrails, SSO and an auditor role, budget-aware routing, alerts and bulk onboarding. I also built Bedrock cost reconciliation against invocation logs and the AWS bill (Athena) by team, user and model, with Bedrock tax shown separately; checking it against the real invoice exposed and fixed a 4x over-count. I own delivery and planning since Aug 2026; the lead engineer reviews and merges.

The problem

An enterprise bank runs open-source LiteLLM as one AI gateway in front of the models its internal teams use. To run it as a bank, it needed enterprise controls: spend that matches the AWS bill, an audit trail, protection for personal data and proper access control. It had to get there without commercially licensed code and without changing the LiteLLM core, so the bank can keep taking upstream releases.

What I built

I work on an outsourced contract in a two-engineer team, since August 2026: I built the extension layer and own delivery and planning; the lead engineer reviews and merges. Everything ships as a separate extension package plus admin pages on top of LiteLLM:

Tamper-evident audit log — append-only and hash-chained, exportable to Sentinel.
Vietnamese PII guardrails — detection of Vietnamese personal data, with a review flow for false positives.
SSO and an auditor role for the people who review how the gateway is used.
Budget-aware routing — requests are downgraded to a cheaper model as a budget runs low.
Alerts — by email, SES and SNS, with editable templates.
Bulk onboarding — teams and users imported from a spreadsheet.
Cross-account access through assumed roles, X-Ray-linked traces, and Terraform for the AWS stack.

How it works

The core piece is Bedrock cost reconciliation. Every day it checks the gateway's own spend records against Bedrock invocation logs and the AWS bill (queried with Athena), broken down by account, team, user and model, with Bedrock tax shown separately so the numbers line up with the invoice. When I validated it line by line against the real invoice, it exposed a 4× over-count, which I traced and fixed in the reconciliation. Because the controls live outside the LiteLLM core, upstream releases still merge cleanly.

Outcome

Gateway spend now matches the real AWS bill, every request leaves an audit record that cannot be silently edited, and personal data is screened before it reaches a model. I wrote the architecture docs and runbooks for the handover. The work is ongoing: I still maintain and extend the gateway.