LiteLLM
Berrie AI Incorporated (BerriAI) ·www.litellm.ai· As of: 2026-07-30
Our verdict
A model gateway makes nobody distinguishable. Without one you lose sight of spend and access, you cannot swap a model when price or quality shifts, and you have no evidence when someone audits you. With one you have merely removed the risk of falling behind. The edge comes from the applications on top, not from the routing layer.
Who it fits
Solo
suitable with caveats
The library pays off immediately. Running the proxy with its own database, updates and security upkeep is usually more work than a single application justifies for one person.
Mid-market
suitable
From the second team that calls models onward, the proxy is cheaper than scattered keys. Self-hosting costs no licence, only operating time, and the credentials stay in house.
Enterprise
suitable with caveats
Directory based sign-in, audit logs and access controls sit behind the paid edition, whose price is not published. On top of that the vendor is small, and the compliance reports are only handed out under contract.
Buy, switch or build
Off-the-shelf
Straight to the model provider or through its cloud gateway
Every application talks to the model directly, or everything runs through one large cloud provider's gateway. Fastest to start, and billing already sits inside an existing contract. The price is lock-in: keys spread out, swapping a model turns into a rebuild, and the spend view stops at that provider's boundary.
AI-native
Managed gateway as a service
Services such as Portkey, OpenRouter or Helicone deliver the same routing as a ready made endpoint with nothing to operate. That saves time and costs the decisive point: every call and every log passes through a third party. For companies with audit duties that is the harder conversation, not the easier one.
Build it yourself
Your own thin proxy in front of the provider APIs
A two hundred line service that holds keys and forwards calls is built in a day. The expensive part is what follows: fallbacks, retries, per model cost accounting, streaming, tool calls, onboarding each new provider. That aftermath is precisely what the tool contains, and it is available as open source.
Our advice
- Solo: Use the library, do not run the proxy yet.
- Mid-market: Self-host. The free scope covers key management, budgets and fallbacks.
- Enterprise: Self-host, but get price, audit report and response times in writing before licensing.
Who is behind it
Staying power: Young
Three years in market, a small team and a funding round that is tiny relative to how widely the software is used. Adoption of the code is the strongest signal here, not the vendor's balance sheet.
- Founded and cohort: 2023, Y Combinator Winter 2023, founders Krrish Dholakia and Ishaan Jaffer · Source · As of 2026-07-30
- Funding: USD 1.6 million seed funding (Y Combinator, Gravity Fund, Pioneer Fund); higher figures circulate in databases without vendor confirmation · Source · As of 2026-07-30
- Team size: 10 people according to the company's own entry in the Y Combinator directory · Source · As of 2026-07-30
- Code adoption: around 55,100 stars and 10,200 forks on GitHub, with releases shipping continuously · Source · As of 2026-07-30
- Attestations: SOC 2 Type 2 and ISO 27001 per the company page; elsewhere the documentation still names SOC 2 Type I with a Type II process under way · Source · As of 2026-07-30
- Security incident: 24.03.2026: compromised packages v1.82.7 and v1.82.8 on PyPI carrying a credential stealer, available for around 40 minutes; Docker images and source code not affected · Source · As of 2026-07-30
Cost of leaving: Low
The core is MIT licensed and runs on your own infrastructure. The interface is OpenAI compatible, the configuration lives in a file, and keys and spend logs live in a Postgres database you own. Leaving means pointing the applications back at the model providers directly and taking your data with you. The only real cost is giving up the paid features once you depend on them.
Regulation and data
| Data processing agreement | not required when self-hosted, not publicly substantiated for the managed cloudIn self-hosted mode the vendor processes no content, so no processing agreement is needed. A publicly available data processing agreement for the managed edition could not be found on litellm.ai or docs.litellm.ai.Source · As of 2026-07-30 |
|---|---|
| Storage location | your own infrastructure when self-hosted; managed edition either in the EU region Frankfurt (eu-central-1) or in the USA (us-west-1, us-east-1)For self-hosting the vendor states explicitly that no data and no telemetry reach its servers. The location is therefore determined solely by your own environment.Source · As of 2026-07-30 |
| Subprocessors | none when self-hosted; the managed edition runs on GCP and AWS, with the database partly on NeonDBNo complete, maintained subprocessor list with change notification is published. Only these three names appear in the documentation.Source · As of 2026-07-30 |
| Third-country transfer | controllable when self-hosted, depends on the model providers connectedThe gateway itself transfers nothing to a third country. The transfer arises from the model behind it, and that selection can be restricted per key and per team in the configuration.Source · As of 2026-07-30 |
| Training on customer data | no vendor access to content when self-hostedBecause the calls never leave your own environment, the training question does not arise against this vendor but against whichever model provider sits behind it. That stays a separate check per model.Source · As of 2026-07-30 |
| Retention and deletion | set by the operator itself, spend logs deletable automatically via a retention periodThe documentation shows a configuration value that deletes logs after a set period. The operator sets that period, not the vendor.Source · As of 2026-07-30 |
| Certifications | SOC 2 Type 2 and ISO 27001 per the company page, reports only on request on the paid planThe statements contradict each other. The company page names SOC 2 Type 2, while the documentation page still says SOC 2 Type I with a Type II process whose target date has long passed. Ask for the actual report before signing.Source · As of 2026-07-30 |
| EU AI Act, Article 50 | unclearNeither the company page nor the documentation addresses the transparency duties. The gateway is not a model provider, so the duty usually falls on whoever operates the application. What helps is that every call can be logged in one place.no source given · As of 2026-07-30 |
| Audit logging | spend and usage logs in the free edition, audit logs with a retention rule only in the paid editionAnyone who needs an auditable trail over key issuance and permission changes needs the licence. Plain cost and usage transparency comes without one.Source · As of 2026-07-30 |
What it really costs
Entry price
The core is free under the MIT licence, runs as a Docker image and needs a Postgres database of your own. Directory based sign-in is free for up to five people. At first the only cost is running it.
Source · As of2026-07-30
What gets expensive
In two places. First at the step into the paid edition: directory based sign-in beyond five people, audit logs, IP based access limits and the response times only come with a licence, and its price is published nowhere. Second in operations: the March 2026 incident shows this layer has to be patched fast. Anyone without that as a fixed routine pays for it in another currency later.
What it displaces
- Model keys scattered across individual services and people
- Vendor specific libraries buried in application code
- Cost visibility per model provider instead of per team
Interfaces
- OpenAI-compatible HTTP API for more than 100 providers
- Python library
- Configuration file and admin interface
- Your own Postgres database for keys, budgets and logs
- Log export to OpenTelemetry, Langfuse, Langsmith, Arize
At Convios: Not used by us
Not in use at Convios. This record rests on the vendor's public sources, not on our own operating experience.
Evidence
- 1.Core under MIT licence, the enterprise directory under a separate licence · As of 2026-07-30
- 2.Paid scope, response times, attestations, free sign-in for up to five people, price on request only · As of 2026-07-30
- 3.No data and no telemetry at the vendor when self-hosted, regions of the managed edition, attestations · As of 2026-07-30
- 4.Audit logs, access restrictions, tenant separation and guardrails in the paid scope · As of 2026-07-30
- 5.Compromised packages v1.82.7 and v1.82.8 on 24 March 2026, Docker images unaffected · As of 2026-07-30
- 6.Founding year, cohort, founders, team size and seed funding · As of 2026-07-30
- 7.Retention period for spend logs is configurable · As of 2026-07-30
Last reviewed: 2026-07-30byDr. Oliver Gausmann, Convios GmbH
Details out of date? Let us know.