llmxy Open Source

A public, self-hosted LLM gateway with user and admin consoles, an OpenAI-compatible API, provider routing, quota enforcement, and usage billing.

The repository contains the complete web, administration, and API services required to operate the gateway yourself. A direct FastAPI relay is available for simple deployments, while an optional Envoy path handles higher-throughput traffic without moving billing work onto the request path.

View source on GitHub Explore the Learnmark product

Repository: wilsonwu/llmxy
Deployment: Docker Compose or independently operated services
API surface: OpenAI-compatible chat, embeddings, and model endpoints
Languages: Python and TypeScript

Included in the repository

Open-source capabilities

User console

Provide account access, plans, top-ups, API key management, model discovery, and per-user usage and billing views.

Administration console

Manage users, orders, upstream channels, models, rates, plans, routing policies, route weights, and operational statistics.

Unified API

Expose OpenAI-compatible chat completions, embeddings, and model discovery while connecting to OpenAI, Anthropic, and Gemini upstreams.

Gateway controls

Apply authentication, balance and quota checks, rate limiting, usage recording, and billing through one control plane.

Smart routing

Configure channels, models, routes, policies, and weights from the admin application, with changes propagated to running gateway instances.

Optional Envoy relay

Run a C++ hot path with external authorization and asynchronous gRPC access logs while retaining the direct FastAPI relay for smaller deployments.

Architecture

How the project is organized

Website: Next.js user application for accounts, subscriptions, keys, model access, and usage.
Admin: Next.js control plane for providers, routing, plans, users, billing, and Envoy instances.
API: FastAPI service backed by PostgreSQL and Redis for protocol translation, authorization, quotas, and billing.
Envoy: Optional front proxy for the high-throughput relay path, external authorization, and asynchronous usage reporting.

Technology stack

Built for self-hosting and extension

Python 3.11+FastAPINext.js 14TypeScriptTailwind CSSshadcn/uiPostgreSQL 16+Redis 7+EnvoygRPCDocker Compose

Prerequisites

Docker with Docker Compose for the shortest setup path
Or Python 3.11+, Node.js 18+, pnpm, PostgreSQL 16+, and Redis 7+
Local JWT, encryption, database, and provider credentials

Quick start

git clone https://github.com/wilsonwu/llmxy.git
cd llmxy
cp .env.example .env
docker compose up -d --build

Project repositories

Read the code, documentation, and project history.

Source repository Architecture documentation