maxim & bifrost

bifrost

i joined early, before stable v1 and i've stayed on the core team since, shipping across pretty much every layer — provider integrations, the compat plugin, billing, config storage, observability and the test infrastructure. this is what i've mostly worked on:

providers

• new providers: elevenlabs (speech + transcription), groq stt / tts, openrouter stt / tts, and deepseek as a first-class provider.
• anthropic-compatible apis for deepseek, fireworks and sglang (set per key), plus anthropic files api support (file_id references, content-type forwarding).
• vllm: moved it to the native responses api, fixed stream truncation and token usage under --skip-pipeline, and added a ci workflow that starts real vllm instances to test against.
• multi-deployment: changed ollama and sglang from one base url per provider to one per key, so you can load-balance across local instances serving different models (and older configs still work).
• azure model router, bedrock mantle and vertex: cost calculation, routing grok models, and usage tracking plus model-name normalization for anthropic models on vertex.

litellm → bifrost migration

built the migration tool for teams moving off litellm. it reads litellm's database directly and maps models, virtual keys, users, teams and orgs to their bifrost equivalents.

billing & cost accuracy

• batch api pricing: batch jobs can span several models, but the log row only records model: "mixed", which has no price. i reprice those rows model by model from the batch's per-model breakdown and write the cost and breakdown in one update so they can't disagree. i also stopped the background recalculation job from zeroing batch rows or charging for duplicate rows, and show batch pricing in the logs ui.
• governance on batches (enterprise): model allowlists and access-profile budgets are now enforced on inline batch requests and on requests that have no model.
• streaming cost fixes: image generation/edit streams, and pricing overrides set on a virtual key or provider key that weren't being applied to streamed responses.

config & storage

• encryption backfill deadlock: when several nodes booted against the same postgres, their startup encryption passes locked rows in opposite orders and deadlocked, and offset pagination skipped or repeated rows while they were being updated. fixed it with one transaction and one row lock per row, cursor-based pagination and concurrent vault writes, and tested it with multiple nodes running against postgres.
• plugin config sync: replaced version-based plugin merging with sha-256 hashes of each plugin's config.json entry. if the file is unchanged, edits made in the ui or api survive restarts; if it changed, the file wins. older rows get a hash added on first boot.

compat plugin

this is the layer that makes "any client, any provider" actually work. it roughly does this:

• request-type conversion — if a model doesn't support the endpoint a client called, the request is converted (say chat → responses), sent to the model, and the response converted back. streaming too: responses events get rebuilt from chat chunks.
• datasheet-driven param dropping — unsupported parameters are dropped or converted per model, based on bifrost's datasheet. most compatibility fixes now ship as a datasheet update instead of a release, and older gateways stay compatible longer. every drop or conversion is logged, so it's never silent.
• coding harness support — codex compatibility (namespaced tools, dropping the web-search tool) and claude code fixes (empty thinking blocks, trailing assistant messages). on azure, deepseek's responses endpoint rejects reasoning.effort and its chat endpoint doesn't support web search, so requests from claude code, codex, cursor and opencode go through chat completions while everyone else stays on responses.
• reasoning across providers — dropping reasoning when a model can't combine it with tool calls, disabling deepseek's thinking under tool_choice: required, carrying summary text and signatures through anthropic/bedrock/gemini reasoning streams, and working out finish_reason from the responses api's final state.
• a long tail of per-provider quirks — cachePoint/cache_control for models without prompt caching, the developer role on anthropic/bedrock/gemini, reasoning_content on groq and cerebras, mistral's reasoning effort values.

observability

• multiple otel collector profiles — send traces and metrics to several destinations at once (datadog, langfuse, prometheus, etc).
• logs ui: time-based filtering and deletion, the model the provider actually served, and cached vs uncached input token breakdowns.

testing & infra

wrote a large share of bifrost's test suites: playwright e2e tests for providers, virtual keys, mcp oauth and the mcp registry (and rbac, guardrails and access profiles on enterprise), and azure / bedrock / vertex / batch cases in the provider test harness. also cleaned up the helm values schema and wrote docs for codex, github copilot, openwebui and azure model router.

---

maxim

maxim is an ai observability and evaluation platform — logging, tracing, evals, test runs, prompt management, and simulations for teams building on LLMs. i have worked across the whole stack — the main platform (next.js monorepo on Nx), and both Python and JS/TS SDKs.

internal bifrost deployment

maintains internal bifrost instance that powers maxim's ai layer. it allows us to do things like:

• BYOK (bring your own key), provide provider-specific configs (like vertex's regions, bedrock arn, etc.), multiple deployments for Ollama and vLLM.
• provider and model discovery — maxim supports 1000+ models, across 20+ providers.
• inference, voice simulations, and streaming all go through bifrost.

human & retro evals

external human raters on log repositories, a variable catalog for evaluator suggestions, and the human-eval sheet/table UI (annotation forms, comparison views, xlsx export for human evaluations in test runs).

logging pipeline

consolidated the logging surface into a single logging API, added a LogLine class in the JS SDK for manually pushing logs, and fixed a nasty bug where multiple Maxim SDK instances on the same API key stepped on each other.

added the LogLine apis to collect log lines and push them to the logging apis. this gives client ownership of the log export process.

kept both SDKs in lockstep — prompt id / prompt version passthrough, variable mapping, withLogger/with_logger for logging on prompt runs, streaming fixes for the agno integration (python), and making sure a broken log-repository connection doesn't block logger creation.

prompts, evaluator, simulation apis

built out a big chunk of the public API surface — prompt tools, prompt versions v2, prompt deployment by version number, prompt partials, and model / evaluator management apis. Also shipped variable mapping for SDK test runs and prompt-version fallback logic when creating new versions.

ui work

• session table improvements (tool-call display, saved search, tooltips for cost / tokens / prompt versions)
• drag-to-reorder and filtering in trace and session tables
• dataset split search and migration fixes
• dashboard and email-template fixes and a long tail of UI bugfixes (dropdown state, title race conditions, drag-and-drop, rich text handling).