DevOps Engineer - AI Runtime Services

The role

This person will own the shared runtime for AI at Demandbase, including the LLM gateway, cost controls, and observability. They will build and support both product-facing runtime and internal AI tooling.

What you will do

• Manage the LLM gateway including keys, routing, rate limits, failover, and provider quotas
• Own reliability, SLOs, on-call, incident response, and postmortems for the runtime
• Control costs through per-team and per-model attribution, budgets, and quotas
• Provide LLM observability including traces, spans, prompt/response capture, and billing visibility
• Build evals and experimentation infra including shared harnesses, datasets, and scoring plumbing
• Implement caching and guardrails for latency reduction and safety enforcement

Required qualifications

• Strong production infrastructure background in Kubernetes, AWS, Terraform, and GitOps
• Experience operating systems that require on-call response
• Python proficiency for owning services and tooling
• Real on-call and incident-response experience
• Observability fluency including system instrumentation and metrics/logging backend management
• Hands-on familiarity with how LLM systems work, such as building agents, RAG pipelines, or internal tools
• Familiarity with tokens, context windows, prompt/response tracing, and eval suites

Nice to have

• An LLM gateway or proxy (LiteLLM or similar) in production
• LLM observability tooling: Datadog LLM Observability, LangSmith, Braintrust, Arize, or similar
• Eval frameworks and LLM-as-judge scoring in a real workflow
• FinOps instincts such as cost attribution, showback, and quota design
• Guardrails, PII detection, or content-filtering systems
• GCP alongside AWS

Benefits

• Group Medical, Personal Accident, and Term Life Insurance
• Preventive healthcare covering dental, vision, and OPD needs
• Mental health support
• Fitness benefit
• Car lease policy
• Gratuity

About the company

Demandbase is a pipeline AI platform that empowers GTM teams to automate growth at scale. The company has offices in San Francisco, New York, Austin, Seattle, India, and the United Kingdom.