Senior Software Engineer - LLM Ops & Evals

The role

You will own the LLM gateway and build a shared evaluation platform for measuring AI quality. You will manage infrastructure for AI workloads and ensure observability, security, and reliability for production systems.

What you will do

• Own the LLM gateway including routing, provider failover, rate limits, retries, and cost attribution
• Deploy, version and deprecate models across clouds, regions and environments using infrastructure as code
• Build the shared evaluation platform with versioned datasets, experiment tracking, and reporting
• Own the infrastructure for async and long-running AI workloads
• Own observability for AI traffic including latency, token usage, cost and errors
• Take part in the on-call rotation, run incidents, and close the follow-ups
• Implement security and compliance controls like retention, access control, RBAC and SSO
• Define and maintain clean integration contracts between the platform and consuming teams
• Partner with product and ML engineers to create self-service platform capabilities

Required qualifications

• 5+ years in backend or platform engineering
• Strong production Python
• Experience building or running LLM inference infrastructure
• Experience with cloud at the infrastructure level and infrastructure as code
• Experience running a shared service in production
• Hands-on experience with observability tools (OpenTelemetry, Grafana)
• Comfort with privacy and compliance work including PII handling, anonymisation, retention, and access control
• Experience building platform or shared-service capabilities consumed by multiple internal teams

Nice to have

• Experience with LLM or agent evaluation
• Temporal or another durable workflow engine
• Self-hosted inference, capacity planning, load testing
• Synthetic data or document anonymisation pipelines
• Document AI: VLMs, OCR, structured extraction and the metrics that go with it
• Domain experience in audit, accounting or fintech

About the company

DataSnipper builds AI solutions that empower audit and finance professionals. The platform is used by more than 600,000 professionals across 3,000+ organizations in 175 countries.