Senior AI Engineer — Inference & Agent Systems

The role

This person will build AI agents that synthesize information across heterogeneous sources. They will work on inference optimization, agent architecture, evaluation frameworks, and infrastructure.

What you will do

• Drive TTFT below 400ms for multi-step agent pipelines
• Optimize streaming and KV cache strategy, prompt compression, and dynamic context window management
• Implement multi-provider routing across OpenAI, Anthropic, Gemini, and open-weight models
• Design and implement Plan-Execute-Synthesize pipelines using parallel DAGs
• Build reliable orchestration on top of Temporal
• Enforce structured output via JSON schema validation and retry loops
• Design tool call schemas
• Own the eval framework including ground truth datasets and automated scoring pipelines
• Perform latency regression testing and adversarial test case design
• Manage model serving, cold start optimization, and async worker architecture

Required qualifications

• Built something that runs in production at a meaningful scale
• Worked on inference pipelines where TTFT was the primary metric
• Built multi-step agent systems
• Written eval harnesses from scratch
• Debugged LLM non-determinism in production
• Worked with streaming LLM responses and built infrastructure around partial output handling
• Shipped inference systems at a real-time AI product, a model serving infrastructure company, or an agent platform
• Or built eval/harness infrastructure that a team of 10+ engineers trusted to catch regressions

Nice to have

• Fine-tuned models
• Used LangChain/LlamaIndex
• Strong ML research background
• Stack familiarity with Go, Python, Temporal, Kafka, PostgreSQL, Docker

Before you apply

• Include one system you built where latency was the primary constraint, what you measured, what you changed, and what moved
• Include a link to anything public such as code, writing, or talks
• No cover letter required

About the company

Arcana is building AI agents that synthesize information across heterogeneous sources and deliver structured, reasoned answers in real time.