Senior AI Engineer — Inference & Agent Systems

The role

This person will drive inference optimization and design agent architectures for real-time AI agents. They will own the evaluation framework and build infrastructure for model serving and observability.

What you will do

• Drive TTFT below 400ms for multi-step agent pipelines
• Optimize streaming and KV cache strategy, prompt compression, and dynamic context window management
• Implement multi-provider routing across OpenAI, Anthropic, Gemini, and open-weight models
• Design and implement Plan-Execute-Synthesize pipelines using parallel DAGs
• Build reliable orchestration on top of Temporal
• Enforce structured output through JSON schema validation and retry loops
• Design tool call schemas
• Own the evaluation framework including ground truth datasets and automated scoring pipelines
• Perform latency regression testing and adversarial test case design
• Manage model serving, cold start optimization, and async worker architecture

Required qualifications

• Built something that runs in production at a meaningful scale
• Worked on inference pipelines where TTFT was the primary metric
• Built multi-step agent systems
• Written eval harnesses from scratch
• Debugged LLM non-determinism in production
• Worked with streaming LLM responses and built infrastructure around partial output handling
• Shipped inference systems at a real-time AI product, a model serving infrastructure company, or an agent platform
• Or built eval/harness infrastructure that a team of 10+ engineers trusted to catch regressions

Nice to have

• Fine-tuned models
• Used LangChain/LlamaIndex
• Strong ML research background
• Stack familiarity with Go, Python, Temporal, Kafka, PostgreSQL, Docker

Before you apply

• Send one system you built where latency was the primary constraint including what you measured, what you changed, and what moved
• Send a link to anything public such as code, writing, or talks
• No cover letter required

About the company

Arcana is building AI agents that synthesize information across heterogeneous sources and deliver structured, reasoned answers in real time.