Senior / Staff AI Engineer — Evals & Model Quality

The role

This person will own evaluations and continuously improve the quality of AI agents across the platform. They will build evaluation frameworks, high-quality datasets, and automated grading systems.

What you will do

• Build evaluation frameworks and infrastructure to measure AI agent accuracy, quality, and reliability.
• Develop golden datasets, benchmarks, and regression suites.
• Design evaluations for multi-step agent trajectories, tool use, reasoning, and task completion.
• Develop LLM-as-judge evaluations, deterministic graders, human evaluation workflows, and rubrics.
• Integrate offline and online evaluations into development and release processes.
• Benchmark frontier and open-weight models across accuracy, latency, cost, and reliability.
• Turn production failures, customer feedback, and agent traces into new evaluation cases.

Required qualifications

• 5+ years of software engineering experience
• Strong hands-on experience building production AI/ML systems
• Experience building evaluation frameworks, LLM-based applications, model benchmarking systems, or applied AI infrastructure
• Strong understanding of modern LLMs, agent architectures, evaluation methodologies, and model performance tradeoffs
• Strong Python engineering skills
• Ability to build reliable, production-grade systems
• Ability to independently own ambiguous technical problems
• Clear communicator

Nice to have

• Experience with LLM-as-judge, human evaluation systems, agent trajectory evaluations, or large-scale benchmarking
• Experience with open-weight models, fine-tuning, post-training, or model routing
• Background in B2B SaaS and 0-to-1 experience

Benefits

• Fully covered, best-in-class health, dental, and vision benefits
• Competitive compensation
• Meaningful stock options
• Company 401(k)
• Unlimited PTO
• Lunch and dinner onsite
• Team events, such as happy hours and off-sites
• Pre-tax commuter benefits

About the company

FurtherAI is building AI agents that run the insurance industry. The company has raised more than $30M from investors including Andreessen Horowitz and YC.