Fieldguide

Senior AI Engineer, Quality

San Francisco, CA or Remote (USA). Posted 6 months ago

The role

You will own the evaluation infrastructure to ensure AI agents perform reliably at enterprise scale. You will build a unified platform, automated pipelines, and production feedback loops for model evaluation.

What you will do

  • Design and build a unified evaluation platform for agentic systems and audit workflows
  • Build observability systems to surface agent behavior and failure modes
  • Own the evaluation infrastructure stack including integration with LangSmith and LangGraph
  • Translate customer problems into concrete agent behaviors and workflows
  • Integrate and orchestrate LLMs, tools, retrieval systems, and logic
  • Build automated pipelines to evaluate new models within hours of release
  • Design evaluation harnesses for complex Agentic systems and workflows
  • Implement comparison frameworks for effectiveness, consistency, latency, and cost
  • Design guardrails and monitoring systems to catch quality regressions
  • Work with SMEs and ML Engineers to create evaluation datasets

Required qualifications

  • Multiple years of experience shipping production software in complex, real-world systems
  • Experience with TypeScript, React, Python, and Postgres
  • Built and deployed LLM-powered features serving production traffic
  • Implemented evaluation frameworks for model outputs and agent behaviors
  • Designed observability or tracing infrastructure for AI/ML systems
  • Worked with vector databases, embedding models, and RAG architectures
  • Experience with evaluation platforms (LangSmith, Langfuse, or similar)
  • Comfort operating in ambiguity and taking responsibility for outcomes

Nice to have

  • experience with audit and accounting workflows

Benefits

  • Competitive compensation packages with meaningful ownership
  • Flexible PTO
  • 401k
  • Wellness benefits, including a bundle of free therapy sessions
  • Technology & Work from Home reimbursement
  • Flexible work schedules

About the company

Fieldguide builds software for assurance and audit practitioners in cybersecurity, privacy, and financial audit. The company is a remote-first, San Francisco-based Vertical AI company.