Senior AI Systems Quality Engineer

The role

You will build and ship automated validation frameworks and evaluation pipelines for AI systems. You will design an AI testing platform integrated with Databricks and MLflow to ensure agentic systems are production-ready and trustworthy.

What you will do

• Build and ship production-grade, automated validation frameworks, test harnesses, and evaluation pipelines.
• Design and evolve an AI testing platform integrated with Databricks and MLflow.
• Create large-scale, scenario-based test suites to validate agentic workflows.
• Validate orchestration behavior and stress-test non-deterministic system behavior.
• Embed quality by design through system contracts, guardrails, and safe-degradation patterns.
• Define measurable quality signals for LLM systems and integrate them into CI/CD pipelines.
• Ensure AI validation runs automatically on model, prompt, and code changes.
• Build reusable libraries and components for consistent AI quality practices.
• Own aspects of AI release readiness and go/no-go criteria.
• Partner with AI, platform, security, and delivery teams to translate mission needs into quality criteria.

Required qualifications

• 7+ years of software engineering experience, primarily in backend or platform systems.
• Proven experience designing and implementing AI testing automation in production environments.
• Demonstrated ability to build custom validation, evaluation, or testing frameworks for complex, distributed systems.
• Strong proficiency in Python and/or TypeScript within modern AI engineering stacks.
• Hands-on experience with AI-powered systems, including LLM-based or agentic workflows and non-deterministic behavior.
• Experience designing or contributing to AI testing at scale, including regression frameworks, long-tail evaluation, and large test coverage.
• Deep understanding of CI/CD integration, including embedding automated tests and quality gates into deployment pipelines.
• Solid understanding of AWS cloud-native architectures.
• Track record of engineering for quality, reliability, governance, and safety as core system design principles.
• Working knowledge of security, privacy, and operational risk in regulated or mission-critical environments.
• Experience with AI testing methodologies, including evaluation of non-deterministic outputs, drift detection, bias/fairness testing, and robust regression strategies.
• Proven ability to establish measurable trust thresholds for AI systems.

Nice to have

• Experience with Databricks-native environments and Medallion architecture.
• Experience using MLflow for model evaluation, lineage tracking, and auditability.
• Exposure to observability tools (e.g., Datadog, Prometheus, Grafana) for distributed AI workflows.
• Familiarity with LLM evaluation techniques, guardrails, and policy enforcement frameworks.
• Experience evaluating performance, latency, or cost regressions in AI systems.
• Ability to clearly document system behavior and quality trade-offs for technical and business audiences.
• Formal training or certification in AI/ML systems (e.g., ISTQB AI Testing, AWS ML Specialty, Google ML Engineer).
• Experience designing and iterating on prompts, agent behaviors, and orchestration logic.
• Familiarity with using AI systems to generate and expand test scenarios.

Benefits

• Unlimited paid time off
• Work from anywhere
• Comprehensive health coverage
• Equity for every employee
• Growth-focused environment
• Home office setup allowance
• Monthly cell phone allowance

Before you apply

• Requires current authorization to work in the United States without the need for employer sponsorship.
• Interviews are conducted by live video with your camera on.
• Candidates may be asked to complete identity and location verification.

About the company

Abacus Insights is transforming how data and AI work for health plans. They provide an AI and Data Platform for Payer Intelligence to help health plans create a single, trusted data foundation.