Staff Applied AI Engineer, Product & Agent Performance

The role

You will own the product-layer decisions that shape how Arcadia's agentic AI behaves in live healthcare workflows. This covers prompting, retrieval and context, memory, evaluation and escalation, working with Product and Engineering.

What you will do

• Design and iterate on agent behavior across real, live workflows, including long-horizon, multi-turn agentic tasks
• Design retrieval and context architecture so agents stay grounded in real source data
• Design memory and state handling across multi-turn and multi-agent flows
• Create context and prompt templates for consistent agent behavior
• Build and run evaluations against real production conditions
• Author evaluation rubrics and thresholds that weight failures by severity and cost
• Design and validate escalation paths that route agents to human review based on confidence and uncertainty
• Design for cost-aware performance alongside latency, reliability and accuracy
• Evaluate and sign off on model changes before they reach a customer
• Maintain product-level AI documentation, including model cards, intended use, limitations and known failure modes

Required qualifications

• Bachelor's degree in Engineering or a related field and/or master's in Data Science or a related field
• 4+ years of production software engineering experience
• 3+ years of hands-on ownership of ML, LLM or agentic systems in production
• Direct experience in healthcare, finance or another regulated industry
• Ability to diagnose why an agent failed and attribute the fix to instruction, retrieval, context or memory design
• Hands-on experience with RAG architecture, production-grounded evaluation frameworks, and fallback or human-in-the-loop logic
• Working familiarity with AWS AI/ML services, including Bedrock and SageMaker
• Evidence-led judgment and the credibility to push back on launch decisions

Nice to have

• Experience applying AI to healthcare data or workflows where safety and calibrated uncertainty affect care teams or patients
• Experience with long-horizon, multi-turn or multi-agent workflows
• Experience with product-level AI documentation such as model cards

Benefits

• Opportunity to define how agent performance, safety and readiness are measured for production healthcare workflows
• Meaningful ownership across prompts, context, memory, evaluations and escalation patterns at product scale
• Cross-functional role translating production evidence into AI improvements across the platform
• Mission-driven company working to improve how patients receive care
• Flexible, remote-friendly culture
• Employee-driven programs and initiatives for personal and professional development

Before you apply

• Staff-level individual contributor role

About the company

Arcadia is a healthcare data and analytics platform used by health systems, ACOs, payers and life sciences organizations. It is backed by Nordic Capital.