Research Engineer, Evals & RL Environments

The role

You will build evaluation harnesses, simulated environments, and data pipelines to turn clinical knowledge into training assets. You will work with engineers and clinicians to define tasks, build systems, and ensure results are trustworthy.

What you will do

• Build harnesses that run agents through reproducible tasks
• Develop multi-step and multi-visit environments with patient state and clinical tools
• Work with clinicians to turn clinical workflows into tasks, rubrics, and success criteria
• Build data pipelines for preparing, validating, versioning, and tracing datasets
• Develop scoring and reward mechanisms
• Test graders and environments for ambiguity, shortcuts, data leakage, and reward hacking
• Run experiments, analyze trajectories, and identify performance improvements
• Work with applied AI engineer to turn evaluation findings into product improvements
• Build with privacy, access controls, and permitted data use in mind

Required qualifications

• Strong programming skills and experience building reliable software for experiments, data processing, or agent systems
• Hands-on experience building an agent evaluation harness, an RL environment, or closely related infrastructure
• The ability to translate an open-ended problem into a runnable task with measurable outcomes
• Careful judgment about data quality, reproducibility, grading reliability, and benchmark results
• Comfort debugging across datasets, environment state, agent behavior, and scoring code
• The ability to work with clinicians and turn domain expertise into precise specifications and tests
• Evidence of independent execution through shipped systems, research implementations, open-source work, or projects you’ve owned

Nice to have

• RL environments, simulated tool use, or long-horizon agent benchmarks
• Synthetic data generation, human review workflows, or dataset quality systems
• Reward design, model-based graders, and validation against expert judgments
• Building evaluation or training data products for AI labs
• Healthcare data, clinical reasoning evaluations, or longitudinal patient workflows
• Healthcare experience

Benefits

• Medical, dental, and vision
• HSA/FSA eligible
• Flexible, unlimited vacation
• Meal stipends
• Team retreats

About the company

Nolla Health is building AI-powered healthcare to improve systems involved in delivering care.