Research Engineer, Benchmarks & Evaluation

The role

You will own the benchmarking effort by turning real finance workflows into benchmarks and RL environments. You will design graders, run benchmark suites across models, and publish external benchmarks on model performance in finance work.

What you will do

• Build benchmarks from real workflows
• Own the benchmarking effort
• Design graders using rubrics, programmatic checks and llm-as-judge
• Close the loop with post-training by turning evals into reward functions and RL environments
• Publish external benchmarks and write-ups

Required qualifications

• Hands-on experience building LLM benchmarks or evals that people used to make real decisions
• Strong Python and software engineering
• Statistical judgment regarding variance, confidence intervals, sample size, and inter-rater agreement
• Working knowledge of post-training (SFT, RL with verifiable rewards) and how evals feed into it
• Comfort with SQL and large datasets
• Taste for what is worth measuring

Nice to have

• Industry experience on an eval, benchmarking or post-training team
• Built RL environments or verifiers for agentic tasks
• Published benchmark or evaluation research (NeurIPS Datasets & Benchmarks, ICML, ACL)
• Built or contributed to eval frameworks (verifiers, lm-evaluation-harness, HELM or similar)
• Familiarity with accounting, finance or other expert-domain work
• Public work we can look at: a benchmark, an eval suite, a failure-analysis write-up

Benefits

• Flexible PTO
• Centralized home-office equipment ordering
• Health and wellness stipend
• Budget for intra-office travel
• Weekly coffee stipend
• 100% medical, dental & vision insurance coverage for you, with partial coverage for dependents
• One Medical annual membership
• 401(k), including employer match on contributions made while employed by Ramp
• Fertility HRA (up to $10,000 per year)
• Parental leave: up to 16 weeks (birthing + bonding) or 8 weeks (bonding only) at 100% pay
• Pet insurance
• In-office perks: lunch, snacks, drinks, and more

About the company

Ramp is building the smart infrastructure for finance teams. Ramp Labs is Ramp's AI research group.