Senior Data Scientist AI Evaluation

The role

You will design how the company measures whether models and agents are correct. You will build ground truth, scoring methods, and evaluation loops to improve systems.

What you will do

• Design AI evaluations including ground truth, metrics, and scoring methods for models and agents.
• Build repeatable eval loops to track quality and catch regressions.
• Work with engineering and analytics engineering to operationalize eval harnesses.
• Translate eval results into actionable recommendations for system improvements.
• Set evaluation guidelines, documentation, and review practices.
• Foster evaluation best practices and build a culture of measurable AI quality.

Required qualifications

• Track record of quantitative measurement rigor such as LLM/model evaluation, metric validation, or experimentation.
• Strong statistical and ML foundation including sample sizing, confidence intervals, significance, and handling non-determinism.
• Ability to validate automated graders against human ground truth.
• Proficiency in Python and SQL.
• Experience evaluating models in production environments.
• Strong judgment in defining quality metrics and ground truth for ambiguous outputs.
• Excellent communication and cross-functional collaboration skills.
• Strong problem-solving ability in fast-paced, greenfield environments.
• 6–10 years in quantitative data science or ML, with focused experience in measurement or evaluation.

Nice to have

• Hands-on LLM/agent evaluation in production, including eval harnesses, LLM-as-judge calibration, and CI regression gates.
• Experience evaluating text-to-SQL, analytics agents, or other systems where correctness is verifiable against data.
• Background in fintech, brokerage, or other domains where a wrong answer has real business or risk consequences.
• Fluency with AI tools in research and engineering workflows.
• A quantitative degree

Benefits

• Competitive Salary & Stock Options
• Health Benefits
• New Hire Home-Office Setup: One-time USD $500
• Monthly Stipend: USD $150 per month via a Brex Card

About the company

Alpaca is a US-headquartered, global leader in agent-first brokerage infrastructure for stocks, ETFs, options, crypto, fixed income, 24/5 trading, and more. It serves hundreds of financial institutions across 40 countries.