Evaluations Team Lead

The role

You will lead a team to build a shared evaluation platform for the NEXUS model. You will define evaluation standards, benchmark the model against competitors, and manage a small team while staying hands-on with code and experiment design.

What you will do

• Build a shared evaluation platform for engineering, research, and Applied AI.
• Define evaluation standards for metrics, data splits, leakage prevention, calibration, uncertainty, and benchmark contamination.
• Build reproducible pipelines with versioned inputs and artifacts.
• Integrate regression checks into research and release workflows.
• Benchmark NEXUS against competing approaches using fair tuning budgets, data access, compute, and latency measurement protocols.
• Support Applied AI’s customer POC evaluations with tooling, methodological guidance, and analysis.
• Measure predictive quality, latency, and cost across deployment configurations, task types, and dataset characteristics.
• Turn findings into research priorities, release recommendations, and evidence-backed customer improvement plans.
• Hire and develop the team, set priorities, and stay hands-on with code and experimental design.

Required qualifications

• Experience owning evaluation for tabular ML systems used in production or consequential customer decisions.
• Strong statistical judgment: choosing metrics and validation schemes, estimating uncertainty, comparing models across datasets, and accounting for repeated experimentation.
• Practical experience finding leakage in preprocessing, feature construction, joins, temporal dependencies, and related entities across splits.
• Strong Python and SQL skills.
• Familiarity with scikit-learn and gradient-boosted trees.
• Experience building reliable ML tooling or platforms used by other teams.
• Experience designing fair model comparisons, including hyperparameter search, resource budgets, and end-to-end latency measurement.
• Prior people management experience, including hiring, technical coaching, and performance feedback.
• Clear written and spoken communication with researchers, engineers, and customer data scientists.

Nice to have

• Experience evaluating tabular foundation models or AutoML systems.
• Experience measuring how model optimisations affect predictive quality and inference performance.
• Experience with relational or multi-table data, and Snowflake or Databricks environments.
• Experience evaluating automated or agent-driven ML workflows, including failures that aggregate metrics can hide.

Benefits

• Competitive compensation with salary and equity
• Comprehensive health coverage for you and your dependents
• Paid parental leave for all new parents, inclusive of adoptive and surrogate journeys
• Relocation support for employees moving to join the team in one of our office locations

About the company

Fundamental is an AI research lab pioneering the future of enterprise decision-making. Their flagship model, NEXUS, is a Large Tabular Model used by Fortune 100 companies.