Data Scientist

The role

As the first Data Scientist focused on evaluation, you will establish how to measure data quality and safety. You will build datasets, experiments, quality measures, and feedback loops to improve models and pipelines.

What you will do

• Define high-quality and safe-to-deliver data across de-identification, structure preservation, semantic coherence, and customer utility
• Design representative samples and build corpora with explicit provenance, labeling policy, agreement, adjudication, and versioning
• Turn ambiguous concepts into measurable claims with known uncertainty
• Evaluate detectors, models, prompts, judges, thresholds, review workflows, and pipeline changes
• Break aggregate results into modalities, providers, entity classes, customer contexts, languages, formats, and risk tiers
• Connect local measures to escaped sensitive information, over-redaction, preserved data utility, review burden, rework, and delivery acceptance
• Build reproducible analysis, evaluation pipelines, and high-fidelity environments using Python and SQL
• Establish holdout and evaluation practices
• Use modern AI tools for analysis, corpus development, coding, review, and hypothesis generation

Required qualifications

• At least three years of professional experience in applied science, data science, machine learning, quantitative research, or a closely related role
• Designed evaluations or experiments that changed a product, model, release, or operational decision
• Understand sampling, uncertainty, precision, recall, F1, calibration, agreement, class imbalance, distribution shift, and imperfect labels
• Can investigate messy, multi-stage data systems
• Comfortable writing Python and SQL and building reproducible technical artifacts
• Can protect the independence of an evaluation while collaborating closely with the people whose work it evaluates
• Startup experience
• Use AI tools fluently
• Communicate uncertainty and difficult findings directly

Nice to have

• Experience evaluating NER, entity resolution, information extraction, document understanding, multimodal, retrieval, or LLM systems
• Experience with privacy, de-identification, data quality, model risk, safety, or other high-trust decision systems
• Experience designing human-review, adjudication, weak-supervision, or active-learning systems
• Experience building adversarial corpora, replay systems, simulation environments, programmatic verifiers, or model-judge evaluations
• Experience connecting offline measures to escaped defects, customer outcomes, review effort, or preserved data utility
• Experience measuring quality across multi-stage batch or data pipelines

About the company

Replay is building the first-ever bank for enterprise data. The company helps businesses monetize historical operating data by licensing it to frontier AI and data labs for model training.