Data Scientist - Evaluations, Chanakya

The role

The Data Scientist anchors the evaluations function by designing, building, and maintaining evaluation frameworks. You will measure model and system quality in operational contexts for high-stakes use cases.

What you will do

• Design and build evaluation frameworks for AI outputs across domain-specific requirements
• Define quality metrics in collaboration with domain experts and clients
• Run structured evaluation cycles pre- and post-deployment
• Build dashboards that surface model quality in production
• Identify failure modes, edge cases, and distribution shifts
• Collaborate with the MLOps Engineer to operationalise eval pipelines
• Build and manage domain-specific datasets for fine-tuning, evaluation, and benchmarking
• Publish internal findings and quality reports

Required qualifications

• 3–6 years in data science, ML research, or applied AI
• At least 2 years working with LLMs in production contexts
• Strong statistics and probability fundamentals
• Experience designing evaluation frameworks from scratch
• Python proficiency
• Comfort with pandas, NumPy, HuggingFace datasets, RAGAS, EleutherAI Eval Harness, LangSmith, or equivalent
• Experience with prompt engineering, model fine-tuning, or RLHF in applied settings
• Ability to work with unstructured domain data: PDFs, doctrine documents, transcripts, and field reports

Nice to have

• Prior work in high-stakes domains (healthcare, legal, defence, finance)
• Red-teaming or adversarial evaluation experience

About the company

Sarvam is building the bedrock of Sovereign AI for India by developing a full-stack sovereign AI platform. The company works with leading enterprises and public institutions and is backed by Lightspeed, Peak XV, and Khosla Ventures.