Member of Technical Staff, Evals Platform

The role

You will build infrastructure and workflows connecting evaluation and post-training. You will design scalable systems, pioneer new benchmarks, and improve user-facing fine-tuning workflows.

What you will do

• Scale Eval Infrastructure by designing systems to eliminate bottlenecks and build programmatic access.
• Track state-of-the-art benchmarks and reproduce published results.
• Design and build new benchmarks for complex or domain-specific use cases.
• Build and improve user-facing workflows for post-training and fine-tuning.
• Partner with customers and internal stakeholders to convert bespoke workflows into productized solutions.

Required qualifications

• 1–7 years of software engineering or data science experience
• Strong System Design Skills
• Hands-on experience building or working with sandbox environments and sandbox infrastructure
• Deep understanding of LLM evaluations and using results to guide model improvement
• Understanding of the GenAI Lifecycle

Nice to have

• 3+ years of software engineering or applied data science experience
• Experience working with the Harbor framework or similar container registry and orchestration tools
• Interest in discovering model performance and writing up/publishing insights publicly
• Interest in the hardware side of AI, GPU constraints, and inference optimization techniques
• Experience in fast-paced environments where you own features end-to-end

About the company

Fireworks is a platform for specialized intelligence that enables companies to build, train, and serve AI models. It is a Series D company valued at $17.5 billion.