Member of Technical Staff - Platform

The role

You will own the platform that runs benchmarks for evaluating LLMs at scale. This involves working across the stack on Python libraries, web platforms, distributed systems, cloud infrastructure, and tooling.

What you will do

• Build distributed systems to run evaluations across multiple models, benchmarks, and machines at scale
• Deploy cloud infrastructure using IAC, including deployment pipelines, servers, logging, monitoring, etc.
• Contribute to the internal and external libraries maintained
• Develop full-stack features for the platform using React/TypeScript and Python/Django
• Perform code and architecture reviews for other members of the team
• Help establish engineering best practices across the organization
• Collaborate closely with the research team to ensure infrastructure meets their needs

Required qualifications

• Strong engineering fundamentals
• Track record of building things of significant scope
• Significant experience in Python, especially in a professional setting
• Familiarity with common concepts like VMs, containerization, load balancers, databases, etc.
• Experience working with LLM APIs
• Understanding of concepts like temperature, tokenization, reasoning, etc.
• Experience working with development sprints, Git workflows, and pull request reviews
• Strong ability to provide and receive feedback effectively
• Ability to break down fuzzy problems into clear and actionable steps
• Tenacity to develop and iterate quickly

Nice to have

• Experience with frontend development, ideally React
• Experience with Django, FastAPI, or other Python-based HTTP servers
• Experience working with AWS infrastructure, including IaC
• Experience at early-stage startups or your own company
• Interest in AI/ML systems and evaluation

Benefits

• Highly competitive salary and meaningful ownership
• Relocation and transportation support
• Health/dental insurance coverage
• Lunch and dinner provided
• Free snacks/coffee/drinks
• 401K plan
• Unlimited PTO
• $1,500 housing stipend (within one-mile radius)

About the company

Vals is a team building the standard for evaluating the ability of LLMs to perform real-world tasks. The core methodology comes from NLP evaluation research conducted at Stanford.