Member of Technical Staff (Quality Engineering)

The role

This person will own quality for the AI that powers the Solstice platform. They will build evaluation systems, test non-deterministic ML systems, and maintain end-to-end testing suites.

What you will do

• Build evaluation systems to score quality.
• Make model and prompt changes safe by flagging quality, cost, or latency regressions.
• Test agents for failures like goal drift, tool loops, or malicious instructions.
• Protect compliance-critical paths by verifying claims against source material.
• Own end-to-end testing across the app using a Playwright suite.
• Run hands-on manual and exploratory QA.
• Get CI/CD quality gates in place.
• Use production signals for drift detection and new regression tests.
• Harden background jobs to survive retries, timeouts, and crashes.
• Set the testing bar and define what good testing looks like.

Required qualifications

• Strong Python
• Real experience building test infrastructure and running it automatically in CI/CD
• Strong end-to-end and UI test automation, especially with Playwright
• A genuine manual and exploratory QA discipline
• Experience testing non-deterministic, ML, or LLM-based systems, or the appetite to build that capability from scratch
• Comfort with evaluation methods: golden datasets, LLM-as-judge (rubric, pairwise, reference-based), and calibrating those judges against human or expert labels
• A statistical way of thinking about quality (variance, pass@k, regression detection)
• An instinct for error analysis
• Independent thinking and a strong sense of ownership
• Clear communication
• A serious work ethic

Nice to have

• Hands-on experience with agent or LLM frameworks (LangChain, LangGraph)
• Experience with eval and LLM-observability tools (LangSmith, Langfuse, Arize Phoenix, Braintrust, RAGAS, Promptfoo, OpenAI Evals, or similar)
• Comfort in a modern frontend stack (TypeScript and React)
• Adversarial or red-team testing
• Backend experience with async Python services and task queues
• Experience in a regulated industry such as pharma, healthcare, or finance
• MLOps or LLMOps experience, including defining quality SLOs

Benefits

• Health, dental, and vision insurance
• Ground-floor equity opportunity
• Visa sponsorship (O-1, H-1B, TN) available
• Unlimited PTO
• 401(k) + 4% company match
• In-office lunches 5 days/week
• Professional growth stipend
• Relocation support
• Work in a high-velocity, high-impact environment

About the company

Solstice is redefining how life sciences organizations commercialize their therapeutics by building a commercial engine for pharmaceutical marketers. They are building applications at the frontiers of AI, experimenting with LLMs and new frameworks.