Senior AI QA Engineer

The role

You will build frameworks to ensure clinical insights are accurate, safe, and reliable. This role involves high-level software testing and data engineering to validate complex medical outputs.

What you will do

• Architect multi-layered validation frameworks using deterministic checks, semantic similarity metrics, and model-based evaluations.
• Develop systems to evaluate clinical pipeline outputs for faithfulness, safety, and hallucination detection.
• Leverage agentic AI tools to prototype test harnesses, red-team clinical logic, and build validation utilities.
• Execute integration and regression tests for data-heavy backend processes.
• Work with Data Scientists and Product Managers to define Ground Truth datasets and clinical evaluation rubrics.
• Perform root cause analysis on system failures related to code logic, data drift, or model behavior.

Required qualifications

• 6+ years of technical experience in Quality Assurance with a focus on system architecture and backend data validation.
• Expert-level skills in Python for building custom test scripts and working within AI/ML ecosystems.
• Proven experience testing model outputs using diverse metrics like Semantic Similarity, NLP metrics, and custom scoring algorithms.
• Familiarity with tools like LangSmith, Arize Phoenix, or MLflow.
• Proficiency with AI-native development tools such as Cursor, Windsurf, Claude Code.
• Proficiency with frameworks like PyTest.
• Experience with traditional tools like Cypress, Playwright, or Selenium for end-to-end flows.
• Solid understanding of API testing (Postman/RestAssured).
• Database validation (SQL/NoSQL).
• Experience serving as a Subject Matter Expert (SME) on test process development for large-scale systems.

Nice to have

• Clinical Domain Knowledge: Familiarity with healthcare data standards such as FHIR, HL7, or SNOMED CT.
• Healthcare Compliance: Understanding of HIPAA regulations and the nuances of handling ePHI/PHI.
• Statistical Foundation: Understanding of sensitivity, specificity, and F1-scores in a clinical context.

About the company

GW RhythmX is revolutionizing healthcare through connected, AI-native intelligence that unites clinical insight, patient engagement, and system-wide care orchestration. A SymphonyAI Group company, it leverages assets including $1B+ in R&D investment and longitudinal data related to 300 million patients.