Engineer II, QA

The role

You will develop testing strategies, evaluation frameworks, and quality metrics for GenAI systems. This role involves building automated test suites and monitoring systems for LLM-powered applications.

What you will do

• Design and implement testing strategies for GenAI features
• Develop automated test suites for prompt testing
• Create evaluation frameworks to measure GenAI quality
• Build and maintain test datasets and golden examples
• Implement monitoring and alerting systems for production GenAI features
• Perform adversarial testing for failures, hallucinations, biases, or security vulnerabilities
• Collaborate with engineers to define acceptance criteria and quality gates
• Develop tools and frameworks for testing GenAI implementations
• Conduct user acceptance testing and gather feedback
• Document testing procedures, known issues, and quality metrics

Required qualifications

• 3+ years of QA or test engineering experience
• Strong understanding of GenAI technologies including LLMs, prompt engineering, and AI application patterns
• Experience with test automation frameworks and scripting (Python, JavaScript)
• UI testing (Playwright, Puppeteer)
• Performance testing
• Scenario testing
• Experience working in Gen AI Systems+ product dev
• Knowledge of software testing methodologies (functional, integration, regression, performance, security testing)
• Ability to design test cases and evaluation criteria for non-deterministic systems
• Strong analytical and problem-solving skills with attention to detail
• Experience with API testing tools (Postman, REST Assured) and backend testing
• Familiarity with CI/CD pipelines and automated testing integration

Nice to have

• Experience testing conversational AI, chatbots, or agentic systems
• Knowledge of ML model evaluation metrics and techniques
• Familiarity with LLM evaluation frameworks (LangSmith, PromptFoo, Ragas)
• Experience with performance testing and load testing AI APIs
• Understanding of responsible AI principles including fairness, transparency, and safety testing
• Background in enterprise software or SaaS QA
• Experience with test management tools (TestRail, Zephyr, Jira)
• Knowledge of security testing methodologies for AI systems
• Scripting experience with Python, including working with LLM APIs

About the company

Anaplan is a team of innovators focused on optimizing business decision-making through an AI-infused scenario planning and analysis platform. Their customers include global companies like Coca-Cola, LinkedIn, Adobe, LVMH, and Bayer.