QA Analyst - LOIS for Word

The role

You will own quality for LOIS for Word, Filevine's AI-powered legal drafting product. You will define the test strategy, process and automation, and evaluate both functional bugs and AI output quality.

What you will do

• Define and own the test strategy, process, and QA playbook for the product
• Move QA upstream by partnering with product and engineering early
• Evaluate AI output quality across accuracy, citation reliability, and appropriately scoped reasoning
• Design and maintain evaluation sets that catch regressions over time
• Test how AI-generated edits apply to documents, including formatting fidelity
• Build automated guardrails integrated into CI/CD for critical flows
• Write structured, reproducible bug reports
• Partner with legal domain experts to translate customer use into test coverage
• Approach QA with a product and design mindset
• Compare behavior across product surfaces and document discrepancies

Required qualifications

• 2+ years of QA experience, including ownership of test strategy for a product or major feature area
• Track record of building QA process and automation from scratch in a fast-moving environment
• Comfort operating in high ambiguity where features ship quickly
• Strong analytical and problem-solving skills, with attention to functional correctness and AI output quality
• Ability to reason about non-deterministic systems and define what correct means
• Excellent written communication for clear, actionable bug reports
• Experience with bug tracking tools and test management workflows

Nice to have

• Experience testing AI or LLM-based products, especially output accuracy and eval design
• CI/CD experience setting up pipelines and release gates for automated tests
• A design or UX sensibility
• Prior experience in legal technology or comfort reading legal contracts
• Programming experience beyond test automation

Benefits

• Medical, Dental, & Vision Insurance (for full-time employees)
• Maternity & paternity leave (for full-time employees)
• Short & long-term disability
• Competitive & Fair Pay
• Opportunity to learn from a dedicated leadership team
• Top-of-the-line company swag

Before you apply

• Communication about this role will only come from representatives with email addresses using filevine.com; other addresses are not affiliated with Filevine

About the company

Filevine is a Legal AI company delivering Legal Operating Intelligence. Its LOIS system connects context across legal matters, and it has earned AI awards and recognition from Deloitte and Inc.