QA Engineer

The role

You will design and execute an evaluation framework for LLM and agentic AI outputs. You will write automated test suites, set up CI/CD quality gates, and produce compliance documentation for SOC 2 Type II standards.

What you will do

• Design and execute the evaluation framework for LLM and agentic AI outputs.
• Write assertions and define behavioral contracts.
• Own regression baselines for model behavior.
• Write and maintain automated test suites for backend APIs, document ingestion pipelines, AI inference workflows, and frontend surfaces.
• Perform performance and load testing for latency-sensitive inference paths.
• Set up and enforce quality gates in CI/CD pipelines.
• Triage production bugs and write reproduction cases.
• Produce test artifacts, audit logs, and process documentation for SOC 2 Type II standards.
• Work with Forward Deployed Engineering on client-side validation and production issue reproduction.

Required qualifications

• Seven or more years in software QA engineering.
• At least two years personally testing AI or ML systems.
• Experience writing test cases against LLM outputs.
• Experience building evaluation pipelines from scratch.
• Fluent in Python.
• Built automated suites using pytest, Playwright, or Selenium.
• Hands-on experience with RAGAS, DeepEval, LangSmith, or comparable evaluation tooling.
• Ability to trace failures from application layer to infrastructure.
• Knowledge of Azure, async systems, and REST APIs.
• Experience integrating QA gates into CI/CD pipelines.

Nice to have

• Experience in fintech, banking, or another regulated environment.
• Familiarity with document processing pipelines.
• Familiarity with multi-agent architectures.
• Familiarity with RAG validation.
• Familiarity with observability tooling such as Arize or Langfuse.

About the company

Titan builds AI software for banks, including purpose-built small language models, a banking ontology, and AI bankers. The company is backed by leading fintech investors and operates under banking compliance, audit, and model-risk standards.