Senior Software Development Engineer in Test (SDET), QA - AI Systems

The role

This person will own quality for AI agent systems by combining deep test automation with manual and exploratory testing. They will build evaluation harnesses, simulation environments, and CI/CD quality gates for agents that act autonomously in industrial environments.

What you will do

• Design, build, and maintain test frameworks, evaluation harnesses, and regression suites for LLM-based agents.
• Build automated pipelines that run evals and test suites on every model, prompt, or tool-integration change.
• Develop simulators, mocks, and sandbox environments for industrial systems and tool APIs.
• Implement API, contract, integration, and end-to-end tests.
• Develop performance, load, and reliability tests for agent workflows.
• Perform hands-on exploratory testing of agent behavior.
• Manually review agent traces and outputs to judge reasoning quality, appropriateness of actions, and safety.
• Execute manual regression and release verification passes.
• Conduct manual red-teaming sessions in partnership with security teams.
• Support user acceptance testing and pilot rollouts with Forward Deployed Engineers.

Required qualifications

• 5+ years as an SDET, QA engineer, or software engineer with a strong testing focus.
• Proven experience in both manual and automated testing, including exploratory testing, test planning, and writing high-quality bug reports.
• Strong programming skills (Python preferred).
• Experience building test frameworks and infrastructure from the ground up.
• Hands-on experience with CI/CD systems (e.g., GitHub Actions, GitLab CI, Jenkins) and containerized test environments (Docker, Kubernetes).
• Experience testing APIs and microservices, including contract, integration, and end-to-end testing.
• Hands-on experience testing LLM-based products, chatbots, or AI agents.
• Practical experience with eval frameworks or tooling (e.g., promptfoo, DeepEval, RAGAS, LangSmith) or a track record of building your own.
• Understanding of how LLM agents work: prompting, tool/function calling, context management, RAG, memory, and orchestration frameworks.
• Experience designing test data and labeled datasets, including sourcing, sampling, and managing dataset drift over time.
• Familiarity with LLM-specific failure modes: hallucination, prompt injection, context poisoning, tool misuse, goal drift, and non-determinism.
• Strong written communication skills.

Nice to have

• Experience with human-in-the-loop evaluation workflows (labeling pipelines, inter-rater reliability, rubric design).
• Experience with performance and load testing tools (e.g., k6, Locust) and chaos or fault-injection testing.
• Background in ML/data science sufficient to read model evals and statistical significance.
• Experience red-teaming or doing adversarial/security testing on ML systems.
• Familiarity with observability and tracing tools for LLM applications (e.g., LangSmith, Arize, Langfuse, Weights & Biases) and general observability stacks (OpenTelemetry, Grafana, Datadog).
• Experience testing systems in industrial, IoT, or operational technology (OT) environments, including industrial protocols (OPC UA, MQTT, Modbus) or simulation and hardware-in-the-loop setups.
• Experience supporting UAT, pilots, or customer-facing validation of complex products.
• Experience setting up test and eval infrastructure from scratch at a startup or fast-moving team.

Benefits

• Innovative Environment
• Collaborative Culture
• Professional Growth
• Competitive Compensation: salary and equity package

About the company

Nexxa is building the best AI systems for heavy industries, enabling machines, systems and operations to think, decide and act autonomously across manufacturing, large-scale infrastructure, logistics and legacy environments.