Senior AI Engineer, Agents (Python) (f/m/d)

The role

A senior AI engineer will own agent quality for chat and voice agents. You will orchestrate and evaluate models, manage prompting and context, and build eval sets and metrics.

What you will do

• Map chat and voice agents end to end
• Partner with backend engineers
• Review real user conversations and identify failure modes
• Own agent quality measurement
• Build eval sets and metrics for key use cases
• Benchmark models on quality, latency and cost
• Establish a repeatable model for improving agent quality
• Run evals on every prompt, model and harness change
• Turn real conversations into new test cases automatically
• Set the longer term roadmap for agent quality

Required qualifications

• Evidence of ownership and shipping LLM agents to real users
• Ability to prove with data that agents got better
• Technical major at one of the major technical universities like TUM, ETH, KIT, RWTH, EPFL, etc.
• Strong Python
• Ability to debug agent behavior across prompts, tools, context and model choice
• Hands on experience building eval pipelines, LLM judges and model benchmarks
• Knowledge of statistics to tell a real gain from noise
• Ability to ground decisions in real user data
• Deep curiosity
• Ability to gather context quickly, form a clear point of view, and keep pace
• Low ego teammate who communicates clearly and owns outcomes
• Familiarity with AI coding tools

Nice to have

• Experience with the latest agent frameworks
• Experience with realtime voice agents
• Experience with multilingual agents

Benefits

• Strong base salary
• Meaningful equity ownership
• 30 days of PTO
• Hybrid setup
• Free meals and drinks
• Deutschlandticket
• Best in class equipment
• Wellpass

About the company

Bliro is an AI assistant for field sales that processes and documents customer interactions. It is trusted by lighthouses of the German Mittelstand and is backed by several capital firms.