Staff Data Scientist, Agent Improvement Loop

The role

You will own how Yalo evaluates its AI agents, including evals, trace and error analysis, and the improvement loop that turns production findings into fixes. You will work with Product, Engineering, Delivery and Customer Success to show what moved the metric.

What you will do

• Own the evals strategy for agents: pre-release behavioral testing, production monitoring, and the judges that score conversations
• Run continuous error analysis on agent traces at scale and rank improvement opportunities by impact
• Turn findings into fixes and new evals, ship them, and measure results with before-and-after and causal methods
• Improve agent skills, harness and CLI using trace evidence, through owning teams' repos and review process
• Partner with delivery and customer teams to take customer flows from underperforming to reliably healthy
• Calibrate and validate LLM judges against human review and set the bar for evidence at Yalo
• Write production-quality Python and TypeScript for evals and analysis tooling, with tests and CI
• Mentor data scientists and engineers on measurement, experimentation and evaluation practice
• Help shape quarterly and annual planning for agent quality

Required qualifications

• BS or MS in a technical field (Computer Science, Statistics, Applied Math, Operations Research, Economics, etc.)
• 8+ years delivering data science solutions with measurable business impact in cross-functional teams
• Hands-on experience evaluating LLM-based systems or agents: eval design, LLM-as-judge, rubric design, and judge calibration
• Expert in Python and SQL. Comfortable in TypeScript, or ready to be
• Strong production engineering habits: version control, code review, testing, CI/CD, and shipping tools others use
• Experience with GenAI systems in production: prompt and tool design, agent frameworks (e.g., LangGraph), and LLM observability tooling (e.g., LangSmith)
• Skilled at turning messy trace and conversation data into clear findings for technical and non-technical audiences
• Strong product and business sense. You frame quality work in terms of customer outcomes

Nice to have

• Experience with conversational AI, customer-facing agents or tool-using agents
• NLP and clustering for failure-pattern discovery
• Causal inference and experimentation skills: A/B tests, difference-in-differences, pre/post analysis
• Exposure to MLOps or LLMOps platforms (Vertex, SageMaker, MLflow, or similar) and big data tools
• Familiarity with AI safety and responsible AI practices
• Comfort in an ambiguous, fast-changing environment
• Experience mentoring other data scientists

Benefits

• Unlimited PTO policy
• Competitive rewards on the market range
• Remote working is available (-+3 hours CT)
• Flexible time (driven by results)
• Start-up environment
• International teamwork
• You and nothing else limit your career here

About the company

Yalo is on a mission to bring conversational commerce to the world, with roots in Emerging Markets such as Latin America. The company focuses on helping businesses, including small shop owners, serve their customers through conversations.