Metaforms
Senior AI Engineer
Bengaluru. Posted 4 months ago
The role
You will own the design, development, and continuous improvement of AI agent systems. You will work on agent harness architecture, evaluation, production monitoring, and reliability for high-stakes outputs.
What you will do
- Own the agent harness for production agents
- Lead research and implementation for long-context handling and cascading-error challenges
- Drive context engineering strategy and experimentation frameworks
- Define structured rubrics for evaluating AI outputs
- Build continuous monitoring, tracing, and failure-mode analysis for agents in production
- Create tooling for domain experts to refine skill files, eval sets, and knowledge bases
- Build eval suites including regression sets, golden datasets, and LLM-as-judge pipelines
- Develop evaluation datasets for DSLs, structured data transforms, and computed outputs
- Design human-in-the-loop and review workflows
Required qualifications
- Built and operated agentic systems in production
- 4+ years of engineering experience
- At least 1 year focused on LLM/agent systems in production
- Deep hands-on experience with frontier model APIs (Anthropic, OpenAI, Gemini), evaluation frameworks, and AI system optimization
- Strong Python skills
- Solid grasp of context engineering and evaluation methodology
- Strong instincts for debugging complex, non-deterministic system failures
- High ownership
Nice to have
- Go or TypeScript a plus
- Experience with LLM observability and eval tooling (Braintrust, Langfuse, LangSmith, Weave, promptfoo, or in-house equivalents)
- Background in semantic parsing, DSLs, or structured-output generation
- Prior work on computer-use or browser agents
- Experience with human-in-the-loop agent workflows where proposals are reviewed before apply, or agents over large structured datasets
Benefits
- Full family health insurance
- $1,000 USD annual learning and development budget
- Dedicated mentor and coaching support
- Free snacks and dinner at the office
About the company
Metaforms is a Series A funded company that provides an agent layer for market research. Their AI agents write production survey code, QA deployments, process datasets, and generate reports.