AI Research Engineer - Datadog AI Research (DAIR)

The role

You will partner with Research Scientists to turn research ideas into working systems. You will build data, tooling, and infrastructure to enable rapid iteration and a path from prototype to production.

What you will do

• Build and operate multimodal data pipelines, training and evaluation infrastructure, benchmarks, and internal tooling
• Implement models, run experiments at scale, and profile for reliability, performance, and cost
• Build simulation environments and replay infrastructure for agent training and evaluation
• Orchestrate distributed training and distributed RL with Ray
• Establish automated benchmarks and regression tests for world model predictions, agent performance, and simulation fidelity
• Collaborate with Research Scientists, Product, and Engineering to integrate capabilities into products
• Contribute to research publications at top-tier conferences
• Produce high-quality code, documentation, and open-source artifacts

Required qualifications

• Depth in distributed computing, RL Infra, and ML systems for training and inference at scale
• Proficient in Python
• Familiar with a systems language (e.g., Rust, C++, or Go)
• Comfortable with modern cloud and data infrastructure
• Practical experience implementing and operating ML training and inference systems (e.g., PyTorch or JAX)
• Experience with containerization, orchestration, and GPU acceleration
• Practical experience with large-scale model training and fine-tuning
• Experience with frameworks like Megatron-LM, DeepSpeed, SkyRL, VeRL, or TorchTitan
• Experience with techniques such as SFT, RLVR, RLHF, and efficient inference (quantization, speculative decoding)
• Ability to explain design and performance trade-offs clearly to technical and non-technical audiences
• Experience supporting or contributing to research publications

Nice to have

• Experience with Ray, Slurm, or similar frameworks
• Strong software engineering skills in domains such as observability, SRE, or security
• Experience bridging research prototypes and real-world product applications
• Passion for pushing the boundaries of AI with a focus on customer impact and scalable deployment
• Hands-on experience with GPU programming and optimization, including CUDA
• Experience writing production data pipelines and applications
• Experience building simulation or sandbox environments for agent training

Benefits

• Competitive global benefits
• New hire stock equity (RSUs)
• Employee stock purchase plan (ESPP)
• Opportunity to collaborate with colleagues in New York City and Paris
• Opportunity to attend and present at conferences and meetups
• Intra-departmental mentor and buddy program
• Inclusive company culture
• Ability to join Community Guilds

About the company

Datadog is a global SaaS business delivering observability and security for the cloud age. It is a collaborative, people-first community that helps organizations solve complexity through digital transformation and infrastructure monitoring.