AI Researcher - Pre-training (f/m/d)

The role

You will advance the architecture and training of next-generation foundation models. You will design and run experiments on architecture, optimization, stability, and scaling using large-scale clusters.

What you will do

• Design and run ablations on core elements of the training recipe
• Implement PyTorch-based architectural changes
• Evaluate effects on convergence, stability, and training efficiency
• Run and analyze small-scale proxy experiments
• Contribute to hyperparameter scaling laws and scale-up methods
• Monitor large-scale training runs and investigate convergence issues
• Work with the team to track down distributed training failures
• Follow literature and reproduce promising results at small scale
• Work with Compute Performance, Data, Evaluation, and Post-Training teams

Required qualifications

• Proficient in Python
• Hands-on experience with PyTorch
• Strong foundation in machine learning shown through research, open-source contributions, or substantial professional or personal projects
• Solid mathematical background in linear algebra, probability, and optimization
• Understand how transformers work
• Have pre-trained LLMs from scratch
• Design experiments carefully
• Write clean, maintainable code
• Communicate clearly and act on feedback

Nice to have

• Experience training models on multi-GPU or multi-node setups
• Familiarity with distributed frameworks like Megatron-LM, DeepSpeed, or torchtitan
• Familiarity with scaling laws, hyperparameter transfer, or recent work on optimizers and training stability
• Exposure to Mixture-of-Experts, efficient attention variants, or other inference-efficient designs
• Publications at top-tier venues (NeurIPS, ICML, ICLR)
• Contributions to open-source ML training libraries
• Curiosity about hardware and systems constraints

About the company

Aleph Alpha is a European company doing foundation model pre-training. They build models for customers in finance, manufacturing, and public administration.