Senior AI Researcher- Pre-training (f/m/d)

The role

A Senior AI Researcher will advance the architecture and training of next-generation foundation models. This role involves designing inference-efficient architectures, optimizing training recipes, and training models on a large scale cluster.

What you will do

• Own core elements of the training recipe including optimizers, schedules, and initialization
• Design PyTorch-based architectural improvements to maximize convergence, stability, and training efficiency
• Develop hyperparameter scaling laws and scale-up methodologies
• Use small-scale proxy experiments to predict multi-thousand-GPU behavior
• Investigate complex convergence issues like loss spikes and divergence
• Resolve distributed system failures like communication bottlenecks, race conditions, and synchronization errors
• Partner with Compute Performance, Data, Evaluation, and Post-Training teams to align the model lifecycle with hardware constraints

Required qualifications

• Proficient in Python and deeply familiar with PyTorch-based training workflows
• Strong track record in machine learning research and software engineering through shipped models, impactful open-source contributions, or published research
• Strong mathematical foundation comfortable reasoning formally about optimisation, scaling behaviour, and training dynamics
• Deep understanding of transformer training dynamics, optimisation, and the behaviour of large distributed training jobs
• Ability to design rigorous experiments and translate empirical observations into robust training decisions
• Hands-on experience pre-training large models (e.g., 7B+ parameters) on substantial infrastructure (e.g., 100+ GPU clusters)
• Apply strong software engineering practices including writing maintainable, well-tested code
• Ability to implement complex model architectures efficiently and reliably and to debug complex issues
• Collaborate effectively within a research and engineering team
• Able to work in Germany and collaborate regularly on site in Heidelberg

Nice to have

• Hands-on experience training LLMs or multimodal models on large GPU clusters using distributed frameworks (e.g., Megatron-LM, DeepSpeed, torchtitan)
• Familiarity with scaling laws, hyperparameter transfer, or methods for predicting large-scale training behavior from smaller proxy runs
• Experience profiling distributed jobs and diagnosing training anomalies like loss spikes, numerical instability, or optimizer pathologies
• Exposure to sparse training approaches (e.g., Mixture-of-Experts) and an understanding of their routing and systems trade-offs
• Demonstrated research excellence through top-tier publications (NeurIPS, ICML, ICLR), impactful open-source contributions, or significant shipped technical work
• Strong curiosity about the hardware and systems constraints that shape scale

Benefits

• 30 days of paid vacation
• Access to a variety of fitness & wellness offerings via Wellhub
• Mental health support through nilo.health
• Substantially subsidized company pension plan
• Subsidized Germany-wide transportation ticket
• Budget for additional technical equipment
• Flexible working hours
• Hybrid working model
• JobRad® Bike Lease

About the company

Aleph Alpha is a company in Europe doing foundation model pre-training. They build models that understand German and meet European regulatory requirements in Heidelberg.