Senior AI R&D Engineer - Performance - Pre-training (f/m/d)

The role

You will engineer systems to train foundation models at scale. You will maximize hardware utilization and training throughput on large-scale GPU clusters.

What you will do

• Profile training loops using PyTorch Profiler, Nsight Systems and Nsight Compute to identify system- and kernel-level bottlenecks.
• Configure and tune composite parallelism strategies to optimize load balance and minimize bottlenecks.
• Partner with AI Researchers to define model architectures for hardware efficiency.

Required qualifications

• Proficient in Python and the PyTorch library.
• Strong engineering background in parallel and/or distributed systems.
• Hands-on experience with modern machine learning techniques especially large language models and their life cycle.
• Deep understanding of the CUDA programming model.
• Experience in distributed programming with APIs like NCCL or MPI.
• Experience analysing profiling traces with tools such as PyTorch Profiler and Nvidia Nsight.
• Regular on-site collaboration in Heidelberg.

Nice to have

• Contributions to modern distributed training frameworks (e.g., TorchTitan, Megatron-LM, DeepSpeed).
• Familiarity with low-precision training formats (MXFP4, MXFP8) and their impact on numerical stability and throughput.
• A deep understanding of NCCL communication primitives, NVSHMEM or CUDA IPC and their performance.
• A proven track record of implementing and optimising modern transformer-based model training.
• A proven track record working on the NVIDIA Blackwell architecture.

Benefits

• 30 days of paid vacation
• Access to a variety of fitness & wellness offerings via Wellhub
• Mental health support through nilo.health
• Substantially subsidized company pension plan
• Subsidized Germany-wide transportation ticket
• Budget for additional technical equipment
• Flexible working hours
• Hybrid working model
• JobRad® Bike Lease

About the company

Aleph Alpha is one of the few companies in Europe doing serious foundation model pre-training. They are building models in Heidelberg that understand German and meet European regulatory requirements.