HPC/ML Infrastructure Engineer

The role

An experienced HPC infrastructure engineer will lead bringup, administration, and operations for a large anime AI training cluster. You will bridge the gap between researchers and GPU machines to ensure SLURM jobs, parallel filesystems, and networks function for training anime models.

What you will do

• Lead bringup, administration, and operations on an anime AI training cluster
• Serve as the bridge between researchers and bare GPU machines
• Ensure SLURM jobs are running
• Ensure parallel filesystems are serving
• Ensure network is transmitting
• Ensure anime models are training

Required qualifications

• Experience with SLURM deploys through Slinky on K8s
• Experience with provisioning through warewulf/MAAS/ansible
• Experience with filesystems through WEKA/VAST/Ceph
• Experience with VPN and access through tailscale
• Experience with monitoring via the Grafana/Prometheus stack
• Linux sysadmin skills including wrangling ldap, triaging dmesg, and setting sticky bits on directories
• Comfortable working with physical computers

Nice to have

• Bay area is strongly preferred

Before you apply

• Visa sponsorships are available

About the company

Spellbrush has a tiny research team working to train anime image models.