Software Engineer, Data Infrastructure

The role

You will design, build, and operate a distributed storage system that feeds model training and evaluation. This involves running systems on Kubernetes clusters at petabyte scale and working on networking, I/O, and consistency problems.

What you will do

• Design, build, and operate the distributed storage system that feeds model training and evaluation
• Run this system multiple on Kubernetes clusters at petabyte scale
• Work with researchers and training-infra teams on throughput, latency, and durability requirements
• Work through networking, I/O, and consistency problems of moving large datasets and checkpoints across regions and backends

Required qualifications

• Strong storage fundamentals, including replication, consistency, caching, and data lifecycle management
• Strong coding ability in Python or Go
• Experience running stateful systems on Kubernetes, including Persistent Volumes, CSI drivers, and StatefulSets
• Hands-on experience with cloud object storage such as S3 as well as POSIX-style filesystems

Nice to have

• Experience with parallel or HPC filesystems such as Weka, VAST, or Lustre
• Familiarity with the data-loading and checkpointing patterns used in large-scale model training
• An interest in how large models are trained and evaluated

Benefits

• Weekly lunch stipend of $75/£75 or equivalent
• Full health and dental benefits, including a separate budget for mental health
• RRSP matching, 401K, Pension Scheme
• 100% Parental Leave top-up for up to 6 months
• Annual enrichment benefits for arts & culture, fitness/wellness, quality time, and workspace improvement
• Education & learning stipend for conferences, courses, and coaching
• 6 weeks of paid vacation
• Budget for traveling to other offices if you are remote
• Annual company offsite
• Daily lunch program, snacks, and social events for those in the office
• Co-working benefit for those not near an office
• $500 home office stipend

About the company

Cohere is a security-first enterprise AI company that builds foundation AI models and end-to-end products. It is a global technology company headquartered in Toronto.