Software Engineer, ML Workload Platform

Canada, Remote. Posted 7 days ago

The role

You will build the platform that runs training, evaluation, and reinforcement-learning workloads. This includes job lifecycle, scheduling, workload networking, and storage. You will work with AI researchers and ML engineers on stability, scalability, and observability.

What you will do

Own the platform that schedules and runs training, evaluation, and reinforcement-learning jobs on Kubernetes
Build networking and runtime isolation including agent and sandbox environments
Work with the data infrastructure team on storage for checkpoints and other training artifacts
Troubleshoot production failures and performance problems
Build automation, infrastructure-as-code, and observability

Required qualifications

Run a job platform, batch system, or Kubernetes controllers in production
Shown distributed systems judgment regarding retries and partial failure
Worked on ML training workflows, RL sandboxes, container networking (CNI), or checkpoint storage at scale
Written software in Python or Go
Worked with researchers, ML engineers, or another team whose goals were not identical to yours
Found a bottleneck, proposed a fix, and driven it in a fast-moving environment

Nice to have

Direct GPU-cluster experience

Benefits

Weekly lunch stipend of $75/£75 or equivalent
Full health and dental benefits, including a separate budget for mental health
RRSP matching, 401K, Pension Scheme
100% Parental Leave top-up for up to 6 months
Annual enrichment benefits for arts & culture, fitness/wellness, quality time, and workspace improvement
Education & learning stipend for conferences, courses, and coaching
6 weeks of paid vacation
Budget for traveling to other offices if remote
Annual company offsite
Daily lunch program and snacks for those in office
Co-working benefit for those not near an office
$500 home office stipend

About the company

Cohere is a security-first enterprise AI company that builds foundation AI models and end-to-end products. It is a global technology company headquartered in Toronto.