Staff Engineer, Crusoe Managed Data Transfer (Distributed Systems)

The role

You will be the technical lead and systems architect for the Managed Data Transfer service. You will design the control plane, transfer worker fleet, and integration with storage and network to move petabyte-scale training data.

What you will do

• Own the end-to-end design of MDT including job API, scheduler, transfer workers, storage integration and network paths.
• Evaluate the production prototype and migrate current users onto the managed service.
• Set the data-plane strategy for throughput and correctness.
• Own the interfaces between MDT and Storage, Network, Kubernetes, IAM, Console, Billing and Monitoring.
• Design for new clouds and Crusoe-internal transfers with a pluggable transfer engine.
• Set the bar for reliability, observability and on-call.
• Work with Product on scope, pricing trade-offs and milestones.
• Mentor engineers through design reviews, RFC feedback and direct coaching.

Required qualifications

• Hands-on experience designing and operating data-intensive distributed systems at scale.
• 8+ years shipping production software, including leading a service from prototype to general availability.
• Understanding of object storage semantics, file systems and TCP throughput.
• Ability to profile and fix a transfer that is slower than it should be.
• Proven ability to evolve a live system toward a better design while keeping risk to current customers low.
• Experience defining API boundaries and schema contracts with partner teams.
• Experience carrying customer-facing on-call and building maintainable, observable and predictable systems.
• Proficiency in Go or another modern compiled language.

Nice to have

• Experience building or operating bulk data transfer systems, such as AWS DataSync, Google Storage Transfer Service, or rclone at petabyte scale.
• Familiarity with cloud networking and interconnects, such as AWS Direct Connect or GCP Interconnect.
• Experience with cross-account cloud IAM and threat modeling for services that hold customer credentials.
• Approximate nearest-neighbor (ANN) indexing in depth: HNSW, IVF, product and scalar quantization, DiskANN-style indexes that live on SSD.
• Experience with AI, HPC or Physical AI data pipelines.
• Go strongly preferred

Benefits

• Competitive compensation and equity packages
• Restricted Stock Units
• Paid time off, paid holidays & leave of absence programs
• Comprehensive health, dental & vision insurance
• Employer contributions to HSA account
• Paid parental leave
• Paid life insurance, short-term and long-term disability
• Professional development & tuition reimbursement
• Mental health & wellness support
• Commuter benefits (parking & transit)
• Cell phone stipend
• 401(k) Retirement plan with company match up to 4% of salary

About the company

Crusoe is a vertically integrated AI infrastructure company that owns and operates each layer of the stack to power AI workloads. They use an energy-first approach to make AI infrastructure better for the world and faster for innovators.