Member of Technical Staff - ML Systems & Inference

The role

You will build inference systems that execute models end-to-end in production. You will work on batching, scheduling, KV cache management, and memory placement across different hardware.

What you will do

• Build inference systems that execute models end-to-end in production
• Determine how requests are batched and scheduled
• Determine how stages are placed and scaled
• Manage how KV cache and intermediate state move between accelerators
• Balance latency, throughput, and utilization across hardware
• Work across model serving, batching, scheduling, concurrency, KV cache management, and memory placement
• Help bring up models on novel hardware
• Support new model architectures and inference techniques
• Improve performance under real production workloads
• Partner with compiler, kernel, networking, and distributed systems engineers

Required qualifications

• Strong software engineering fundamentals
• Experience building or operating ML inference or model serving systems
• Comfort reasoning about performance, memory usage, and system behavior under load
• Bachelor's degree in a relevant field, or an equivalent combination of education, training, and professional experience

Nice to have

• Experience with inference runtimes such as TensorRT-LLM, vLLM, or custom serving systems
• Deep understanding of modern model architectures and attention mechanisms
• Experience with batching, scheduling, and concurrency control in inference systems
• Familiarity with KV cache management and memory placement strategies
• Experience profiling and tuning latency- and throughput-critical systems
• Software development experience in Python and C++

About the company

Gimlet is building the first multi-silicon neocloud designed for fast, efficient AI inference. They combine large-scale compute infrastructure with an execution platform that partitions AI workloads and maps each stage to the hardware best suited to run it.