Senior Inference Optimization ML Engineer

The role

You will own cost-efficient inference end-to-end by making foundation models faster and cheaper to run. You will port and optimize models for various hardware including AMD, TPU, wafer-scale, and consumer GPUs.

What you will do

• Make foundation models faster and cheaper to run in terms of latency, throughput, and cost per inference.
• Port and optimize models for non-NVIDIA and lower-cost hardware.
• Deploy and optimize models on Jetson/Thor-class embedded compute.
• Evaluate hardware by building cost/performance models for each chip and model family.
• Build hardware-portable optimizations using Triton, Pallas, or MLIR-based compilers.
• Apply model compression including quantization, pruning, and distillation.
• Ensure results are correct and consistent across backends.
• Build benchmarking and regression infrastructure across hardware.
• Work with research engineers to design efficient and portable models.

Required qualifications

• 2+ years of experience in inference optimization, ML systems, ML compilers, or a closely related field.
• Deep hands-on experience with PyTorch.
• Strong understanding of compute, memory bandwidth, memory capacity, and interconnect bottlenecks.
• Experience with model optimization techniques: quantization (INT8/FP8/FP4, AWQ), distillation, pruning, and compilation (torch.compile, XLA, TensorRT, IREE/MLIR, TVM, or vendor compilers).
• Familiarity with inference serving frameworks and their support for different hardware (e.g. vLLM on ROCm/TPU, SGLang, JetStream, Triton Inference Server).
• Excellent debugging and measurement skills.
• A high-ownership, pragmatic mindset.

Nice to have

• Experience optimizing inference for diffusion models.
• Production or deep project experience with at least one accelerator stack other than NVIDIA datacenter GPUs: AMD ROCm, Google TPU, wafer-scale and dataflow accelerators, Groq, Tenstorrent, AWS Trainiu
• Kernel authoring on more than one backend: HIP or Composable Kernel on AMD, Pallas on TPU, Triton on both NVIDIA and AMD, CUTLASS/CuTe.
• Contributions to open-source ML compilers or hardware backends (PyTorch, OpenXLA, Triton, vLLM, IREE, TVM).
• Experience running inference on wafer-scale systems, TPU pods, or fleets of consumer GPUs.
• Hands-on experience with Jetson/Thor-class platforms (TensorRT on Jetson, JetPack, DLA, power and thermal profiling).
• Experience with multimodal or video model inference (variable-length sequences, packing/bucketing).
• Speculative decoding, continuous batching, disaggregated prefill/decode, or other LLM serving optimizations.
• Experience with streaming or low-latency systems for real-time robot control.
• Experience working with hardware vendors on driver or compiler bugs and early-access silicon.

About the company

Rhoda AI is building the next generation of generalist intelligent robots. They own the full robotics stack from hardware to foundation world models.