Senior Machine Learning Engineer, Inference - USA

The role

This person will optimize and serve models for high-volume, realtime workloads. They will take models from research, containerize them, and ensure reliable production serving.

What you will do

• Take a model from the research team, containerize it, and optimize its serving.
• Ensure models run reliably in production.
• Profile code to optimize performance on NVIDIA GPUs.
• Handle thousands of concurrent connections using distributed systems.
• Design benchmarks or prototypes to solve unclear problems.

Required qualifications

• PhD in CS, Physics, Math, or equivalent practical experience building backend or ML systems.
• Proficiency in C++, CUDA, Rust, or highly optimized Python.
• Experience with Kubernetes, Ray, custom load balancing, multi-GPU/multi-node inference, and handling concurrent connections.

Nice to have

• Inference Optimization: Deep understanding of modern serving frameworks and techniques like vLLM or TRT-LLM.
• Model Acceleration: Hands-on experience with quantization, distillation, caching strategies, continuous batching, paged attention, and speculative decoding.
• Public work: Non-trivial systems programming projects, open-source contributions to major inference engines, or deep-dive technical write-ups.

Benefits

• Relocation assistance
• Bonus
• Equity
• Benefits

About the company

Inworld is a research lab and inference provider focused on realtime AI for consumer-facing applications. They build first-party speech models, serve LLMs, and run inference for high-volume, realtime workloads.