Software Engineer, Inference Runtime

The role

You will push forward the inference stack on-device and in the cloud. You will integrate new inference engines, bring up new models, and optimize execution for CPU and GPU targets.

What you will do

• Maintain and push forward the inference stack on-device and in the cloud
• Bring up new model architectures and multimodal models
• Improve latency, throughput, memory use, and reliability across CPU, CUDA, Metal, Vulkan, and ROCm runtimes
• Build runtime capabilities for model loading, batching, scheduling, caching, and distributed execution
• Benchmark and diagnose correctness and performance problems across the inference stack
• Contribute upstream to open-source projects such as llama.cpp and MLX

Required qualifications

• Significant experience building production ML systems, inference runtimes, or performance-sensitive infrastructure
• Strong programming ability in Python and C++
• Deep understanding of transformer architectures and the mechanics of model inference
• Experience profiling CPU or GPU workloads and reasoning about compute, memory, synchronization, and data movement
• Experience with PyTorch and inference systems such as llama.cpp, MLX, ExecuTorch, vLLM, SGLang, or TensorRT-LLM
• Strong debugging instincts across model code, runtime internals, operating systems, and CPU or GPU execution
• Takes personal responsibility for the correctness and performance of their work

Nice to have

• Past contributions to open-source inference runtime projects such as llama.cpp, MLX, ExecuTorch, vLLM, SGLang, or TensorRT-LLM

Benefits

• Competitive salary and equity grants
• Great medical, vision, dental healthcare plans
• Catered team lunch / expensed dinners in the office
• Flexible PTO
• Flexible WFH
• Sun-drenched office in SoHo in NYC

About the company

LM Studio is used by millions of people around the world to run AI on their own computers, and now with Bionic - also in the cloud.