Senior Staff Inference Performance Engineer

The role

You will own end-to-end performance of workloads deployed on d-Matrix's inference services. You will diagnose performance problems across the full serving path and protect throughput, latency, and stability targets.

What you will do

• Diagnose performance problems across the full serving path.
• Isolate problems to a specific layer and owner.
• Write minimal, reproducible problem statements for cross-stack findings.
• Protect throughput, latency, and stability targets for orchestrated workloads.
• Assess scaling from single deployments to fleet operation.
• Capture workload node requirements in the orchestration layer configuration.
• Build lightweight telemetry and benchmarks.
• Turn recurring investigations into playbooks and mentor engineers.

Required qualifications

• 8+ years of hands-on performance or systems engineering.
• Track record of finding performance overhead through profiling in unfamiliar, sparsely documented systems.
• Deep systems fundamentals, including Linux internals, NUMA, CPU pinning and cpusets, PCIe, memory and cache behavior, and network paths.
• Experience running performance-sensitive workloads on Kubernetes.
• Working knowledge of LLM inference serving, including batching, KV-cache, prefill/decode, quantization, and speculative decoding.
• Proficiency in Python and C/C++.
• Clear technical writing.
• Sound judgment about ownership.
• Fluency with AI and agentic tools for investigation, code search, log triage, and drafting instrumentation.

Nice to have

• Experience with LLM serving frameworks such as vLLM, SGLang, or TensorRT-LLM.
• Hands-on experience building or tuning speculative decoding, disaggregated serving, or heterogeneous serving architectures.
• Experience tuning KV-cache management, request and batch scheduling, or weight prefetch and residency in an LLM serving system.
• GPU programming and profiling experience (such as CUDA, Nsight, NCCL).
• Experience with pre-production or bring-up hardware, or in organizations where root causes routinely cross team boundaries.
• Rust experience.
• Startup experience or comfort in environments where priorities shift quickly.

About the company

d-Matrix has pioneered a memory-centric architecture that brings compute directly into memory. The team builds chips, software, and rack-scale systems to tackle AI infrastructure challenges.