Senior Staff LLM Inference Engineer

The role

You will build and optimize LLM inference systems across the full stack, from kernel-level optimization to distributed orchestration. You will prototype emerging use cases and build proof-of-concept systems using D-Matrix silicon, CPUs, GPUs, and custom accelerators.

What you will do

• Identify and prototype emerging LLM inference use cases suited to heterogeneous hardware deployments.
• Build compelling proof-of-concept systems that demonstrate D-Matrix capabilities.
• Develop and tune custom kernels and operator-level optimizations.
• Drive quantization, sparsity, and batching strategies tailored to the D-Matrix computational model.
• Build and maintain inference runtimes, serving frameworks, and evaluation tooling.
• Contribute to distributed inference systems including tensor/pipeline parallelism and KV-cache management.
• Work closely with hardware architects to provide actionable inference workload insights.
• Partner with product and business development to translate POCs into customer-facing demonstrations.
• Contribute to technical publications, whitepapers, and open-source projects.

Required qualifications

• Bachelor’s degree in Computer Science, Electrical Engineering, or a related field, and 10+ years of relevant engineering experience or equivalent demonstrated experience.
• Strong proficiency in Python and C/C++.
• Hands-on experience optimizing LLM inference — attention kernels, KV cache, batching strategies, quantization (INT8/FP8/INT4).
• Experience with at least one major inference framework (vLLM, SGLang, TensorRT-LLM, ONNX Runtime, or similar) at a contributor level.
• Familiarity with GPU kernel programming (CUDA/Triton) and performance profiling tools.

Nice to have

• Master’s or PhD in Computer Science, Electrical Engineering, or a related field, with 6+ years of relevant industry experience.
• Experience with heterogeneous compute deployments — scheduling inference workloads across dissimilar hardware.
• Familiarity with custom silicon or ASIC-based inference.
• Experience with distributed inference: tensor parallelism, pipeline parallelism, disaggregated serving.
• Contributions to open-source inference or ML systems projects.
• Experience with production inference serving at scale.
• Familiarity with speculative decoding, mixture-of-experts routing, or long-context-serving techniques.
• Working familiarity with the material in the JAX Scaling Book or equivalent systems-level understanding of modern LLM training and inference.

Benefits

• Competitive compensation
• Equity

About the company

d-Matrix has pioneered a memory-centric architecture that brings compute directly into memory. The team works on chips, software, and rack-scale systems to tackle AI infrastructure challenges.