AI Infrastructure Engineer

The role

You will own the inference-serving stack end to end. You will deploy and operate large language and multimodal models and build distributed inference systems.

What you will do

• Run large language and multimodal models using inference engines like vLLM, SGLang, and TensorRT-LLM.
• Build and maintain distributed inference systems with NVIDIA Dynamo, Kubernetes, Helm, and Ansible.
• Tune parallelism strategies, batching, KV caching, routing, and scheduling.
• Build reproducible benchmarks for latency, throughput, concurrency scaling, and system efficiency.
• Enable tool calling, agentic and coding workflows, RAG, and long-context applications.
• Evaluate new models, frameworks, and optimizations.
• Partner with hardware, model, and systems engineers to ship improvements.

Required qualifications

• BS, MS, or PhD in Computer Science, Computer Engineering, or a related field, or equivalent practical experience.
• 2+ years of engineering experience in LLM inference, ML systems, GPU systems, or related infrastructure.
• Hands-on experience deploying and optimizing at least one LLM inference engine (e.g., vLLM, SGLang, or TensorRT-LLM).
• Solid understanding of LLM inference fundamentals: prefill/decode, batching, KV caching, and latency/throughput tradeoffs.
• Strong Python and Linux skills.
• Hands-on experience with Kubernetes and containerized deployments.
• Comfortable benchmarking and troubleshooting systems.
• Communicating clearly in a small engineering team.

Nice to have

• Experience with Helm, Ansible, or similar deployment and configuration tools.
• Experience with distributed inference systems like NVIDIA Dynamo, including parallelism strategies (TP/EP/DP/PP), prefill/decode disaggregation, and KV-cache-aware routing.
• Experience with advanced inference optimizations such as prefix caching, speculative decoding, and quantization.
• Experience analyzing LLM architectures and tuning deployments across different hardware platforms to meet SLAs.
• Contributions to vLLM, SGLang, TensorRT-LLM, or related open-source projects.
• Experience owning production infrastructure, ideally at an early-stage startup.

Benefits

• Competitive salary with performance-based bonus and early-stage equity grant
• 100% employer-paid Health, Dental, and Vision coverage for you and your dependents
• 401(k) match with immediate vesting
• Access to financial advisors
• 100% employer-paid Life, Disability, and AD&D insurance
• Fitness stipend
• Wellness & mental health perks
• 20 vacation days
• 15 company holidays (including 3 floating days of your choosing)
• Daily lunch stipend
• Enterprise-level Claude & ChatGPT access with a generous token budget
• On-site parking and EV charging

Before you apply

• Visa sponsorship and relocation assistance available

About the company

Netpreme is building a solution that allows AI memory to scale independently of AI compute. The company is based in Cambridge, MA, and Santa Clara, CA.