Plaud

Machine Learning Engineer, Inference & Serving (Speech LLM) - San Francisco

San Francisco, CA, Remote. Posted 6 months ago

The role

This person will build and deploy high-throughput, ultra-low-latency inference engines for large language models or foundational speech models. They will work at the intersection of the core ML training team and the backend infrastructure team to optimize GPU performance.

What you will do

  • Build and deploy high-throughput, ultra-low-latency inference engines for large language models or foundational speech models
  • Manage tradeoffs between latency, throughput, and Time-To-First-Token or Time-To-First-Audio in real-time streaming environments
  • Implement continuous batching and KV cache management
  • Manage stateful connections for real-time conversational AI
  • Identify and eliminate hardware bottlenecks using knowledge of GPU architectures and memory hierarchy
  • Collaborate between the core ML training team and the backend infrastructure team
  • Optimize performance out of a cluster of GPUs

Required qualifications

  • Hands-on experience building and deploying high-throughput, ultra-low-latency inference engines for large language models or foundational speech models
  • Understand the tradeoffs between latency, throughput, and Time-To-First-Token (or Time-To-First-Audio) in real-time streaming environments
  • Practical experience with continuous batching, KV cache management (e.g., PagedAttention), and stateful connections necessary for real-time conversational AI
  • Deep understanding of GPU architectures (NVIDIA Ampere/Hopper) and the memory hierarchy
  • Communicate clearly and collaborate effectively

Nice to have

  • Familiarity with modern LLM serving frameworks like vLLM, TensorRT-LLM, SGLang, or NVIDIA Triton Inference Server
  • Active open-source contributions to LLM serving frameworks
  • Experience handling continuous audio streams over WebSockets or WebRTC
  • Deploying neural audio codecs
  • Managing chunked audio generation
  • Implementing speculative decoding, lookahead decoding, or chunked prefill
  • Experience with post-training quantization (PTQ), deploying models in FP8, INT8, AWQ, or GPTQ
  • Deploying multi-GPU (Tensor Parallelism) and multi-node inference pipelines
  • Managing autoscaling infrastructure using Kubernetes

Benefits

  • Top-tier healthcare for employees and dependents, including dental and vision
  • Generous employer subsidy for healthcare
  • 401(k) plan with company matching
  • Unlimited PTO
  • 13 paid holidays
  • 12 weeks of paid new parent leave
  • Choice of top-of-the-line laptops/workstations
  • Annual offsites
  • Fully stocked office

About the company

Plaud is a San Francisco-based company building the real-world AI interface for professionals to amplify intelligence. It is a bootstrapped, profitable company with a $300M revenue run rate.