Machine Learning Engineer (Real-Time Speech Translation)

The role

You will own the real-time speech translation backend end-to-end. You will build a low-latency streaming system using production model serving platforms and in-house adaptive machine translation models.

What you will do

• Build and manage services for high-throughput, real-time audio and text streaming
• Integrate and serve streaming speech recognition and machine translation models
• Develop logic for model-based confidence scoring and human intervention routing
• Architect and scale production ML infrastructure on GPU-accelerated Kubernetes clusters
• Implement batching, load balancing, and autoscaling strategies
• Establish instrumentation for real-time performance and optimize latency
• Define technical contracts and interfaces for audio ingestion and downstream services
• Partner with frontend and platform engineering teams for system integration

Required qualifications

• BS or MS in Computer Science or a related field, or equivalent practical experience
• 3+ years building production backend or ML serving systems in Python
• Strong async programming (asyncio) skills
• Hands-on experience with real-time streaming transport: WebSocket or gRPC bidirectional streaming, session state, backpressure, and connection lifecycle handling
• Experience serving ML models in production on GPUs (Ray Serve, Triton, vLLM, or similar), with Docker and Kubernetes
• Experience integrating speech or NLP models into production systems, ideally streaming ASR (partial hypotheses, endpointing, VAD)
• Ability to profile, instrument, and optimize a real-time or low-latency system
• Effective use of AI coding agents (Claude Code, Codex, or similar)
• US citizenship and residence in the United States

Nice to have

• Ray Serve specifically, including streaming responses and model multiplexing
• Familiarity with simultaneous or incremental MT concepts (retranslation, prefix stability, wait-k policies)
• Machine translation quality estimation (COMET/CometKiwi class models) or other confidence estimation in production
• Message brokers for real-time fan-out and state distribution (RabbitMQ or similar)
• Streaming text-to-speech integration and time-to-first-audio optimization
• WebRTC and SFU concepts, or voice pipeline frameworks (LiveKit Agents, Pipecat)
• Handling of CJK and other non-Latin text in NLP pipelines
• Observability tooling (Datadog, Prometheus) for production ML systems

Before you apply

• Requires US citizenship and residence in the United States

About the company

LILT is a machine learning company founded in 2015 that uses AI, machine translation, and human-in-the-loop expertise to translate content. The company aims to make the world's information accessible to everyone regardless of language.