Staff Software Engineer

The role

You will design and deliver solutions for complex infrastructure problems. You will architect and build an AI Inference Gateway platform to serve machine learning and generative AI workloads.

What you will do

• Architect and build Coupang's next-generation AI Inference Gateway platform.
• Design and develop request routing, model endpoint abstraction, traffic shaping, load balancing, failover, caching, and policy enforcement mechanisms.
• Drive the technical vision and roadmap for AI inference infrastructure.
• Develop infrastructure components in Go, Java, or Python.
• Design multi-tenant platform capabilities including authentication, authorization, quota management, cost attribution, rate limiting, and governance controls.
• Partner with ML Platform, Model Serving, Data, and Product Engineering teams.
• Lead architecture and design reviews and mentor senior engineers.
• Optimize system performance, latency, throughput, and infrastructure efficiency.
• Define observability standards through metrics, tracing, logging, and SLO-based operations.
• Investigate complex production issues and implement architectural solutions.

Required qualifications

• 8+ years of professional software development experience.
• 5+ years of experience designing and operating large-scale distributed systems in production.
• Strong hands-on programming expertise in one or more of Go, Java, or Python.
• Proven track record of building highly available, mission-critical platform or infrastructure services.
• Experience designing API platforms, service gateways, service mesh, or large-scale networking infrastructure.
• Deep understanding of microservices architecture, distributed systems design, and cloud-native technologies.
• Experience with Kubernetes and containerized workloads in production environments.
• Experience operating services on AWS, Azure, or GCP.
• Strong understanding of observability, reliability engineering, capacity planning, and production operations.

Nice to have

• Experience building AI/ML inference platforms, LLM gateways, model serving infrastructure, or GPU-accelerated workloads.
• Deep expertise in Kubernetes ecosystem technologies such as Gateway API, Ingress Controllers, Service Mesh (Istio, Linkerd, Envoy), and platform networking.
• Experience with inference serving frameworks such as vLLM, Triton Inference Server, TensorRT-LLM, Ray Serve, KServe, SGLang, or similar technologies.
• Strong understanding of high-performance networking, gRPC, HTTP/2, streaming protocols, API gateways, and service proxy architectures.
• Experience designing large-scale traffic management systems including routing, retries, circuit breaking, rate limiting, and request prioritization.
• Experience optimizing latency, throughput, and resource utilization for CPU and GPU workloads.
• Familiarity with GenAI and LLM ecosystems including model deployment, prompt routing, RAG systems, model observability, and AI governance.
• Experience with distributed data systems such as Kafka, Cassandra, Redis, MongoDB, or similar technologies.
• Strong understanding of concurrency, synchronization, asynchronous programming, and non-blocking I/O.

Before you apply

• Complete the attached Internal Transfer Request Form and submit.
• Apply with your Coupang e-mail address.

About the company

Coupang is a fast-growing e-commerce company and a dominant force in South Korean commerce. It features a startup culture with the resources of a large global public company.