Senior Software Engineer, AI Gateway

The role

This person will design and own core components of the AI Gateway. They will implement features like prompt engineering, caching, and observability for AI services using Golang and/or Python.

What you will do

• Design and own core components of the AI Gateway
• Lead the implementation of features like prompt engineering, caching, and observability
• Write clean, maintainable Golang and/or Python code
• Influence the technical roadmap and architecture
• Identify and mitigate technical debt
• Architect solutions for integrating with Large Language Models and AI platforms
• Mentor junior engineers through code reviews and design discussions
• Drive the full development lifecycle from ideation to deployment in Kubernetes
• Collaborate with product managers and cross-functional teams

Required qualifications

• 5+ years of experience in backend development
• Experience with AI/ML infrastructure, LLMs, or frameworks like LangChain or LlamaIndex
• Deep understanding of distributed systems, microservices architecture, and API design principles
• Proven experience designing and operating services in a cloud-native environment (Kubernetes, Docker, AWS/GCP/Azure)
• Familiarity with API gateways, service mesh, or related infrastructure technologies
• Strong computer science fundamentals (data structures, algorithms, concurrency)

Nice to have

• Contributions to open-source projects
• Experience with performance tuning and optimization of high-throughput systems
• Knowledge of Lua/OpenResty

About the company

Kong Inc. is the AI Connectivity Company building the connectivity layer of AI. Their unified API and AI platform enables organizations to secure, manage, accelerate, govern, and monetize the flow of intelligence across APIs and AI traffic.