Staff Software Engineer, Data Infrastructure

The role

You will architect and build systems that ingest, store, and serve massive volumes of real-time operational data. You will design a high-throughput data integration platform and optimize distributed data processing pipelines.

What you will do

• Designing and operating a high-throughput, real-time data integration platform across diverse customer environments
• Architecting a scalable open table format layer for reliable data storage at petabyte scale
• Building and optimizing distributed data processing pipelines with Apache Spark and adjacent streaming technologies
• Driving performance, reliability, and cost efficiency across the full data infrastructure stack
• Collaborating with platform and product engineering teams to define data contracts, schemas, and integration patterns
• Establishing best practices, tooling, and patterns that raise the quality bar for data infrastructure

Required qualifications

• 8+ years of experience architecting and operating large-scale data infrastructure systems in production environments
• Deep expertise with open table formats, particularly Apache Iceberg
• Extensive hands-on experience with Apache Spark for batch and streaming data processing at scale
• Strong background in real-time data integration and stream processing, leveraging technologies such as Apache Kafka, Apache Flink, or equivalents
• Solid experience with data pipeline orchestration using Airflow or similar tools
• Strong software engineering fundamentals in Python and/or Scala
• Extensive experience with AWS or comparable cloud platforms, including S3-based data lake architectures
• Experience with Kubernetes and containerized deployment of data workloads
• Degree in Computer Science, Engineering, or a related field, or equivalent practical experience
• Located in New York and open to working in office

About the company

Peregrine helps public safety organizations, state and local and governments, federal agencies, and private-sector institutions address society’s challenges with an AI-enabled platform. The company supports hundreds of customers across 30+ states and two countries.