Staff Software Engineer, Data Infrastructure

Peregrine Technologies; New York (NY); On-site; USD 200,000 - 275,000; Full time

Benefits offered by this job

Comprehensive benefits

Equity options

Bonus opportunities

Job summary

A leading technology company in New York seeks a Staff Data Infrastructure Engineer to architect systems for real-time operational data. You will design high-throughput data integration platforms and collaborate closely with teams to deliver robust and scalable solutions. Ideal candidates possess 8+ years in large-scale data infrastructure, strong expertise in Apache Iceberg, and solid software engineering skills in Python or Scala. The salary range is $200,000 - $275,000 annually plus benefits and equity.

Qualifications

• 8+ years of experience in large-scale data infrastructure.
• Deep expertise with Apache Iceberg and related technologies.
• Strong software engineering fundamentals in Python or Scala.

Responsibilities

• Architect and build systems for real-time operational data.
• Design and operate high-throughput data integration platforms.
• Collaborate with engineering teams to define data contracts.

Skills

Data integration

Distributed systems

Cloud platforms

Apache Spark

Apache Kafka

Kubernetes

Data resilience

Python

Scala

Education

Degree in Computer Science, Engineering, or related field

Tools

AWS GovCloud

Apache Iceberg

Airflow

Job description

Role

We are looking for a Staff Data Infrastructure Engineer to join our growing team, where you will have deep ownership over the data layer that underpins everything Peregrine does. You will architect and build the systems that ingest, store, and serve massive volumes of real-time operational data — enabling our customers to make critical decisions with speed and confidence.

This is a senior individual contributor role for someone who thrives on hard technical problems and brings the experience and judgment to shape foundational infrastructure decisions. You will tackle a wide range of complex challenges, including:

• Designing and operating a high-throughput, real-time data integration platform across diverse customer environments
• Architecting a scalable open table format layer for reliable data storage at petabyte scale
• Building and optimizing distributed data processing pipelines with Apache Spark and adjacent streaming technologies
• Driving performance, reliability, and cost efficiency across the full data infrastructure stack
• Collaborating with platform and product engineering teams to define data contracts, schemas, and integration patterns
• Establishing best practices, tooling, and patterns that raise the quality bar for data infrastructure across the organization

Our stack is constantly evolving but is built on AWS GovCloud, Apache Iceberg, Apache Spark, Apache Kafka, Airflow, Kubernetes, and more.

About You

• Deep passion for data infrastructure — you care about building systems that are correct, fast, and resilient at scale
• Thrive on ambiguity and are energized by defining the right solution to hard, open-ended problems
• Strong technical vision with the ability to translate complex data requirements into clean, durable infrastructure designs
• Desire to own significant portions of the data stack end-to-end, from ingestion to serving
• Committed to operational excellence — you build things you’re proud to operate

What We Look For

• 8+ years of experience architecting and operating large-scale data infrastructure systems in production environments
• Deep expertise with open table formats, particularly Apache Iceberg — including schema evolution, partitioning strategies, compaction, and time travel
• Extensive hands‑on experience with Apache Spark for batch and streaming data processing at scale
• Strong background in real‑time data integration and stream processing, leveraging technologies such as Apache Kafka, Apache Flink, or equivalents
• Solid experience with data pipeline orchestration using Airflow or similar tools
• Strong software engineering fundamentals in Python and/or Scala, with a track record of writing production-quality code
• Extensive experience with AWS or comparable cloud platforms, including S3-based data lake architectures
• Experience with Kubernetes and containerized deployment of data workloads
• Degree in Computer Science, Engineering, or a related field, or equivalent practical experience

Salary & Location

Located in San Francisco, New York, or Washington DC and open to working in office.

Salary Range: $200,000 - $275,000 Annually + Benefits + Equity (if applicable) + Bonus (if applicable)

Equal Opportunity Statement

Peregrine Technologies is committed to creating an inclusive environment for all employees. We celebrate diversity and are a proud equal opportunity employer. All qualified applicants will receive consideration for employment without regard to race, color, religion, gender, gender identity or expression, sexual orientation, national origin, genetics, disability, age, or veteran status.