Staff Data Infra Engineer — Real-Time, Scalable

Peregrine; Washington (District of Columbia); On-site; USD 200,000 - 275,000; Full time

Benefits offered by this job

Benefits

Equity

Bonus

Job summary

Peregrine Technologies is seeking a Staff Data Infrastructure Engineer to own the data layer across ingestion, storage, and serving of massive real-time data. You will architect scalable pipelines and collaborate with platform and product teams to improve decision-making speed and reliability.

The role demands 8+ years in data infra, expertise with Iceberg, Spark, Kafka, Airflow, Kubernetes, and AWS data lakes. On-site roles in major US cities are available with competitive compensation.

Qualifications

• 8+ years of experience architecting and operating large-scale data infrastructure systems.
• Expertise with Apache Iceberg, including schema evolution, partitioning, compaction and time travel.
• Hands-on experience with Apache Spark for batch and streaming data processing at scale.
• Experience with real-time data integration using Kafka, Flink, or equivalents.
• Data pipeline orchestration with Airflow or similar tools.
• Strong Python/Scala production-grade coding fundamentals.
• Extensive AWS experience with data lake architectures (S3).
• Knowledge of Kubernetes and containerized deployment of data workloads.
• Degree in Computer Science or Engineering, or equivalent practical experience.

Responsibilities

• Designing and operating a high-throughput, real-time data integration platform across diverse customer environments.
• Architecting a scalable open table format layer for reliable data storage at petabyte scale.
• Building and optimizing distributed data processing pipelines with Spark and related streaming tech.
• Driving performance, reliability, and cost efficiency across the full data infrastructure stack.
• Collaborating with platform and product teams to define data contracts, schemas, and integration patterns.
• Establishing best practices, tooling, and patterns to raise data infrastructure quality.

Skills

Python

Scala

Data pipelines

Distributed systems

Data infrastructure

Education

CS/Engineering degree

Tools

Apache Spark

Apache Kafka

Apache Iceberg

Airflow

Kubernetes

AWS GovCloud

S3

Job description

Peregrine Technologies is seeking a Staff Data Infrastructure Engineer to own the data layer across ingestion, storage, and serving of massive real-time data. You will architect scalable pipelines and collaborate with platform and product teams to improve decision-making speed and reliability.

The role demands 8+ years in data infra, expertise with Iceberg, Spark, Kafka, Airflow, Kubernetes, and AWS data lakes. On-site roles in major US cities are available with competitive compensation.