Data Engineer, AWS Spark ETL & Lakehouse (Hybrid)

Peregrine Advisors; Washington (District of Columbia); Hybrid; USD 130,000 - 185,000; Full time

Benefits offered by this job

Medical, dental, vision insurance (emp

401(k) with 100% match up to 4%

Unlimited PTO

Sponsored certifications

Job summary

Peregrine Advisors is seeking a data engineer to design and operate AWS-based pipelines that ingest, process, and store federal agency data. You will architect S3 data lakes with Iceberg, leverage Spark ETL on Glue/EMR, and build robust data quality, lineage, and governance.

You will work in a hybrid DC-area setting and partner with cross-functional teams to deliver auditable, scalable data solutions. You will need 4+ years of experience, Master’s in a relevant field, and familiarity with tools

Qualifications

• 4+ years of data engineering experience.
• Spark ETL on AWS (Glue, Amazon EMR) in Python and PySpark.
• S3 data-lake design (Parquet, partitioning, lifecycle) feeding Apache Iceberg tables, Amazon Aurora PostgreSQL, and DynamoDB.
• Event orchestration (Lambda, Step Functions, SQS/SNS) with secrets management and monitoring.
• Data quality, validation, and lineage.
• Infrastructure-as-code (CloudFormation or Terraform).
• Federal information technology or high-volume data experience.
• Familiarity with AI-assisted developer tooling.

Responsibilities

• Build and maintain ingest-process-store pipelines on AWS.
• Design and manage a data lake feeding platform stores via Iceberg, Aurora PostgreSQL, and DynamoDB.
• Implement event orchestration and monitoring with CI/CD and secure access controls.
• Ensure data quality, validation, and lineage across data flows.
• Collaborate in a regulated environment ensuring accuracy and auditability.

Skills

Data engineering

AWS

ETL

Education

Master's degree in a relevant field

Tools

Glue

EMR

PySpark

Python

S3

Trino

Iceberg

Aurora PostgreSQL

DynamoDB

Lambda

Step Functions

CloudFormation

Terraform

SQS/SNS

Apache Ranger

DataStage

Job description

Peregrine Advisors is seeking a data engineer to design and operate AWS-based pipelines that ingest, process, and store federal agency data. You will architect S3 data lakes with Iceberg, leverage Spark ETL on Glue/EMR, and build robust data quality, lineage, and governance.

You will work in a hybrid DC-area setting and partner with cross-functional teams to deliver auditable, scalable data solutions. You will need 4+ years of experience, Master’s in a relevant field, and familiarity with tools