Remote AWS Lakehouse Data Engineer (ETL/ELT, PySpark)

System One, Inc.; McLean (VA); À distance; USD 140 000 - 200 000; Plein temps

Avantages offerts par ce poste

Health benefits

401(k) plan

Résumé du poste

System One, Inc. is seeking an AWS Lakehouse Data Engineer to build and operate batch and streaming data pipelines. You will design ETL/ELT workflows with Python and PySpark, implement CDC and schema validation, and create a Delta Lakehouse-style platform on AWS using Iceberg and S3.

The role emphasizes data governance, IAM/KMS security, IaC, and CI/CD for data platforms, collaborating with cross-functional teams to deliver scalable, observable data solutions.

Qualifications

• Bachelor's degree or 4 years of equivalent practical experience.
• 6 years of relevant data-engineering experience.
• Hands-on with AWS-native data lake/lakehouse on S3.
• Strong Python/PySpark ETL/ELT development and data modeling.
• Experience with Apache Iceberg including ACID and time travel.
• Advanced SQL for analytics, reporting, and visualization.
• Data governance, cataloging, lineage, and access control using AWS.
• IaC and CI/CD for data platforms across environments.
• Experience with testing, monitoring, and cost optimization.

Responsabilités

• Build and operate batch and streaming data pipelines.
• Design, test, and optimize ETL/ELT pipelines with Python and PySpark.
• Implement CDC, data contracts, schema validation, and reusable transforms.
• Improve reliability via testing, orchestration, monitoring, retries.
• Create a Delta Lakehouse-style platform on AWS with S3 and Iceberg.
• Enable fast queries using Athena, EMR, Glue, and Redshift where applicable.
• Partitioning, file sizing, caching, lifecycle policies to optimize cost.
• Establish dev/test/prod environments with consistent config and promotion.
• Implement governance, IAM, KMS, lineage, and data quality checks.
• Develop IaC, CI/CD pipelines, and platform observability.

Connaissances

Python

PySpark

SQL

Apache Iceberg

AWS Lakehouse

Data governance

IaC

CI/CD

Formation

Bachelor's degree or equivalent

Outils

Apache Iceberg

AWS

Terraform

CloudFormation

Jenkins

Git

Description du poste

System One, Inc. is seeking an AWS Lakehouse Data Engineer to build and operate batch and streaming data pipelines. You will design ETL/ELT workflows with Python and PySpark, implement CDC and schema validation, and create a Delta Lakehouse-style platform on AWS using Iceberg and S3.

The role emphasizes data governance, IAM/KMS security, IaC, and CI/CD for data platforms, collaborating with cross-functional teams to deliver scalable, observable data solutions.