AWS Lakehouse Data Engineer

System One, Inc.; McLean (VA); Remote; USD 140,000 - 200,000; Full time

Benefits offered by this job

Health benefits

401(k) plan

Job summary

System One, Inc. is seeking an AWS Lakehouse Data Engineer to build and operate batch and streaming data pipelines. You will design ETL/ELT workflows with Python and PySpark, implement CDC and schema validation, and create a Delta Lakehouse-style platform on AWS using Iceberg and S3.

The role emphasizes data governance, IAM/KMS security, IaC, and CI/CD for data platforms, collaborating with cross-functional teams to deliver scalable, observable data solutions.

Qualifications

• Bachelor's degree or 4 years of equivalent practical experience.
• 6 years of relevant data-engineering experience.
• Hands-on with AWS-native data lake/lakehouse on S3.
• Strong Python/PySpark ETL/ELT development and data modeling.
• Experience with Apache Iceberg including ACID and time travel.
• Advanced SQL for analytics, reporting, and visualization.
• Data governance, cataloging, lineage, and access control using AWS.
• IaC and CI/CD for data platforms across environments.
• Experience with testing, monitoring, and cost optimization.

Responsibilities

• Build and operate batch and streaming data pipelines.
• Design, test, and optimize ETL/ELT pipelines with Python and PySpark.
• Implement CDC, data contracts, schema validation, and reusable transforms.
• Improve reliability via testing, orchestration, monitoring, retries.
• Create a Delta Lakehouse-style platform on AWS with S3 and Iceberg.
• Enable fast queries using Athena, EMR, Glue, and Redshift where applicable.
• Partitioning, file sizing, caching, lifecycle policies to optimize cost.
• Establish dev/test/prod environments with consistent config and promotion.
• Implement governance, IAM, KMS, lineage, and data quality checks.
• Develop IaC, CI/CD pipelines, and platform observability.

Skills

Python

PySpark

SQL

Apache Iceberg

AWS Lakehouse

Data governance

IaC

CI/CD

Education

Bachelor's degree or equivalent

Tools

Apache Iceberg

AWS

Terraform

CloudFormation

Jenkins

Git

Job description

AWS Lakehouse Data Engineer

Work Model: Remote – offsite

Responsibilities

• Build and operate data pipelines (batch and streaming) from APIs, relational databases, file drops, event streams, and external partners.
• Design, implement, test, and optimize ETL/ELT pipelines using Python and PySpark to produce analytics-ready datasets for reporting, visualization, and machine learning.
• Implement incremental processing, change data capture (CDC), data contracts, schema validation, and reusable transformation frameworks.
• Improve pipeline reliability through automated testing, orchestration, monitoring, retries, and operational runbooks.
• Design and implement a Delta Lakehouse-style data platform on AWS using native services to provide Databricks-like capabilities for data engineering, analysis, and data visualization.
• Build and manage a scalable lakehouse on Amazon S3 using Apache Iceberg and open columnar formats such as Apache Parquet.
• Implement SQL-like table reliability features including ACID transactions, schema evolution, snapshot isolation, and time travel using Apache Iceberg.
• Enable fast, interactive queries of lakehouse data via AWS-native services like Amazon Athena, Amazon EMR, AWS Glue, and Amazon Redshift where appropriate.
• Optimize performance and cost through partitioning, file sizing, caching, lifecycle policies, and separating compute from storage.
• Establish standardized environments for development, testing, and production with consistent configuration and controlled promotion.
• Implement data governance, access control, lineage, and quality measures utilizing AWS-native services including AWS Lake Formation, AWS Glue Data Catalog, IAM, KMS.
• Create a metadata repository with cataloging, ownership, classification, tagging, and discoverability features.
• Enable end-to-end data lineage for audit and regulatory compliance.
• Apply policy-based access, least privilege, data classification, retention, encryption, and secure handling controls.
• Build data quality checks for freshness, completeness, validity, and anomaly detection, and publish SLA/SLO metrics.
• Automate AWS provisioning with Infrastructure as Code (IaC), develop CI/CD pipelines for data components, and ensure platform observability.
• Work collaboratively with cross-functional teams and maintain high-quality engineering documentation.

Requirements

• Bachelor's degree in Engineering, Information Technology, Computer Science, Data Engineering, or related field, or four (4) years of equivalent practical experience.
• Six (6) years of relevant experience.
• Hands-on experience building AWS-native data lake or lakehouse architectures on Amazon S3.
• Strong experience developing production ETL/ELT pipelines with Python and PySpark, including data modeling, transformation, and performance tuning.
• Hands-on experience with Apache Iceberg, including ACID transactions, schema evolution, time travel, and query optimization.
• Advanced SQL skills supporting analytical workloads, reporting, and data visualization.
• Proven experience with data governance, cataloging, lineage, and access control using AWS services.
• Knowledge of AWS security fundamentals: IAM, KMS, secrets management, network security, logging, SDLC.
• Proven experience with Infrastructure as Code (IaC) and operating data platforms across environments.
• Experience with CI/CD pipelines for data workflows with testing, deployment, environment promotion, and rollback.
• Troubleshooting distributed data workloads, performance optimization, and cost management skills.
• Excellent collaboration and communication skills to coordinate with cross-team stakeholders.

Would Be Nice to Have

• Experience with Databricks, Delta Lake, migrating workloads to AWS-native services, and Apache Iceberg.
• Familiarity with AWS Step Functions, MWAA, Kinesis, DMS, Lambda, MSK, or similar services.
• Experience with modern DevOps tools: Git, Terraform, CloudFormation, Jenkins, CodePipeline, GitHub Actions, Docker.
• Knowledge of BI and visualization tools like Amazon QuickSight, Tableau, Power BI.
• Familiarity with AI-assisted coding tools such as GitHub Copilot, ChatGPT, Cursor, or Kiro.
• Knowledge of graph modeling, ontology, taxonomy, entity resolution, and hybrid retrieval techniques.

System One, and its subsidiaries including Joulé and Mountain Ltd., are leaders in delivering outsourced services and workforce solutions across North America. We help clients get work done more efficiently and economically, without compromising quality. System One not only serves as a valued partner for our clients, but we offer eligible employees health and welfare benefits coverage options including medical, dental, vision, spending accounts, life insurance, voluntary plans, as well as participation in a 401(k) plan.

System One is an Equal Opportunity Employer. All qualified applicants will receive consideration for employment without regard to race, color, religion, sex (including pregnancy, childbirth, or related medical conditions), sexual orientation, gender identity, age, national origin, disability, family care or medical leave status, genetic information, veteran status, marital status, or any other characteristic protected by applicable federal, state, or local law.