Remote AWS Lakehouse Data Engineer for AI & Analytics

Delan Associates, Inc.; United States; Remoto; USD 120.000 - 180.000; Tempo pieno

Vantaggi offerti da questo lavoro

Remote work

Descrizione del lavoro

Delan Associates, Inc. is seeking an AWS Lakehouse Data Engineer to design, implement, and operate a cloud-native data platform powering AI/ML, analytics, reporting, and data visualization. The position offers 100% remote work with preference for the DMV area.

You will build and operate batch and streaming data pipelines, implement Delta Lakehouse on S3 with Iceberg, and enable fast querying using Athena, EMR, Glue, and Redshift, while enforcing data governance and security at scale.

Competenze

• 6+ years of relevant data engineering experience.
• Hands-on experience implementing AWS-native data lake or lakehouse architectures using Amazon S3 and services such as AWS Glue, Amazon Athena, Amazon EMR, AWS Lake Formation, and Amazon Redshift.
• Strong experience developing production ETL/ELT pipelines using Python and PySpark, including data modeling, transformation, testing, performance tuning, and error handling.
• Hands-on experience with Apache Iceberg, including ACID transactions, snapshots, schema and partition evolution, time travel, table maintenance, and query optimization.
• Advanced SQL skills and experience supporting analytical queries, semantic layers, reporting tools, and data visualization workloads.
• Experience implementing metadata management and governance capabilities, including cataloging, lineage, ownership, classification, policy enforcement, and fine-grained access controls.
• Experience with AWS security fundamentals, including IAM and least privilege, KMS encryption, secrets management, network security, logging, and secure SDLC practices.
• Experience provisioning AWS resources using IaC and operating data platforms across multiple environments.
• Experience building or operating CI/CD pipelines for data workflows, including testing, packaging, deployment automation, environment promotion, and rollback.

Mansioni

• Build and operate data pipelines (batch and streaming).
• Implement, test, and optimize ETL/ELT pipelines using Python and PySpark to produce analytics-ready datasets.
• Implement incremental processing, CDC, data contracts, schema validation, and reusable transformations.
• Improve pipeline reliability through automated testing, orchestration, monitoring, retry handling, and runbooks.
• Deliver an AWS-native Lakehouse Data Platform.
• Design and implement a Delta Lakehouse-style platform on AWS for data science and visualization.
• Build and manage a scalable lakehouse on S3 using Iceberg and Parquet.
• Enable fast querying with AWS-native services such as Athena, EMR, Glue, and Redshift.
• Optimize performance and cost via partitioning, compaction, sizing, statistics, caching, lifecycle policies.
• Establish standardized dev/test/prod environments with controlled promotion across stages.
• Metadata, governance, access control, lineage, and quality.
• Implement a managed metadata repository for cataloging and discovery.
• Enable end-to-end lineage and auditability.
• Apply policy-based access and fine-grained controls; data classification and retention.

Conoscenze

Python

PySpark

AWS

Delta Lake

Apache Iceberg

SQL

ETL/ELT

Data governance

CI/CD for data

IaC

Spark

Formazione

Bachelor's degree or equivalent practical experience

Strumenti

AWS Glue

Amazon Athena

Amazon EMR

AWS Lake Formation

Apache Airflow (MWAA)

Apache Iceberg

Docker

Terraform

GitHub Actions

Descrizione del lavoro

Delan Associates, Inc. is seeking an AWS Lakehouse Data Engineer to design, implement, and operate a cloud-native data platform powering AI/ML, analytics, reporting, and data visualization. The position offers 100% remote work with preference for the DMV area.

You will build and operate batch and streaming data pipelines, implement Delta Lakehouse on S3 with Iceberg, and enable fast querying using Athena, EMR, Glue, and Redshift, while enforcing data governance and security at scale.