AWS Lead Data engineer

Marlabs; Newark (NJ); On-site; USD 120,000 - 160,000; Full time

Job summary

Marlabs Inc. is seeking a Data Engineer for an onsite 12-month assignment in Newark, NJ. The role focuses on designing and implementing data ingestion, transformation, and processing solutions using AWS services, PySpark, and modern lakehouse patterns.

Responsibilities include guiding batch/large-scale processing, onboarding data from legacy systems, and building reusable pipelines for enterprise data domains. F2F interviews are required.

Responsibilities

• Lead the design and implementation of data ingestion, transformation and processing solutions using AWS native services including S3, Glue, EMR, Redshift and Athena.
• Define appropriate engineering patterns for batch and large-scale data processing using Python and PySpark.
• Guide implementation of modern lake/lakehouse patterns, including S3 and Apache Iceberg, where applicable.
• Provide technical direction across design, development, testing, deployment and production support.
• Lead onboarding of data from legacy and enterprise source systems into AWS.
• Drive development of reusable pipelines and engineering components for specific enterprise data domains.
• Support the ADS / Authoritative Data Source approach, where domain data is packaged as trusted data products for enterprise consumption.
• Ensure datasets produced by the engineering team are consumable for downstream analytics and AI use cases.
• Conduct design and code reviews for Python/PySpark and AWS data-engineering components.
• Establish development standards, reusable patterns and engineering best practices.
• Ensure appropriate data validation and reconciliation are incorporated into pipelines.
• Troubleshoot performance and scalability issues across data pipelines and processing workloads.
• Provide day-to-day technical guidance to Data Engineers.
• Break functional/data requirements into engineering tasks and coordinate delivery.
• Communicate technical risks, dependencies and design decisions to stakeholders.
• Mentor engineers and build the team's AWS/Python/PySpark engineering capability.

Skills

Python

PySpark

Data ingestion

Data transformation

ETL/ELT design

DWH concepts

Tools

AWS (S3, Glue, EMR, Redshift, Athena)

Apache Iceberg

Job description

Location : Newark NJ – Onsite (3 days a week)

Duration : 12 Month

F2F interview Must

Key Responsibilities

• Lead the design and implementation of data ingestion, transformation and processing solutions using AWS native services including S3, Glue, EMR, Redshift and Athena.
• Define appropriate engineering patterns for batch and large-scale data processing using Python and PySpark.
• Guide implementation of modern lake/lakehouse patterns, including S3 and Apache Iceberg, where applicable.
• Provide technical direction across design, development, testing, deployment and production support.
• Lead onboarding of data from legacy and enterprise source systems into AWS.
• Drive development of reusable pipelines and engineering components for specific enterprise data domains.
• Support the ADS / Authoritative Data Source approach, where domain data is packaged as trusted data products for enterprise consumption.
• Ensure datasets produced by the engineering team are consumable for downstream analytics and AI use cases.
• Conduct design and code reviews for Python/PySpark and AWS data-engineering components.
• Establish development standards, reusable patterns and engineering best practices.
• Ensure appropriate data validation and reconciliation are incorporated into pipelines.
• Troubleshoot performance and scalability issues across data pipelines and processing workloads.
• Provide day-to-day technical guidance to Data Engineers.
• Break functional/data requirements into engineering tasks and coordinate delivery.
• Communicate technical risks, dependencies and design decisions to stakeholders.
• Mentor engineers and build the team's AWS/Python/PySpark engineering capability.