Lead Data Engineer — GenAI-Driven Pipelines on AWS

AIG; Parsippany-Troy Hills (NJ); Hybrid; USD 125,000 - 135,000; Full time

Benefits offered by this job

Hybrid work arrangement

Total Rewards Program

Job summary

AIG is seeking a Data Engineer Lead to design, build, and operate production-grade data pipelines. You’ll work in the Data Office, leveraging PySpark, Python, and SQL on AWS services (S3, Glue, EMR) with data quality checks and observability.

The role emphasizes collaboration with product owners, stakeholders, and other engineers in a hybrid setting. You’ll mentor junior engineers, participate in architecture discussions, and drive ADRs and runbooks while applying Lakehouse principles and

Qualifications

• Bachelor's degree in Computer Science, Information Systems, Engineering, or related field.
• 7+ years of hands-on data engineering delivering production pipelines end-to-end.
• Strong command of PySpark, Python, and SQL including Spark optimization and performance tuning in Snowflake.
• Hands-on experience with AWS data services: S3, Glue, EMR, Aurora, Lambda, IAM, and CloudWatch.
• Knowledge of Lakehouse patterns and data modeling with fault-tolerant ingestion.

Responsibilities

• Design, build, and operate production-grade PySpark and Python pipelines on AWS EMR, Glue, and S3 with data quality checks and observability.
• Own end-to-end delivery of data products in Agile/Scrum, including CI/CD and release management.
• Translate requirements into pipeline designs, data models, and SLAs.
• Review designs, enforce coding standards, and drive governance and observability across teams.
• Mentor engineers and produce ADRs, runbooks, and data dictionaries.

Skills

PySpark

Python

SQL

Data Modeling

Data Quality

Observability

Agile/Scrum

Education

Bachelor's degree in CS/IS/Engineering

Tools

AWS (S3, Glue, EMR)

Snowflake Cortex

Claude Code CLI

CloudWatch

IAM

Job description

AIG is seeking a Data Engineer Lead to design, build, and operate production-grade data pipelines. You’ll work in the Data Office, leveraging PySpark, Python, and SQL on AWS services (S3, Glue, EMR) with data quality checks and observability.

The role emphasizes collaboration with product owners, stakeholders, and other engineers in a hybrid setting. You’ll mentor junior engineers, participate in architecture discussions, and drive ADRs and runbooks while applying Lakehouse principles and