Data Engineer Lead

American International Group; Parsippany-Troy Hills (NJ); Hybrid; USD 125,000 - 135,000; Full time

Benefits offered by this job

Hybrid work model

Competitive benefits

Job summary

American International Group (AIG) is seeking an experienced Data Engineer Lead to design, build, and operate production-grade data pipelines on AWS, with a focus on reliability, observability, and data quality. You will translate business needs into robust data products and support agile delivery within the Data Office.

The role emphasizes collaboration with product owners and stakeholders, mentoring junior engineers, and adopting GenAI tooling to enhance productivity and code quality across

Qualifications

• 7+ years of hands-on data engineering delivering production data pipelines end-to-end.

Responsibilities

• Design, build, and operate production-grade PySpark and Python pipelines on AWS EMR, Glue, and S3 with integrated data quality checks and observability.

Skills

PySpark

Python

SQL

Spark optimization

Data modeling

Architectural patterns

Analytical thinking

Clear communication

Agile/Scrum

Mentoring

Education

Bachelor's degree in Computer Science / Information Systems / Engineering or related field

Tools

AWS S3

AWS Glue

AWS EMR

AWS Aurora

AWS Lambda

AWS IAM

CloudWatch

Snowflake Cortex

Claude Code CLI

Job description

Data Engineer Lead AIG | Data Office · Data Engineering

About The Role

The Data Engineer Lead is a hands‑on engineering role within AIG's Data Engineering organization. You will design, build, and operate production‑grade data pipelines and platforms — and bring sound engineering judgment to every solution you deliver. We actively leverage GenAI tooling — including Claude Code and Snowflake Cortex — as a genuine productivity multiplier across the development lifecycle. We want someone who is motivated to learn, keeps up with a fast‑moving space, and continuously finds new ways to put these tools to work. You will collaborate closely with the product owner, business stakeholders, and fellow engineers. Strong communication and analytical skills matter here as much as technical depth — the ability to ask the right questions, think through trade‑offs clearly, and explain your reasoning is part of the job.

Responsibilities

Design, build, and operate production‑grade PySpark and Python pipelines on AWS EMR, Glue, and S3 with integrated data quality checks and observability. Own end‑to‑end delivery of data products from requirements through production, working within an Agile/Scrum framework — including sprint planning, CI/CD, release management, and production readiness. Translate business and product requirements into concrete technical designs, defining pipeline structure, data models, and SLAs. Demonstrate a sound understanding of architecture and design patterns, evaluate trade‑offs of design decisions, and articulate clear rationale to peers and stakeholders. Set and enforce engineering standards — coding conventions, data quality frameworks, reusable pipeline patterns, and observability hooks — across the team. Conduct thorough code reviews and pair with engineers on hard problems, raising the technical quality of the whole team. Leverage GenAI tooling — Claude Code CLI, Snowflake Cortex — throughout the development lifecycle to write better code faster and improve solution quality. Collaborate with product owners to refine requirements, surface trade‑offs, and push back constructively when scope is technically unworkable. Communicate design decisions and delivery updates clearly to both engineering peers and non‑technical stakeholders. Mentor junior engineers through review, pairing, and knowledge sharing, building team capability alongside product delivery. Produce clear technical documentation — Architecture Decision Record (ADR), runbooks, data dictionaries, and pipeline lineage notes — as a standard part of delivery, not an afterthought. Evaluate and drive adoption of productivity tooling and best practices, setting team norms for safe and effective use.

Experience / Skills Required

Bachelor's degree in Computer Science, Information Systems, Engineering, or a related technical field. 7+ years of hands‑on data engineering experience with a consistent track record delivering production data pipelines end‑to‑end. Strong command of PySpark, Python, and SQL — including Spark optimization, partition management, and query performance tuning in Snowflake. Hands‑on experience with AWS data services: S3, Glue, EMR, Aurora, Lambda, IAM, and CloudWatch. Working knowledge of architecture and design principles — Lakehouse patterns, data modeling, fault‑tolerant ingestion — with demonstrated ability to apply them in practice. Strong analytical and problem‑solving skills: able to decompose complex requirements, reason through design trade‑offs, and arrive at practical, well‑justified solutions. Clear communicator — able to explain technical decisions and design rationale in plain language to both engineering peers and business stakeholders. Self‑motivated learner with a genuine interest in staying current with evolving tools, frameworks, and engineering best practices.

Preferred / Nice To Have

AWS certification — Solutions Architect Associate or equivalent. Domain familiarity with insurance or financial services data — claims, policy, risk, or actuarial structures. Experience writing ADRs, design notes, or technical documentation in a collaborative engineering team. Demonstrated use of GenAI tooling (e.g. Claude Code CLI or Snowflake Cortex) to improve productivity or solution quality.

Our Culture

AIG's Data Office is a team of builders, thinkers, and pragmatic problem‑solvers. We move with the urgency of a technology company and the accountability of a global insurer. Great data engineering is a team sport here. We write documentation as carefully as we write code, review designs together, mentor across every level, and keep business stakeholders close to the work. We lean into GenAI not as a novelty but as a genuine force multiplier — and we want people who share that mindset. AIG is committed to a diverse and inclusive workforce and encourages candidates from all backgrounds to apply. We offer competitive compensation, comprehensive benefits, and hybrid flexibility.

“The base salary range for this position is $125,000-135,000 and the position is eligible for a bonus in accordance with the terms of the applicable incentive plan. Your actual compensation will be dependent on your skills, experience, and qualifications. If your compensation expectations are above the posted range, we still encourage you to apply—your unique background matters to us. In addition, we’re proud to offer a range of competitive benefits, a summary of which can be viewed here: https://sprcdn-assets.sprinklr.com/248/7cae7257-96d1-4207-bc2c-e1b8447826e3-220371966/2026_AIG_Benefits_Overview.pdf .

Functional Area: DT - Data

AIG PC Global Services, Inc. As a global risk leader with deep industry expertise and innovative solutions to smartly manage risk, AIG enables our clients’ growth in ways they never thought possible. We also do the same for our colleagues, because we know our people are our greatest strength – the source of every insight, every idea and every innovation. When we're working as one team to do what's right for our colleagues and our communities, we can achieve excellence together. We encourage colleagues to give back to the causes they care most about, supporting these efforts through our Volunteer Time Off and Matching Grants Programs.

AIG provides equal opportunity to all qualified individuals regardless of race, color, religion, age, gender, gender expression, national origin, veteran status, disability or any other legally protected categories. AIG is committed to working with and providing reasonable accommodations to job applicants and employees with disabilities. If you believe you need a reasonable accommodation, please send an email to [e-mail].

Additional information about AIG can be found at www.aig.com | YouTube | Instagram | LinkedIn.

Join our Talent Community.