Data Engineering Lead: GenAI‑Driven Pipelines

American International Group; Northern (KY); Hybrid; USD 125,000 - 135,000; Full time

Job summary

American International Group is seeking a Data Engineer Lead within the Data Office to design, build, and operate production-grade data pipelines using PySpark, Python, and SQL on AWS. You will own end-to-end delivery, define data models and SLAs, and uphold engineering standards.

The role includes mentoring engineers and producing clear architecture documentation. In addition, you will collaborate with stakeholders, leverage GenAI tooling to improve productivity, and communicate design

Qualifications

• 7+ years of hands-on data engineering experience delivering production data pipelines end-to-end.
• Bachelor's degree in CS/IS/Engineering or related field.

Responsibilities

• Design, build, and operate production-grade PySpark and Python pipelines on AWS services.
• Own end-to-end delivery of data products from requirements to production.
• Translate business requirements into concrete technical designs, defining pipeline structure, data models, and SLAs.
• Set and enforce engineering standards across the team.
• Conduct thorough code reviews and pair with engineers on hard problems.
• Leverage GenAI tooling to write better code and improve solution quality.
• Collaborate with product owners to refine requirements and communicate decisions clearly.
• Mentor junior engineers and document architecture decisions and runbooks.

Skills

Strong communication

Analytical thinking

Self-motivated learner

Education

Bachelor's degree in Computer Science, Information Systems, Engineering, or related field

Tools

PySpark

Python

SQL

Snowflake

AWS (S3, Glue, EMR, Aurora, Lambda, IAM, CloudWatch)

GenAI tooling (Claude Code CLI, Snowflake Cortex)

Job description

American International Group is seeking a Data Engineer Lead within the Data Office to design, build, and operate production-grade data pipelines using PySpark, Python, and SQL on AWS. You will own end-to-end delivery, define data models and SLAs, and uphold engineering standards.

The role includes mentoring engineers and producing clear architecture documentation. In addition, you will collaborate with stakeholders, leverage GenAI tooling to improve productivity, and communicate design