Lead Data Engineer Scalable Big Data Pipelines (Relocation)

American Family Insurance Agency Staff; Keene (NH); On-site; USD 131,000 - 220,000; Full time

Job summary

American Family Insurance seeks an experienced data engineering lead to architect and deliver scalable data solutions within the company’s big data platform. You will oversee a team, ensure data quality, and collaborate with architecture and development teams to drive re-use and operational efficiency.

Requirements include strong GCP experience, Python ETL expertise, SQL/NoSQL proficiency, and experience with Hadoop/Spark and data modeling.

Qualifications

• Requires strong GCP experience and data engineering background.
• Proficient in SQL or NoSQL and multiple data stores.
• Extensive Python programming for ETL workflows.
• Experience with big data batch processing (Hadoop/Spark).
• Knowledge of data modeling (dimensional modeling, star schemas).
• Familiarity with CI/CD, Git, Jenkins; containerization via Docker.

Responsibilities

• Leads an engineering team to meet project deadlines and priorities.
• Supervises data engineering team members and activities.
• Ensures data quality, security, privacy, and integrity across the lifecycle.

Skills

GCP

SQL/NoSQL

Python ETL

Big data tools

Data modeling

CI/CD tools

Docker/Infra-as-Code

Web services

Tools

Hadoop/Spark

SparkML/H2O

Job description

American Family Insurance seeks an experienced data engineering lead to architect and deliver scalable data solutions within the company’s big data platform. You will oversee a team, ensure data quality, and collaborate with architecture and development teams to drive re-use and operational efficiency.

Requirements include strong GCP experience, Python ETL expertise, SQL/NoSQL proficiency, and experience with Hadoop/Spark and data modeling.