Lead Data Engineer - Scale Data Platforms & Teams

American Family Insurance Agency Staff; Phoenix (AZ); On-site; USD 131,000 - 220,000; Full time

Benefits offered by this job

401(k) plan

Pension plan

Paid time off

Job summary

American Family Insurance is seeking a senior data engineer to lead an engineering team responsible for collecting, storing, and processing data within our big data platform. The role will drive integration with cross-architecture systems and ensure data quality, security, and scalability.

The position emphasizes Python-based ETL workflows, Hadoop/Spark processing, and open-source ML tools. Relocation support is offered; visa transfer is approved for eligible candidates. 10% travel required.

Qualifications

• Experience building customer-focused data solutions.
• Must have GCP experience for this position.
• Strong knowledge of SQL or NoSQL databases and various data stores.
• Extensive Python programming experience, especially ETL workflows.
• Ability to write reusable, robust code and apply design patterns.
• Experience with big data batch processing tools (Hadoop/Spark).
• Familiarity with ML toolkits (sklearn, SparkML, H2O).
• Solid understanding of data modeling and database internals.

Responsibilities

• Leads an engineering team to meet project deadlines and priorities.
• Supervises data engineering team members and activities.
• Ensures data quality, security, privacy, and integrity across the data lifecycle.
• Documents critical workflows and operational support.
• Develops understanding of data sources, granularity, and limitations.
• Provides technical oversight to architecture and development teams for scalable data solutions.
• Creates maintainable, scalable code to load/manipulate data in the warehouse.
• Facilitates communication across project teams and stakeholders.

Skills

Big data

GCP

SQL/NoSQL

Python

ETL pipelines

CI/CD tooling

Distributed computing

Open source ML toolkits

Tools

Docker

CloudFormation

Terraform

Jenkins

Git

Spark

Hadoop

Job description

American Family Insurance is seeking a senior data engineer to lead an engineering team responsible for collecting, storing, and processing data within our big data platform. The role will drive integration with cross-architecture systems and ensure data quality, security, and scalability.

The position emphasizes Python-based ETL workflows, Hadoop/Spark processing, and open-source ML tools. Relocation support is offered; visa transfer is approved for eligible candidates. 10% travel required.