Big Data Platform Lead Engineer

American Family Insurance Agency Staff; Madison (WI); On-site; USD 131,000 - 220,000; Full time

Benefits offered by this job

Relocation support

H1 visa transfer approved

Job summary

American Family Insurance Agency Staff is seeking a data engineering leader to drive batch and streaming data solutions within a big data platform. You will guide an engineering team, ensure data quality and governance, and advance scalable data pipelines used across the organization.

The role emphasizes hands-on Python development, SQL/NoSQL expertise, and cloud-based data architectures, with relocation support and potential visa considerations for eligible candidates.

Qualifications

• Experience building ETL workflows and data-driven solutions.
• Strong SQL or NoSQL experience with diverse data stores.
• Expert Python programming for data pipelines.
• Experience with Hadoop or Spark for distributed processing.
• Knowledge of cloud platforms (AWS, GCP, Azure).
• Understanding of data modeling, star schemas.
• Familiarity with infrastructure-as-code tools like Docker, Terraform.
• Experience with CI/CD and version control.
• Experience with web services.

Responsibilities

• Leads an engineering team to meet project deadlines and priorities.
• Supervises data engineering team members and activities.
• Ensures data quality, security, privacy and integrity of data throughout the lifecycle.
• Documents critical workflows and operational support aspects of team's responsibilities.
• Develops deep understanding of data sources, granularity, availability, and limitations.
• Provides technical oversight and advice to architecture and development teams for reusable, scalable data solutions.
• Creates maintainable, scalable code to load and manipulate data in the data warehouse.
• Facilitates communication upward and across project teams and business stakeholders.

Skills

Python

SQL/NoSQL

ETL pipelines

Data warehousing

Cloud platforms

Data modeling

Big data tools

CI/CD tools

Web services

Docker

Tools

Hadoop

Spark

Jenkins

Git

CloudFormation

Terraform

Docker

SQL databases

Job description

American Family Insurance Agency Staff is seeking a data engineering leader to drive batch and streaming data solutions within a big data platform. You will guide an engineering team, ensure data quality and governance, and advance scalable data pipelines used across the organization.

The role emphasizes hands-on Python development, SQL/NoSQL expertise, and cloud-based data architectures, with relocation support and potential visa considerations for eligible candidates.