Senior Data Engineer - GCP & Big Data Lead

American Family Insurance; Hopkins (MN); On-site; USD 131,000 - 220,000; Full time

Benefits offered by this job

401(k) plan

Pension plan

Annual incentive

Paid holidays

Paid time off

Student loan repayment

Paid-family leave

Job summary

American Family Insurance in Minnesota seeks a senior data engineer to design and implement data ingestion, processing, and warehousing pipelines within our big data platform, aligning applications with the enterprise architecture.

Lead an engineering team, ensure data quality and security, and collaborate with application teams to deliver scalable, reusable data solutions using GCP, Python, SQL/NoSQL, and open source tools.

Qualifications

• GCP experience required for this position.
• Strong SQL or NoSQL knowledge across diverse data stores.
• Extensive Python programming for ETL and data-driven solutions.
• Familiarity with Hadoop or Spark for distributed processing.
• Knowledge of open source ML toolkits (e.g., sklearn, SparkML).
• Experience with data modeling principles (dimensional modeling, star schemas).
• Understanding database internals: indexes, logging, transactions.
• Experience using infrastructure-as-code tools (Docker, CloudFormation, Terraform).
• Proficiency with CI/CD pipelines and software engineering workflows (Jenkins, Git).
• Ability to author and consume web services.

Responsibilities

• Leads an engineering team to meet project deadlines and priorities.
• Supervises data engineering team members and activities.
• Ensures data quality, security, privacy, and integrity across the lifecycle.
• Documents critical workflows and operational aspects of team responsibilities.
• Develops understanding of data sources, granularity, availability, and limitations.
• Provides technical oversight to architecture and development teams for scalable data solutions.
• Creates maintainable, scalable code to load and manipulate data in the warehouse.
• Facilitates communication across project teams and stakeholders.

Skills

GCP

SQL/NoSQL

Python ETL

Design patterns

Reusable code

Hadoop/Spark

ML toolkits

Data modeling

DB internals

IaC tools

CI/CD

Web services

Tools

Docker

CloudFormation

Terraform

Jenkins

Git

Job description

American Family Insurance in Minnesota seeks a senior data engineer to design and implement data ingestion, processing, and warehousing pipelines within our big data platform, aligning applications with the enterprise architecture.

Lead an engineering team, ensure data quality and security, and collaborate with application teams to deliver scalable, reusable data solutions using GCP, Python, SQL/NoSQL, and open source tools.