Lead Data Engineer: GCP-Driven ETL & Big Data Architect

American Family Insurance; Boston (MA); On-site; USD 131,000 - 220,000; Full time

Benefits offered by this job

401(k)

Pension plan

Annual incentive

9 paid holidays

23 days PTO

Student loan repayment

Paid-family leave

Job summary

American Family Insurance in Boston, MA, is seeking a senior data engineer to lead a data engineering team responsible for collecting, storing, and processing data within our big data platform. The role focuses on building scalable ETL workloads and ensuring data integrity across the organization.

You will guide architecture, collaborate with cross-functional teams, and deliver robust data-driven solutions leveraging GCP, Hadoop/Spark, and modern data stores.

Qualifications

• Must have GCP experience for this position.
• Demonstrated experience providing customer-driven solutions and services.
• In-depth knowledge of SQL or NoSQL and experience with various data stores.
• Extensive Python programming experience with ETL workflows.
• Ability to apply design patterns and generalize code for common use cases.
• Experience with Hadoop or Spark for distributed data processing.
• Knowledge of open source ML toolkits (e.g., sklearn, SparkML, H2O).
• Solid data understanding and business acumen in data-rich industries like insurance or financial services.
• Applied data modeling principles (dimensional modeling and star schemas).
• Strong understanding of database internals, such as indexes, logging, and transactions.
• Experience with infrastructure-as-code tools (Docker, CloudFormation, Terraform).
• Experience with software engineering tools and workflows (Jenkins, CI/CD, Git).
• Experience with web services consumption and integration.

Responsibilities

• Leads an engineering team to meet project deadlines and priorities.
• Supervises data engineering team members and activities.
• Ensures data quality, security, privacy, and integrity throughout the data lifecycle.
• Documents critical workflows and operational support aspects.
• Develops deep understanding of data sources, granularity, availability, and limitations.
• Provides technical oversight to application architecture and development teams for scalable, reusable data solutions.
• Creates maintainable, scalable code to load and manipulate data in the data warehouse.
• Facilitates communication across project teams and stakeholders.

Skills

GCP experience

SQL/NoSQL

Python programming

ETL workflows

Distributed data processing

Tools

Docker

CloudFormation

Terraform

Jenkins

Git

Job description

American Family Insurance in Boston, MA, is seeking a senior data engineer to lead a data engineering team responsible for collecting, storing, and processing data within our big data platform. The role focuses on building scalable ETL workloads and ensuring data integrity across the organization.

You will guide architecture, collaborate with cross-functional teams, and deliver robust data-driven solutions leveraging GCP, Hadoop/Spark, and modern data stores.