Lead Data Engineer – Hybrid, Scalable Data Pipelines

State Farm; Bloomington (IL); Hybrid; USD 86,000 - 135,000; Full time

Benefits offered by this job

Annual raise and bonus

Healthcare benefits

Tuition assistance

401(k) with company contributions

Parental leave

Flexible time off

Job summary

State Farm is seeking a Data Engineer to build and scale data products and pipelines that enable better business decisions, growth, and customer retention. This role sits in Enterprise Technology and offers collaboration across teams to tackle complex data challenges and deliver meaningful impact.

You will develop automated, reliable data assets, work with structured and unstructured data, and contribute to modern data solutions aligned to enterprise goals in a fast-moving environment.

Qualifications

• Proficiency in programming languages such as Python, Scala, Java, Bash, etc.
• Hands-on experience with AWS services including ETL tools (Glue, EMR Serverless), Lambda, Step Functions, EventBridge, S3, Kinesis Firehose, Redshift, and Iceberg.
• Experience with distributed data processing frameworks such as Apache Spark, Databricks, and Spring Batch.
• Experience with infrastructure as code tools such as OpenTofu (formerly Terraform) for managing cloud resources and deployments.
• Familiarity with CI/CD pipelines including automated testing, and security scans.
• Knowledge of auditing and monitoring tools: Dynatrace, AWS QuickSight and Cloudwatch.

Responsibilities

• Utilizes industry-adopted languages and frameworks in coding, testing, security, DevOps, DataOps and data engineering practices.
• Develops and maintains reusable, scalable, and compliant data solutions across multiple platforms and compute environments.
• Responsible for the identification, acquisition, cleansing, profiling, and ETL of data used in analytic discovery and production solution deployment across multiple platforms.
• Establishes business domain knowledge for existing State Farm data sources and investigates, recommends, and initiates acquisition of data resources, both internal and external.
• Identifies and consults on emerging technologies and critical core systems, including techniques, tools, data sources, and platforms in the data engineering field.
• Familiar with handling datasets containing mixes of structured and unstructured data.
• Exhibits DataOps mindset where team is accountable for ensuring data aligns to enterprise needs and leveraging automation to deliver quality data solutions.
• Collects and analyzes information to identify customer's technical needs, suggest solutions and develop implementation and integration plans.
• Responsible for the analysis, design, deployment, support, and security of technology to ensure the organization is efficiently managing its technology and data-related assets.
• Applies a wide application of complex principles, theories, and concepts in computer science for data engineering solutions.

Skills

Python

Scala

Java

Bash

Spark

Databricks

OpenTofu

CI/CD

Dynatrace

AWS QuickSight

CloudWatch

Tools

Glue

EMR Serverless

Lambda

Step Functions

S3

Kinesis Firehose

Redshift

Iceberg

Job description

State Farm is seeking a Data Engineer to build and scale data products and pipelines that enable better business decisions, growth, and customer retention. This role sits in Enterprise Technology and offers collaboration across teams to tackle complex data challenges and deliver meaningful impact.

You will develop automated, reliable data assets, work with structured and unstructured data, and contribute to modern data solutions aligned to enterprise goals in a fast-moving environment.