Lead Data Engineer (AWS)

State Farm; Arizona; Hybrid; USD 110,000 - 150,000; Full time

Job summary

State Farm seeks a Data Engineer or Lead Data Engineer to join the SAIL team within the Applied Innovation Lab, focused on Agency and Digital. You will design and build data capabilities to enable experimentation, product development, and emerging technology solutions.

The ideal candidate will have strong data engineering fundamentals, comfort working in ambiguity, and the ability to mentor others while collaborating across technical and business teams.

Qualifications

• 3+ years of Python development for extracting, transforming, processing, and moving data.
• Proficiency writing SQL queries and working with relational databases.
• Experience designing, building, and maintaining ETL/ELT data pipelines.
• Experience with structured, semi-structured, and unstructured data (CSV, JSON, text).
• Experience with AWS services (S3, Glue, Lambda).
• Experience using Git and modern version-control practices.
• Familiarity with CI/CD practices and deploying code into shared or production environments.
• Comfort with Linux and command-line environments.
• Understanding of software engineering principles: testing, maintainability, observability, reusable design.
• Ability to troubleshoot data quality, pipeline, integration, and performance issues.

Responsibilities

• Write code to move and transform data and support data access across teams.
• Collaborate with software engineers and product teams to determine how data can be accessed and analyzed.
• Design and maintain platforms where structured and unstructured data lives.
• Lead data science, machine learning, and generative AI efforts as needed, across the lifecycle from exploration to deployment.
• Provide technical leadership, mentoring, and influence architectural practices within the team.

Skills

Python development

SQL

ETL pipelines

Data formats (CSV/JSON)

AWS

Git

CI/CD

Linux

Testing/Observability

Data quality

Tools

S3

Glue

Lambda

Git

CI/CD tools

Job description

Overview

Being good neighbors – helping people, investing in our communities, and making the world a better place – is who we are at State Farm. It is at the core of how we operate and the reason for our success. Come join a #1 team and do some good!

HYBRID: Qualified candidates must live or relocate within a 180-mile radius of a hub location listed below and should plan to spend time working from home and some time working in the office as part of our hybrid work environment.HUB LOCATIONS: Bloomington, IL ; Richardson, TX ; Tempe, AZ ; Dunwoody, GA

SPONSORSHIP: Applicants for this position are required to be eligible to lawfully work in the U.S. immediately; employer will not sponsor applicants for U.S. work authorization (e.g. H-1B visa) for this opportunity.

Responsibilities

Innovation Group is seeking a Data Engineer or Lead Data Engineer to join our product execution team within SAIL (State Farm Applied Innovation Lab) focused on Agency and Digital. You will work alongside software engineers, product teams, and business partners to design and build the data capabilities that enable experimentation, product development, and emerging technology solutions.

Our teams operate in environments where the problem, data, and solution may not be fully defined at the outset. We are looking for engineers who are comfortable working through that ambiguity, quickly learning new technologies and domains, and turning complex data needs into practical, scalable solutions.

Day to day, the work centers on three areas: writing code to move and transform data; partnering with software engineers and product teams to determine how data can be accessed, analyzed, and used; and designing and maintaining the platforms where structured and unstructured data lives. Strong engineering fundamentals are essential, but success in this role also requires flexibility, curiosity, innovative thinking, and the ability to navigate a complex organization.

The ideal candidate understands the fundamentals of data engineering and how data solutions are designed, built, deployed, monitored, and maintained as production-quality applications rather than one-off scripts. They communicate clearly, collaborate effectively across technical and non-technical teams, seek to understand the business problem behind the request, and continuously develop their skills as technologies and needs evolve. Depending on the initiative, this role may also help lead data science, machine learning, and generative AI efforts, working across the lifecycle from data exploration and experimentation through engineering, deployment, and ongoing operation.

At the Lead Data Engineer level, the role also includes providing technical leadership, mentoring engineers, influencing architecture and engineering practices, and helping shape the direction of the team's data capabilities and solutions.

Qualifications

• 3 plus years with python development for extracting, transforming, processing, and moving data
• Proficiency writing SQL queries and working with relational databases
• Experience designing, building, and maintaining ETL/ELT data pipelines
• Experience working with structured, semi-structured, and unstructured data, including formats such as CSV, JSON, and text
• Experience with public cloud platforms, preferably AWS, including services such as S3, Glue, and Lambda
• Experience using Git and modern version-control practices
• Familiarity with CI/CD practices and deploying code into shared or production environments
• Comfort working in Linux and command-line environments
• Understanding of software engineering principles such as testing, maintainability, observability, and reusable design
• Ability to troubleshoot data quality, pipeline, integration, and performance issues

Additional Skills That Would Be Valuable

• Experience designing data architectures for new products, experiments, or emerging technology use cases
• Familiarity with machine learning concepts and the end-to-end ML lifecycle, including data preparation, feature engineering, model development, evaluation, deployment, monitoring, and iteration
• Experience partnering with data scientists or ML engineers to operationalize models and build the data pipelines and infrastructure that support machine learning solutions
• Familiarity with generative AI, large language models (LLMs), and the data considerations associated with building AI-enabled applications
• Experience designing or building LLM-based workflows, including agentic workflows that use tools, APIs, retrieval, orchestration, or multi-step reasoning to complete tasks
• Experience using AI-assisted development tools as part of the engineering workflow to accelerate coding, testing, debugging, documentation, and technical exploration
• Ability to evaluate AI-generated code and outputs critically and apply appropriate engineering, security, testing, and governance practices
• Experience with APIs, event-driven architectures, or streaming data
• Familiarity with object storage and modern data platforms
• Experience working with large or complex datasets
• Understanding of data governance, security, privacy, lineage, and access-management considerations
• Experience working in Agile or product-oriented engineering environments
• Ability to translate business needs into technical data requirements and solutions
• Experience working across teams and managing technical dependencies
• Comfort experimenting with new technologies and solving problems where the path forward is not yet fully defined