Lead Data Engineer

Expert Technology Services; Madison (WI); On-site; USD 130,000 - 190,000; Full time

Job summary

Expert Technology Services in Madison, WI seeks a Lead Data Engineer to drive the design, development, and deployment of large-scale data solutions. You will provide technical leadership and guide data engineering teams through architecture and implementation across the GCP stack.

You will build and optimize ETL/ELT pipelines with Python, BigQuery, Dataflow, Dataproc/PySpark, Airflow, Pub/Sub, and Cloud Storage, while ensuring high-performance data processing and robust data models for analytics.

Qualifications

• Lead the design, development, and deployment of large-scale data engineering solutions.
• Provide technical leadership to data engineering teams.
• Architect, build, and maintain ETL/ELT pipelines using Python and GCP tools.
• Develop and optimize complex SQL queries and data models.

Responsibilities

• Lead the design, development, and deployment of large-scale data engineering solutions.
• Provide technical leadership and guidance to data engineering teams.
• Architect, build, and maintain ETL/ELT pipelines using Python and GCP tools.
• Ensure high performance and scalability of data processing workflows with BigQuery, Dataflow, Dataproc/PySpark, Cloud Composer (Airflow), Pub/Sub, and Cloud Storage.
• Develop and optimize complex SQL queries and data models to support analytics.
• Collaborate with stakeholders to translate business requirements into robust data solutions.
• Implement CI/CD, DevOps practices, and automated deployment frameworks for data projects.
• Drive initiatives from concept through production, ensuring best practices in coding, testing, and deployment.
• Support data governance, data quality, and compliance as needed.
• Stay current with emerging cloud data engineering and AI/ML enablement.

Skills

Python

SQL

GCP

BigQuery

Airflow

Dataproc/PySpark

Data modeling

CI/CD

DevOps

Leadership

Data governance

Tools

Snowflake

Databricks

Lakehouse

Job description

Job Summary (Lead Data Engineer)

• Lead the design, development, and deployment of large-scale data engineering solutions.
• Provide technical leadership and guidance to data engineering teams.
• Architect, build, and maintain enterprise-level ETL/ELT pipelines using Python and Google Cloud Platform (GCP) tools.
• Ensure high performance and scalability of data processing workflows with BigQuery, Dataflow, Dataproc/PySpark, Cloud Composer (Airflow), Pub/Sub, and Cloud Storage.
• Develop and optimize complex SQL queries and data models to support business analytics and data science.
• Collaborate with stakeholders to translate business requirements into robust data solutions.
• Implement CI/CD, DevOps practices, and automated deployment frameworks for data engineering projects.
• Drive initiatives independently from concept through to production, ensuring best practices in coding, testing, and deployment.
• Support data governance, data quality, and compliance requirements as needed.
• Stay current with emerging technologies, especially in cloud data engineering and AI/ML enablement.
• (Preferred) Leverage experience with Snowflake, Databricks, lakehouse architectures, and feature engineering pipelines.
• (Preferred) Contribute to Agile processes within large enterprise environments.