Lead Data Engineer - Azure, Databricks & Streaming

84.51˚

Chicago (IL)

On-site

USD 125,000 - 207,000

Full time

19 hours ago
Be an early applicant

A complete application in a minute

Tailored resume and cover letter, ready to send.

Get past ATS filters

Benefits offered by this job

Health benefits including medical, and
401(k) with Roth option
Health Savings Account (HSA) withMatch
AD&D and supplemental insurance
Paid time off plus holidays and family

Job summary

84.51° is seeking a Lead Data Engineer to design and implement data ingestion, storage, transform and distribution on Azure Cloud using Databricks, Python, SQL and Spark. You will provide technical leadership to a blended onshore/offshore team and collaborate closely with Product, Data Science, and architects to power our commercial products.

You will build event-driven data solutions, define schemas, ensure data quality, and deliver REST APIs in Kubernetes.

Qualifications

  • Bachelor's degree or equivalent experience in a technical field.
  • 5+ years of professional data engineering experience.
  • 5+ years with Spark (Databricks, PySpark).
  • 5+ years developing with Python and SQL.
  • 2+ years of technical leadership or mentoring.
  • Experience building event-driven data solutions with Spark and messaging systems.
  • Schema management and backward-compatible evolution experience.
  • Proficiency with Azure data services and CI/CD.
  • Strong data modeling and governance skills.
  • Experience with REST APIs and Kubernetes.
  • Experience with monitoring, on-call or incident response.

Responsibilities

  • Provide technical leadership to a blended onshore/offshore team.
  • Lead design and development of Databricks-based solutions.
  • Design and build event-driven data solutions and streaming pipelines.
  • Define and maintain event contracts and schemas.
  • Build streaming pipelines with ordering, late data handling, retries and backfills.
  • Develop data pipelines supporting the commercial business.
  • Design data models, schemas, and operational procedures.
  • Implement data quality checks, testing, monitoring and error handling.
  • Own operational health of pipelines, including incident response.
  • Collaborate with Data Science and senior engineers to ensure best practices.
  • Develop CI/CD, infrastructure as code, and Kubernetes-hosted REST APIs.

Skills

Leadership
Communication
Agile
Problem solving
Mentoring
Team collaboration
Strategic thinking

Education

Bachelor's degree in Computer Science or equivalent

Tools

Databricks
Spark
PySpark
Python
SQL
Azure
Kubernetes
CI/CD
Git
Kafka
Delta Lake
Unity Catalog
REST APIs
Azure Data Factory

Job description

84.51° is seeking a Lead Data Engineer to design and implement data ingestion, storage, transform and distribution on Azure Cloud using Databricks, Python, SQL and Spark. You will provide technical leadership to a blended onshore/offshore team and collaborate closely with Product, Data Science, and architects to power our commercial products.

You will build event-driven data solutions, define schemas, ensure data quality, and deliver REST APIs in Kubernetes.

Get your free, confidential resume review.

or drag and drop your file here.

Similar jobs

Similar jobs worth comparing

Lead Data Engineer: Streaming Data on Azure & Databricks
Lead Data Engineer: Streaming Data on Azure & Databricks

84.51° • Cincinnati (OH), Chicago (IL)

On-site
USD 125,000 - 207,000
Lead Data Engineer: Databricks & Azure Streaming Architect
Lead Data Engineer: Databricks & Azure Streaming Architect

84.51˚ • Cincinnati (OH)

On-site
USD 125,000 - 207,000
Health insurance
401(k) with matching
Paid time off
Lead Data Engineer — Azure, Databricks & Streaming
Lead Data Engineer — Azure, Databricks & Streaming

8451 • Cincinnati (OH)

On-site
USD 125,000 - 207,000
Lead Data Engineer: Azure & Databricks, Scale Personalization
Lead Data Engineer: Azure & Databricks, Scale Personalization

84.51˚ • Chicago (IL)

On-site
USD 125,000 - 207,000
Health insurance
401(k) with Roth option and matching
Paid time off and holidays
Lead Data Engineer: Azure & Databricks, Mentorship
Lead Data Engineer: Azure & Databricks, Mentorship

8451 • Cincinnati (OH)

On-site
USD 125,000 - 207,000
Health insurance
Dental coverage
Vision coverage
+6
Lead Data Engineer - Azure Databricks & Lakehouse
Lead Data Engineer - Azure Databricks & Lakehouse

EPAM Systems • United States

Remote
USD 140,000 - 170,000
Lead Data Engineer - Azure Databricks & ETL
Lead Data Engineer - Azure Databricks & ETL

Ethereum Technologies • City of Albany (NY)

On-site
USD 84,000 - 120,000
Lead Data Engineer – Azure, Databricks & Snowflake (Remote)
Lead Data Engineer – Azure, Databricks & Snowflake (Remote)

Exusia, Inc. • Miami (FL)

Remote
USD 120,000 - 150,000
Lead Data Engineer: Azure Databricks & ETL Leader
Lead Data Engineer: Azure Databricks & ETL Leader

Octapharma Plasma • Charlotte (NC)

On-site
USD 120,000 - 190,000
Medical, dental, vision insurance
401k retirement plan
Paid time off
+1
Lead Data Engineer – Databricks Lakehouse, Unlimited PTO
Lead Data Engineer – Databricks Lakehouse, Unlimited PTO

V4C.ai • United States

Remote
USD 140,000 - 190,000
Unlimited PTO
Health Care Plan (Medical, Dental &amp
Vision
+6