Lead Data Engineer: Streaming Data on Azure & Databricks

84.51°

Cincinnati, Chicago (OH, IL)

On-site

USD 125,000 - 207,000

Full time

2 days ago
Be an early applicant

Get a reply from this employer

Your resume and cover letter, tailored to exactly what they’re hiring for.

Get past ATS filters

Job summary

84.51° is seeking a Lead Data Engineer to design and implement big data solutions on Azure using Databricks, Spark, and Python. You will lead a blended onshore/offshore team and collaborate with Product, Data Science and architects to deliver scalable data products for commercial applications.

You will own pipelines, data models, and governance, driving reliability with CI/CD and IaC while mentoring engineers and shaping technical strategy across the SDLC.

Qualifications

  • Bachelor's degree or equivalent experience in CS/IS/Math or related field.
  • 5+ years of professional data engineering experience.
  • 5+ years developing with Spark (Databricks, PySpark).
  • 5+ years developing with Python and SQL.
  • 2+ years of leading or mentoring engineers (onshore/offshore).
  • Experience with streaming data solutions using Spark Structured Streaming and Kafka/Azure Service Bus.
  • Knowledge of Azure data services and CI/CD practices.
  • Strong data modeling and Delta Lake/schema evolution knowledge.
  • Experience with Unity Catalog and data governance.
  • REST APIs and Kubernetes experience.

Responsibilities

  • Own features end-to-end across the SDLC with onshore/offshore teams.
  • Lead design and development of Databricks-based solutions and set engineering standards.
  • Design and build event-driven data pipelines and streaming ingestion.
  • Define event contracts and schemas with versioning and evolution.
  • Build pipelines handling ordering, late data, duplicates, retries, backfill.
  • Develop business domain expertise and translate for technical teams.
  • Maintain data models, schemas, and operational procedures.
  • Implement data quality checks, monitoring, and incident response.
  • Collaborate with Data Science for model data and feature pipelines.
  • Mentor junior engineers and promote best practices.

Skills

Spark (Databricks)
Python
SQL
Azure
Kubernetes
Data Modeling
CI/CD
Streaming
Event-driven
Leadership

Education

Bachelor's degree

Tools

Databricks
Azure Data Factory
Kubernetes
Git
Delta Lake
Unity Catalog

Job description

84.51° is seeking a Lead Data Engineer to design and implement big data solutions on Azure using Databricks, Spark, and Python. You will lead a blended onshore/offshore team and collaborate with Product, Data Science and architects to deliver scalable data products for commercial applications.

You will own pipelines, data models, and governance, driving reliability with CI/CD and IaC while mentoring engineers and shaping technical strategy across the SDLC.

Get your free, confidential resume review.

or drag and drop your file here.

Similar jobs

Similar jobs worth comparing

Lead Data Engineer: Databricks & Azure Streaming Architect
Lead Data Engineer: Databricks & Azure Streaming Architect

84.51˚ • Cincinnati (OH)

On-site
USD 125,000 - 207,000
Health insurance
401(k) with matching
Paid time off
Lead Data Engineer - Azure, Databricks & Streaming
Lead Data Engineer - Azure, Databricks & Streaming

84.51˚ • Chicago (IL)

On-site
USD 125,000 - 207,000
Health benefits including medical, and
401(k) with Roth option
Health Savings Account (HSA) withMatch
+2
Lead Data Engineer — Azure, Databricks & Streaming
Lead Data Engineer — Azure, Databricks & Streaming

8451 • Cincinnati (OH)

On-site
USD 125,000 - 207,000
Lead Data Engineer: Azure & Databricks, Scale Personalization
Lead Data Engineer: Azure & Databricks, Scale Personalization

84.51˚ • Chicago (IL)

On-site
USD 125,000 - 207,000
Health insurance
401(k) with Roth option and matching
Paid time off and holidays
Lead Data Engineer: Azure & Databricks, Mentorship
Lead Data Engineer: Azure & Databricks, Mentorship

8451 • Cincinnati (OH)

On-site
USD 125,000 - 207,000
Health insurance
Dental coverage
Vision coverage
+6
Lead Data Engineer - Azure Databricks & Lakehouse
Lead Data Engineer - Azure Databricks & Lakehouse

EPAM Systems • United States

Remote
USD 140,000 - 170,000
Lead Azure Databricks Engineer
Lead Azure Databricks Engineer

Ethereum Technologies • Charlotte (NC)

On-site
USD 120,000 - 180,000
Lead Data Engineer — Azure Databricks
Lead Data Engineer — Azure Databricks

Sunray Enterprise, Inc. • Washington

On-site
USD 150,000 - 190,000
Lead Data Engineer - Azure Databricks & ETL
Lead Data Engineer - Azure Databricks & ETL

Ethereum Technologies • City of Albany (NY)

On-site
USD 84,000 - 120,000
Azure Databricks Tech Lead - Data Platform
Azure Databricks Tech Lead - Data Platform

CloudBigD • Charlotte (NC)

On-site
USD 130,000 - 170,000