Lead Data Engineer: Databricks & Azure Streaming Architect

84.51˚

Cincinnati (OH)

On-site

USD 125,000 - 207,000

Full time

14 hours ago
Be an early applicant

A complete application in a minute

Tailored resume and cover letter, ready to send.

Get past ATS filters

Benefits offered by this job

Health insurance
401(k) with matching
Paid time off

Job summary

84.51° is seeking a Lead Data Engineer to design and implement big data solutions on Azure using Databricks, PySpark, Python, and SQL. You will lead an onshore/offshore team, partner with Product, Data Science, and Architects, and own data pipelines and governance.

You will drive streaming and batch processing, build REST APIs, and mentor engineers while ensuring reliability, performance, and cost efficiency across commercial data products.

Qualifications

  • Bachelor's degree in Computer Science, MIS, Math or equivalent experience
  • 5+ years of professional data engineering experience
  • 5+ years developing with Spark (Databricks, PySpark)
  • 5+ years developing with Python and SQL
  • 2+ years leading or mentoring engineers (onshore/offshore)
  • Experience with event-driven/streaming data solutions with Spark Structured Streaming and a messaging platform
  • Schema management for events and backward-compatible evolution
  • Experience with Azure data services (ADLS, Key Vault, AKS, Data Factory)
  • Experience with CI/CD, Git workflows, IaC
  • Strong data modeling, including Delta Lake and schema evolution
  • Experience with Unity Catalog or similar governance
  • REST APIs and Kubernetes experience
  • Production operations, monitoring, incident response and RCA
  • Strong communication and collaboration
  • Experience using AI-assisted development tools

Responsibilities

  • Provide technical leadership to a blended onshore/offshore team
  • Lead design and development of Databricks-based solutions
  • Build event-driven data pipelines and streaming ingestion
  • Define event contracts and schemas with teams
  • Build streaming pipelines handling ordering and late-arriving data
  • Develop domain expertise for technical stakeholders
  • Own pipelines for performance and cost optimization
  • Develop data models, schemas, workflows and procedures
  • Implement data quality checks, testing and monitoring
  • Ensure pipeline health with incident response and RCA
  • Collaborate with Data Science for feature pipelines
  • Mentor junior engineers
  • Promote best practices and continuous improvement

Skills

Spark (Databricks PySpark)
Python
SQL
Leadership
Azure Cloud
Databricks Unity Catalog
REST APIs
Kubernetes
CI/CD
Data governance

Education

Bachelor's degree

Tools

Databricks
Azure Kubernetes Service
Azure Data Factory
Kafka

Job description

84.51° is seeking a Lead Data Engineer to design and implement big data solutions on Azure using Databricks, PySpark, Python, and SQL. You will lead an onshore/offshore team, partner with Product, Data Science, and Architects, and own data pipelines and governance.

You will drive streaming and batch processing, build REST APIs, and mentor engineers while ensuring reliability, performance, and cost efficiency across commercial data products.

Get your free, confidential resume review.

or drag and drop your file here.

Similar jobs

Similar jobs worth comparing

Lead Data Engineer: Streaming Data on Azure & Databricks
Lead Data Engineer: Streaming Data on Azure & Databricks

84.51° • Cincinnati (OH), Chicago (IL)

On-site
USD 125,000 - 207,000
Lead Data Engineer — Azure, Databricks & Streaming
Lead Data Engineer — Azure, Databricks & Streaming

8451 • Cincinnati (OH)

On-site
USD 125,000 - 207,000
Lead Data Engineer - Azure, Databricks & Streaming
Lead Data Engineer - Azure, Databricks & Streaming

84.51˚ • Chicago (IL)

On-site
USD 125,000 - 207,000
Health benefits including medical, and
401(k) with Roth option
Health Savings Account (HSA) withMatch
+2
Lead Data Engineer: Azure & Databricks, Scale Personalization
Lead Data Engineer: Azure & Databricks, Scale Personalization

84.51˚ • Chicago (IL)

On-site
USD 125,000 - 207,000
Health insurance
401(k) with Roth option and matching
Paid time off and holidays
Lead Data Engineer: Azure & Databricks, Mentorship
Lead Data Engineer: Azure & Databricks, Mentorship

8451 • Cincinnati (OH)

On-site
USD 125,000 - 207,000
Health insurance
Dental coverage
Vision coverage
+6
Lead Azure Databricks Engineer
Lead Azure Databricks Engineer

Ethereum Technologies • Charlotte (NC)

On-site
USD 120,000 - 180,000
Lead Data Engineer: Azure Databricks & ETL Leader
Lead Data Engineer: Azure Databricks & ETL Leader

Octapharma Plasma • Charlotte (NC)

On-site
USD 120,000 - 190,000
Medical, dental, vision insurance
401k retirement plan
Paid time off
+1
Lead Data Architect & Engineer - Databricks Lakehouse
Lead Data Architect & Engineer - Databricks Lakehouse

CloudBigD • Atlanta (GA)

On-site
USD 89,000 - 126,000
Lead Data Engineer - Azure Databricks & Lakehouse
Lead Data Engineer - Azure Databricks & Lakehouse

EPAM Systems • United States

Remote
USD 140,000 - 170,000
Azure Databricks Tech Lead - Data Platform
Azure Databricks Tech Lead - Data Platform

CloudBigD • Charlotte (NC)

On-site
USD 130,000 - 170,000