Senior Data Engineer, Azure Databricks

Nextech Solutions

United States

Remote

USD 130,000 - 170,000

Full time

8 days ago

Get a reply from this employer

Your resume and cover letter, tailored to exactly what they’re hiring for.

Get past ATS filters

Job summary

Nextech Solutions is seeking a Senior Data Engineer to design, build, and optimize enterprise data warehouses and production pipelines in Azure Databricks. This role focuses on real-time, event-driven data processing in a regulated environment.

You will own data domains, ensure data quality, performance, and cost efficiency, and collaborate with Data Architecture, Analytics Engineering, QA, and governance teams.

Qualifications

  • 6+ years of progressive data engineering experience in enterprise environments.
  • Deep expertise in Azure Databricks, Delta Lake, Medallion Architecture, Unity Catalog, and production pipelines.
  • Hands-on experience with event-driven streaming data pipelines (Kafka, Kinesis, Confluent).
  • Strong data warehousing experience with dimensional modeling and ETL/ELT processes.
  • Proficiency in SQL, Python, and PySpark, optimizing complex pipelines.
  • Experience with Azure Data Factory, Data Lake Storage, and CI/CD in Azure DevOps.

Responsibilities

  • Design, develop, and maintain enterprise data warehouses and production pipelines.
  • Implement real-time, event-driven data ingestion and processing.
  • Build and optimize data models, transformations, and data quality controls.
  • Ensure data governance, lineage, and access controls across domains.
  • Collaborate with Data Architecture, Analytics Engineering, QA, and Business stakeholders.
  • Mentor and review work of Data Engineers; promote best practices.

Skills

SQL
Python
PySpark
Data modeling
Data governance
Problem solving
Cross-team collaboration
Communication
Mentoring
Analytics

Tools

Azure Databricks
Delta Lake
Unity Catalog
Azure Data Factory
Azure Data Lake Storage
Kafka
AWS Kinesis
Confluent Cloud
Azure DevOps

Job description

Job Title:Senior Data Engineer, Azure Databricks

Location:Fully Remote (on-site in Dallas, Memphis, Cleveland, or Pittsburgh is optional)

Employment Status:Applicants must be currently authorized to work in the U.S. without the need for employment-based sponsorship, now or in the future, including: H-1B, L-1, TN, O-1, E-3, H-1B1, F-1, J-1, OPT, CPT, or any other employment-based visa programs.

Position Overview

The Senior Data Engineer will design, develop, and optimize enterprise data warehouse solutions and production data pipelines within an Azure Databricks environment. This hands-on position will support the continued development and modernization of the data ecosystem including the transition toward real-time, event-driven data processing, while ensuring data quality, reliability, and performance in a highly regulated environment.

Key Responsibilities
  • Design, develop, and maintain enterprise data warehouse solutions and production pipelines across Bronze, Silver, and Gold layers using Azure Databricks, Delta Lake, and medallion architecture.
  • Develop and optimize data ingestion, transformation, and modeling solutions using SQL, Python, PySpark, and Azure Data Factory.
  • Build dimensional and harmonized data models, incorporating data conformance, survivorship logic, and standardized transformation processes.
  • Support the transition from traditional batch processing toward continuous, low-latency, event-driven data ingestion using Azure Event Hubs, Structured Streaming, and emerging Databricks capabilities.
  • Own assigned production data domains, ensuring pipeline reliability, data integrity, performance, and compliance with established service-level requirements.
  • Monitor and optimize pipeline performance, processing latency, scalability, and Databricks compute costs.
  • Troubleshoot complex production issues, perform root-cause analysis, and implement sustainable improvements to data reliability and system performance.
  • Implement automated data validation, reconciliation, quality controls, and testing practices throughout the data lifecycle.
  • Support data governance initiatives involving Unity Catalog, metadata management, lineage, access controls, and Gold-layer data assets.
  • Modernize and consolidate legacy transformation processes while maintaining data accuracy and operational continuity.
  • Establish and maintain engineering best practices for code quality, testing, documentation, version control, and CI/CD deployment within a highly regulated environment.
  • Collaborate with Data Architecture, Analytics Engineering, QA, Data Governance, and business stakeholders to deliver reliable, scalable data solutions.
  • Develop and enhance internal and external partner data integrations to support evolving business and operational requirements.
  • Provide technical guidance and mentorship to Data Engineers through code reviews, knowledge sharing, and engineering best practices.
Qualifications
  • 6+ years of progressive data engineering experience including hands-on development and production support of enterprise data platforms.
  • Deep expertise in Azure Databricks, Delta Lake, Medallion Architecture, Unity Catalog, and production pipeline development and optimization.
  • Hands-on experience developing and operating event-driven streaming data pipelines using Azure Event Hubs, Kafka, AWS Kinesis, or Confluent Cloud.
  • Strong enterprise data warehousing experience encompassing dimensional modeling, complex transformations, data integration, and production operations.
  • Advanced proficiency in SQL, Python, and PySpark, with the ability to independently develop, troubleshoot, review, and optimize complex data pipelines.
  • Hands-on experience with Azure Data Lake Storage, Azure Data Factory, and CI/CD practices using Azure DevOps.
  • Demonstrated ownership of production data pipelines, including monitoring, incident response, performance optimization, and service-level accountability.
  • Experience implementing data contracts using ODCS or equivalent standards preferred.
  • Experience implementing data validation, reconciliation, automated testing, and data quality controls.
  • Strong understanding of data governance, metadata management, lineage, and secure data handling practices.
  • Familiarity with data observability platforms such as Monte Carlo preferred.
  • Experience operating within highly regulated environments including healthcare, pharmacy, pharmaceutical, medical, or another highly regulated industry involving sensitive clinical or operational data is preferred.
  • Ability to analyze complex legacy data environments, improve existing architectures, and produce comprehensive technical documentation.
  • Familiarity with Structured Streaming, Zerobus Ingestion, and Spark Declarative Pipelines preferred.
  • Experience with enterprise data catalog platforms such as Atlan, Collibra, or equivalent technologies preferred.
  • Experience developing governed data assets with documented metadata, lineage, and access controls.
  • Strong communication, problem-solving, and cross-functional collaboration skills.
Get your free, confidential resume review.

or drag and drop your file here.

Similar jobs

Similar jobs worth comparing

Sr Azure Databricks Engineer
Sr Azure Databricks Engineer

ITMC Systems, Inc • United States

Remote
USD 140,000 - 190,000
Databricks Engineer
Databricks Engineer

Insight Global (United States) • United States

Remote
USD 140,000 - 190,000
Senior Data Engineer on-site)
Senior Data Engineer on-site)

Ziosk • Dallas (TX)

On-site
USD 140,000 - 190,000
Senior Data Engineer
Senior Data Engineer

HumCap • Plano (TX)

Hybrid
USD 120,000 - 170,000
Senior Data Software Engineer with Azure Databricks
Senior Data Software Engineer with Azure Databricks

EPAM Systems • United States

Remote
USD 140,000 - 170,000
Senior Software Engineer
Senior Software Engineer

Mphasis • New York (NY)

On-site
USD 60,000 - 118,000
Senior Data Engineer - Databricks
Senior Data Engineer - Databricks

UNAVAILABLE • McLean (VA)

On-site
USD 120,000 - 160,000
Data Engineer
Data Engineer

Ascendum Solutions • Cincinnati (OH)

Hybrid
USD 100,000 - 130,000
Senior Data Engineer
Senior Data Engineer

Accion Labs • United States

Remote
USD 120,000 - 170,000
Data Engineer – Azure Databricks
Data Engineer – Azure Databricks

HIGH BRIDGE CONSULTING LLC • New York (NY)

Hybrid
USD 120,000 - 170,000