Senior Data Engineer

NexTech Solutions; Pittsburgh (Allegheny County); On-site; USD 120,000 - 170,000; Full time

Job summary

NexTech Solutions is seeking a hands-on Senior Data Engineer to build, operate, and improve an enterprise data warehouse and production pipelines in an Azure Databricks environment. The role covers Bronze, Silver, and Gold layers and focuses on ingestion, transformation, modeling, quality, performance, and reliability.

The Senior Data Engineer will own production data domains, simplify legacy layers, support event-driven ingestion, and mentor other engineers as the platform evolves.

Qualifications

• Six+ years of progressive data engineering experience.
• Experience owning an enterprise data warehouse or large-scale data transformation pipelines in production with defined SLAs.
• Deep Databricks experience in production environments.
• Strong experience with Delta Lake, medallion architecture, Unity Catalog, and Databricks performance optimization.
• Advanced SQL skills and Python/PySpark proficiency.
• Ability to independently build, review, troubleshoot, and optimize complex data transformations.
• Strong data warehousing experience, including dimensional modeling and conformance rules.
• Hands-on Azure Databricks, Azure Data Lake Storage, Azure Data Factory, and CI/CD with Azure DevOps.

Responsibilities

• Design, build, and support enterprise data warehouse pipelines and data models across Bronze, Silver, and Gold layers in Databricks.
• Develop ingestion patterns and Bronze-to-Silver transformations for clean, standardized data.
• Create dimensional and harmonized data models with conformance and survivorship logic.
• Own assigned data domains in production, including data-model integrity and ingestion SLAs.
• Build and maintain Delta Lake pipelines and transition workloads to event-driven ingestion.
• Develop streaming ingestion patterns using Spark Structured Streaming.
• Implement data quality gates and data-contract validation across the lifecycle.
• Build Gold-layer tables for Unity Catalog views, lineage, and governance.
• Optimize pipeline performance, latency, reliability, and compute costs.
• Document architecture decisions, data models, and procedures.
• Review PRs and maintain code quality via Azure DevOps, Git, and CI/CD.

Skills

SQL

Python

PySpark

Databricks

CI/CD

Azure DevOps

Data modeling

Communication

Education

Bachelor's degree in Computer Science or related field

Tools

Delta Lake

Unity Catalog

Azure Databricks

Azure Data Lake Storage

Azure Data Factory

Spark Structured Streaming

Kafka

Job description

We are seeking a hands-on Senior Data Engineer to build, operate, and improve an enterprise data warehouse and production data pipelines in an Azure Databricks environment.

This role works across the Bronze, Silver, and Gold layers of a medallion architecture, with a strong focus on data ingestion, transformation, modeling, quality, performance, and reliability. The Senior Data Engineer will own assigned data domains as production systems, simplify legacy transformation layers, support the transition toward event-driven ingestion, and mentor other engineers as the data platform evolves.

Key Responsibilities

• Design, build, and support enterprise data warehouse pipelines and data models across Bronze, Silver, and Gold layers in Databricks.
• Develop ingestion patterns and Bronze-to-Silver transformations that produce clean, standardized, and reliable data.
• Create dimensional and harmonized data models, including conformance, survivorship, and Gold-layer promotion rules.
• Own assigned data domains in production, including data-model integrity, pipeline reliability, ingestion SLAs, incident response, and root-cause remediation.
• Build and maintain Delta Lake pipelines and help transition selected workloads from batch processing to event-driven, low-latency ingestion.
• Develop streaming and high-SLA ingestion patterns using technologies such as Zerobus Ingestion and Spark Declarative Pipelines.
• Implement row-level reconciliation, validation checkpoints, automated quality gates, and data-contract validation throughout the data lifecycle.
• Build Gold-layer tables that support Unity Catalog metric views, lineage tracking, governed access, and downstream analytics.
• Optimize pipeline performance, processing latency, platform reliability, and Databricks compute costs.
• Consolidate legacy transformation layers into a simplified target architecture while validating output accuracy.
• Monitor pipeline health, investigate anomalies, and resolve data-reliability incidents.
• Document architecture decisions, data models, transformation logic, feed SQL, and operational procedures.
• Review pull requests and maintain code quality through Azure DevOps, Git, and CI/CD practices.
• Mentor Data Engineers in Databricks development patterns, SQL, Python, PySpark, testing, and documentation.
• Collaborate directly with data governance, quality assurance, analytics engineering, and business-facing stakeholders to clarify requirements and resolve delivery issues.

Required Qualifications

• Six or more years of progressive data engineering experience.
• Experience owning an enterprise data warehouse or large-scale data transformation pipelines in production with defined SLAs.
• Deep, hands-on Databricks experience in production environments.
• Strong experience with Delta Lake, medallion architecture, Unity Catalog, and Databricks pipeline performance optimization.
• Advanced SQL skills and strong proficiency with Python and PySpark.
• Ability to independently build, review, troubleshoot, and optimize complex data transformations.
• Strong data warehousing experience, including dimensional modeling, harmonized data models, conformance rules, and survivorship logic.
• Hands-on experience with Azure Databricks, Azure Data Lake Storage, Azure Data Factory, and CI/CD practices using Azure DevOps.
• Experience operating production data pipelines, including monitoring, incident response, reliability management, and performance improvement.
• Ability to learn complex, under-documented data environments and create clear, durable technical documentation.
• Strong communication skills and the ability to work directly with technical, governance, quality, analytics, and business stakeholders.

Preferred Qualifications

• Healthcare, pharmacy, specialty-pharmacy, or other regulated-industry experience involving clinical or operational data.
• Production experience with event-driven ingestion using Azure Event Hubs, Kafka, or equivalent technology through Structured Streaming.
• Familiarity with Zerobus Ingestion and Spark Declarative Pipelines.
• Experience implementing data contracts using ODCS or an equivalent standard.
• Familiarity with pipeline-observability platforms such as Monte Carlo.
• Experience with enterprise data-catalog platforms such as Atlan, Collibra, or equivalent tools.
• Experience creating governed data-pipeline assets with documented metadata and lineage.
• Demonstrated technical leadership through mentoring, code review, standards development, and influence without formal management authority.
• Bachelors degree in Computer Science, Data Engineering, Information Systems, or a related discipline, or equivalent professional experience.