Senior Data Engineer

Datavail; United States; On-site; USD 150,000 - 210,000; Full time

Job summary

Datavail is seeking a senior data engineer to design and optimize end-to-end data pipelines on Microsoft Fabric and Azure. You will implement CDC, upserts, and real-time processing while ensuring data quality and observability.

Ideal candidates have 8+ years in data warehousing, Delta Lake mastery, and strong CI/CD experience with GitHub or Azure DevOps. Travel up to 15% is required for client delivery and engagements.

Qualifications

• 8+ years in data engineering and data warehousing with production Azure experience.
• Strong Delta Lake, medallion/lakehouse patterns, and CDC loading knowledge.
• Professional-services or client-facing delivery experience required.
• Authorization to work in the US and willingness to travel up to 15%.

Responsibilities

• Design, build and maintain production-grade data pipelines across Microsoft Fabric and Azure.
• Implement CDC, watermarking, merge/upsert, late-arriving-data and idempotent reprocessing patterns.
• Develop PySpark, Spark SQL and T-SQL transformations across Delta Lake layers.
• Lead engineering workstreams and client communication, mentoring, and reusable accelerators.

Skills

Data engineering

Azure solutions

Data pipelines

CI/CD

Data governance

Security

Client-facing delivery

Tools

Git & GitHub

Azure DevOps

Terraform

Bicep

Purview

Power BI

Data Factory

Spark notebooks

Delta Lake

OneLake

Lakehouse

DAX

KQL

Job description

Job Description

• 8+ years in data engineering/data warehousing; 3+ years delivering Azure solutions and production
• Microsoft Fabric experience
• Advanced T-SQL, Spark SQL, Python/PySpark; working knowledge of KQL and DAX
• Deep experience with Delta Lake, Spark tuning, medallion/lakehouse patterns, CDC and incremental loading
• Strong dimensional modeling and working knowledge of Data Vault 2.0
• Practical CI/CD, Git, automated testing, infrastructure as code, DataOps, governance and security experience
• Authorization to work in the US and ability to travel up to 15%
• Preferred: DP-700, DP-600, DP-203; streaming/event-driven, RAG/GenAI, Fabric IQ/data agents, and industry experience
• Design, build and maintain production-grade batch, incremental, real-time and near-real-time data pipelines across Microsoft Fabric and Azure
• Implement CDC, watermarking, merge/upsert, late-arriving-data and idempotent reprocessing patterns
• Develop PySpark, Spark SQL and T-SQL transformations across bronze, silver and gold Delta Lake layers
• Build Fabric solutions using OneLake, Lakehouse, Warehouse, Data Factory, Dataflows Gen2, Spark notebooks, Real-Time Intelligence and Power BI
• Implement dimensional and Data Vault models, data-quality testing, reconciliation, observability, logging, alerting and SLA monitoring
• Establish Git-based CI/CD and automated deployment using Azure DevOps or GitHub, Fabric deployment pipelines, REST APIs, Bicep and Terraform
• Apply Purview catalog/lineage, least-privilege access, sensitivity controls, encryption and secure connectivity
• Support migrations from SQL Server, SSIS, SSAS, legacy warehouses and Synapse to Fabric/Azure, including conversion, reconciliation and cutover
• Prepare governed data for AI/ML and RAG workloads, including ingestion, chunking, embeddings, indexing and telemetry
• Lead engineering workstreams, estimates, risk management, client communication, mentoring, reusable accelerators and operational handover

Job Description

• 8+ years in data engineering/data warehousing; 3+ years delivering Azure solutions and production
• Microsoft Fabric experience
• Advanced T-SQL, Spark SQL, Python/PySpark; working knowledge of KQL and DAX
• Deep experience with Delta Lake, Spark tuning, medallion/lakehouse patterns, CDC and incremental loading
• Strong dimensional modeling and working knowledge of Data Vault 2.0
• Practical CI/CD, Git, automated testing, infrastructure as code, DataOps, governance and security experience
• Professional-services/client-facing delivery experience
• Authorization to work in the US and ability to travel up to 15%
• Preferred: DP-700, DP-600, DP-203; streaming/event-driven, RAG/GenAI, Fabric IQ/data agents, and industry experience
• Design, build and maintain production-grade batch, incremental, real-time and near-real-time data pipelines across Microsoft Fabric and Azure
• Implement CDC, watermarking, merge/upsert, late-arriving-data and idempotent reprocessing patterns
• Develop PySpark, Spark SQL and T-SQL transformations across bronze, silver and gold Delta Lake layers
• Build Fabric solutions using OneLake, Lakehouse, Warehouse, Data Factory, Dataflows Gen2, Spark notebooks, Real-Time Intelligence and Power BI
• Implement dimensional and Data Vault models, data-quality testing, reconciliation, observability, logging, alerting and SLA monitoring
• Establish Git-based CI/CD and automated deployment using Azure DevOps or GitHub, Fabric deployment pipelines, REST APIs, Bicep and Terraform
• Apply Purview catalog/lineage, least-privilege access, sensitivity controls, encryption and secure connectivity
• Support migrations from SQL Server, SSIS, SSAS, legacy warehouses and Synapse to Fabric/Azure, including conversion, reconciliation and cutover
• Prepare governed data for AI/ML and RAG workloads, including ingestion, chunking, embeddings, indexing and telemetry
• Lead engineering workstreams, estimates, risk management, client communication, mentoring, reusable accelerators and operational handover

About Us

Datavail is a leading provider of data management, application development, analytics, and cloud services, with more than 1,000 professionals helping clients build and manage applications and data via a world-class tech-enabled delivery platform and software solutions across all leading technologies. For more than 17 years, Datavail has worked with thousands of companies spanning different industries and sizes, and is an AWS Advanced Tier Consulting Partner, a Microsoft Solutions Partner for Data & AI and Digital & App Innovation (Azure), an Oracle Partner, and a MySQL Partner.