Senior Data Engineer - Azure Fabric, Delta Lake & DataOps Lead

Datavail; Oregon; Hybrid; USD 120,000 - 180,000; Full time

Job summary

Datavail is seeking an experienced data engineer to design and operate production-grade data pipelines across Microsoft Fabric and Azure. You will implement CDC, upserts, and real-time processing while developing PySpark, Spark SQL and T-SQL transformations across Delta Lake layers.

The role involves dimensional modeling, Data Vault patterns, and end-to-end data governance with Git-based CI/CD and IaC pipelines.

Qualifications

• 8+ years in data engineering/data warehousing (senior level)
• Microsoft Fabric experience
• Advanced T-SQL, Spark SQL, Python/PySpark
• Working knowledge of KQL and DAX
• Delta Lake, Spark tuning, medallion/lakehouse patterns, CDC and incremental loading
• Dimensional modeling and Data Vault 2.0
• CI/CD, Git, automated testing, IaC, DataOps, governance and security
• Client-facing delivery experience
• US work authorization and travel up to 15%

Responsibilities

• Design, build and maintain production-grade data pipelines across Microsoft Fabric and Azure
• Implement CDC, watermarking, merge/upsert, late-arriving-data and idempotent reprocessing patterns
• Develop PySpark, Spark SQL and T-SQL transformations across Delta Lake layers
• Build Fabric solutions using OneLake, Lakehouse, Warehouse, Data Factory, Dataflows Gen2, Spark notebooks, Real-Time Intelligence and Power BI
• Implement dimensional and Data Vault models, data-quality testing, reconciliation, observability, logging, alerting and SLA monitoring
• Establish Git-based CI/CD and automated deployment using Azure DevOps or GitHub, Fabric deployment pipelines, REST APIs, Bicep and Terraform
• Apply Purview catalog/lineage, least-privilege access, sensitivity controls, encryption and secure connectivity
• Support migrations from legacy warehouses and SQL Server ecosystems to Fabric/Azure, including cutover
• Prepare governed data for AI/ML and RAG workloads, including ingestion and indexing
• Lead engineering workstreams, estimates, risk management, client communication, mentoring and handover

Tools

Git

Azure DevOps

GitHub

Terraform

Bicep

Job description

Datavail is seeking an experienced data engineer to design and operate production-grade data pipelines across Microsoft Fabric and Azure. You will implement CDC, upserts, and real-time processing while developing PySpark, Spark SQL and T-SQL transformations across Delta Lake layers.

The role involves dimensional modeling, Data Vault patterns, and end-to-end data governance with Git-based CI/CD and IaC pipelines.