Senior Data Engineer: Azure & Fabric Pipelines Lead

Datavail; United States; On-site; USD 140,000 - 190,000; Full time

Job summary

Datavail is seeking a senior data engineer to design and maintain scalable data pipelines across Microsoft Fabric and Azure. You will implement CDC, upserts and late-arriving data, and develop transformations with PySpark, Spark SQL and T-SQL across Bronze/Silver/Gold Delta Lake layers.

You will build Fabric solutions using OneLake, Lakehouse, Data Factory, Dataflows Gen2, Spark notebooks and Power BI, while applying dimensional models, data quality tests and observability.

Qualifications

• 8+ years in data engineering and data warehousing.
• 3+ years delivering Azure solutions in production.
• Deep Delta Lake, Spark tuning, CDC and incremental loading experience.
• Strong dimensional modeling and Data Vault 2.0 knowledge.
• Proficient in CI/CD, Git, automated testing, DataOps, governance and security.

Responsibilities

• Design, build and maintain production-grade batch, incremental, real-time pipelines across Microsoft Fabric and Azure.
• Implement CDC, watermarking, merge/upsert, late-arriving-data and idempotent reprocessing patterns.
• Develop PySpark, Spark SQL and T-SQL transformations across Delta Lake layers (bronze/silver/gold).
• Build Fabric solutions using OneLake, Lakehouse, Warehouse, Data Factory and Power BI.
• Apply dimensional/Data Vault models, data quality testing, observability and SLA monitoring.
• Establish Git-based CI/CD and automated deployment via Azure DevOps or GitHub; Fabric pipelines, REST APIs, Bicep, Terraform.
• Enforce Purview catalog/lineage, least-privilege access, encryption and secure connectivity.
• Support migrations from legacy SQL Server/SSIS/SSAS to Fabric/Azure; include cutover and reconciliation.
• Prepare governed data for AI/ML and RAG workloads; include ingestion, embeddings and telemetry.
• Lead engineering workstreams, estimates, risk management, client communication and mentoring.

Skills

Data engineering

Data warehousing

Azure solutions

Microsoft Fabric

Delta Lake

Spark PySpark

T-SQL

Dimensional modeling

Data Vault 2.0

CDC / upserts

CI/CD

Data governance

Client-facing

Tools

Power BI

Azure Data Factory

Git/GitHub

Terraform

Bicep

OneLake

Spark Notebooks

REST APIs

Job description

Datavail is seeking a senior data engineer to design and maintain scalable data pipelines across Microsoft Fabric and Azure. You will implement CDC, upserts and late-arriving data, and develop transformations with PySpark, Spark SQL and T-SQL across Bronze/Silver/Gold Delta Lake layers.

You will build Fabric solutions using OneLake, Lakehouse, Data Factory, Dataflows Gen2, Spark notebooks and Power BI, while applying dimensional models, data quality tests and observability.