Senior Data Engineer

Datavail; United States; On-site; USD 140,000 - 190,000; Full time

Job summary

Datavail is seeking a senior data engineer to design and maintain scalable data pipelines across Microsoft Fabric and Azure. You will implement CDC, upserts and late-arriving data, and develop transformations with PySpark, Spark SQL and T-SQL across Bronze/Silver/Gold Delta Lake layers.

You will build Fabric solutions using OneLake, Lakehouse, Data Factory, Dataflows Gen2, Spark notebooks and Power BI, while applying dimensional models, data quality tests and observability.

Qualifications

• 8+ years in data engineering and data warehousing.
• 3+ years delivering Azure solutions in production.
• Deep Delta Lake, Spark tuning, CDC and incremental loading experience.
• Strong dimensional modeling and Data Vault 2.0 knowledge.
• Proficient in CI/CD, Git, automated testing, DataOps, governance and security.

Responsibilities

• Design, build and maintain production-grade batch, incremental, real-time pipelines across Microsoft Fabric and Azure.
• Implement CDC, watermarking, merge/upsert, late-arriving-data and idempotent reprocessing patterns.
• Develop PySpark, Spark SQL and T-SQL transformations across Delta Lake layers (bronze/silver/gold).
• Build Fabric solutions using OneLake, Lakehouse, Warehouse, Data Factory and Power BI.
• Apply dimensional/Data Vault models, data quality testing, observability and SLA monitoring.
• Establish Git-based CI/CD and automated deployment via Azure DevOps or GitHub; Fabric pipelines, REST APIs, Bicep, Terraform.
• Enforce Purview catalog/lineage, least-privilege access, encryption and secure connectivity.
• Support migrations from legacy SQL Server/SSIS/SSAS to Fabric/Azure; include cutover and reconciliation.
• Prepare governed data for AI/ML and RAG workloads; include ingestion, embeddings and telemetry.
• Lead engineering workstreams, estimates, risk management, client communication and mentoring.

Skills

Data engineering

Data warehousing

Azure solutions

Microsoft Fabric

Delta Lake

Spark PySpark

T-SQL

Dimensional modeling

Data Vault 2.0

CDC / upserts

CI/CD

Data governance

Client-facing

Tools

Power BI

Azure Data Factory

Git/GitHub

Terraform

Bicep

OneLake

Spark Notebooks

REST APIs

Job description

• 8+ years in data engineering/data warehousing; 3+ years delivering Azure solutions and production
• Microsoft Fabric experience
• Advanced T-SQL, Spark SQL, Python/PySpark; working knowledge of KQL and DAX
• Deep experience with Delta Lake, Spark tuning, medallion/lakehouse patterns, CDC and incremental loading
• Strong dimensional modeling and working knowledge of Data Vault 2.0
• Practical CI/CD, Git, automated testing, infrastructure as code, DataOps, governance and security experience
• Professional-services/client-facing delivery experience
• Authorization to work in the US and ability to travel up to 15%
• Preferred: DP-700, DP-600, DP-203; streaming/event-driven, RAG/GenAI, Fabric IQ/data agents, and industry experience
• Design, build and maintain production-grade batch, incremental, real-time and near-real-time data pipelines across Microsoft Fabric and Azure
• Implement CDC, watermarking, merge/upsert, late-arriving-data and idempotent reprocessing patterns
• Develop PySpark, Spark SQL and T-SQL transformations across bronze, silver and gold Delta Lake layers
• Build Fabric solutions using OneLake, Lakehouse, Warehouse, Data Factory, Dataflows Gen2, Spark notebooks, Real-Time Intelligence and Power BI
• Implement dimensional and Data Vault models, data-quality testing, reconciliation, observability, logging, alerting and SLA monitoring
• Establish Git-based CI/CD and automated deployment using Azure DevOps or GitHub, Fabric deployment pipelines, REST APIs, Bicep and Terraform
• Apply Purview catalog/lineage, least-privilege access, sensitivity controls, encryption and secure connectivity
• Support migrations from SQL Server, SSIS, SSAS, legacy warehouses and Synapse to Fabric/Azure, including conversion, reconciliation and cutover
• Prepare governed data for AI/ML and RAG workloads, including ingestion, chunking, embeddings, indexing and telemetry
• Lead engineering workstreams, estimates, risk management, client communication, mentoring, reusable accelerators and operational handover