Senior Data Engineer

Datavail Infotech; United States; On-site; USD 130,000 - 170,000; Full time

Job summary

Datavail Infotech is seeking a senior data engineer to design, build and operate scalable data pipelines across Microsoft Fabric and Azure. You will work on batch, streaming and real-time data, implementing robust data quality and security practices.

You bring 8+ years in data engineering, deep SQL skills, PySpark, Delta Lake, and experience with Data Vault 2.0. Travel up to 15% and client-facing delivery are part of the role; familiarity with Fabric components and CI/CD pipelines is essential.

Qualifications

• 8+ years in data engineering and data warehousing.
• 3+ years delivering Azure solutions in production.
• Experience with Microsoft Fabric.

Responsibilities

• Design, build and maintain batch, incremental, real-time data pipelines across Fabric and Azure.
• Implement CDC, watermarking, merge/upsert and idempotent reprocessing patterns.
• Develop PySpark, Spark SQL and T-SQL transformations across Delta Lake layers.
• Build Fabric solutions with OneLake, Lakehouse, Data Factory and Power BI.
• Establish Git-based CI/CD with Azure DevOps or GitHub deployments.
• Apply Purview catalog/ lineage and security controls.
• Support migrations to Fabric/Azure ecosystems from SQL Server/SSIS/SSAS/Synapse.
• Lead engineering workstreams, estimates, risk management and client communication.

Skills

Data engineering

Data warehousing

Azure

Microsoft Fabric

PySpark

Spark SQL

T-SQL

Delta Lake

Data Vault 2.0

CI/CD

Git

DataOps

Governance

Security

Client-facing

Tools

GitHub

Azure DevOps

OneLake

Lakehouse

Data Factory

Dataflows Gen2

Terraform

Bicep

REST APIs

Purview

Job description

• 8+ years in data engineering/data warehousing; 3+ years delivering Azure solutions and production
• Microsoft Fabric experience
• Advanced T-SQL, Spark SQL, Python/PySpark; working knowledge of KQL and DAX
• Deep experience with Delta Lake, Spark tuning, medallion/lakehouse patterns, CDC and incremental loading
• Strong dimensional modeling and working knowledge of Data Vault 2.0
• Practical CI/CD, Git, automated testing, infrastructure as code, DataOps, governance and security experience
• Professional-services/client-facing delivery experience
• Authorization to work in the US and ability to travel up to 15%
• Preferred: DP-700, DP-600, DP-203; streaming/event-driven, RAG/GenAI, Fabric IQ/data agents, and industry experience
• Design, build and maintain production-grade batch, incremental, real-time and near-real-time data pipelines across Microsoft Fabric and Azure
• Implement CDC, watermarking, merge/upsert, late-arriving-data and idempotent reprocessing patterns
• Develop PySpark, Spark SQL and T-SQL transformations across bronze, silver and gold Delta Lake layers
• Build Fabric solutions using OneLake, Lakehouse, Warehouse, Data Factory, Dataflows Gen2, Spark notebooks, Real-Time Intelligence and Power BI
• Implement dimensional and Data Vault models, data-quality testing, reconciliation, observability, logging, alerting and SLA monitoring
• Establish Git-based CI/CD and automated deployment using Azure DevOps or GitHub, Fabric deployment pipelines, REST APIs, Bicep and Terraform
• Apply Purview catalog/lineage, least-privilege access, sensitivity controls, encryption and secure connectivity
• Support migrations from SQL Server, SSIS, SSAS, legacy warehouses and Synapse to Fabric/Azure, including conversion, reconciliation and cutover
• Prepare governed data for AI/ML and RAG workloads, including ingestion, chunking, embeddings, indexing and telemetry
• Lead engineering workstreams, estimates, risk management, client communication, mentoring, reusable accelerators and operational handover