Azure Data Engineer: Lakehouse & Real-Time Pipelines

SMX Services & Consulting, Inc.; United States; Remote; USD 110,000 - 150,000; Full time

Job summary

SMX Services & Consulting, Inc. is seeking an experienced Azure Data Engineer to design, build, and maintain scalable cloud data platforms.

You will own end-to-end pipelines from ingestion to serving analytics-ready datasets for BI and ML, collaborating with data scientists, analysts, and software engineers. The role emphasizes Azure Data Factory, Azure Databricks, Microsoft Fabric, ADLS Gen2 lakehouse, and governance.

Qualifications

• 3+ years of experience as a Data Engineer with 2+ years on Azure cloud.
• Strong SQL (T-SQL, window functions) and Python (PySpark, pandas).
• Hands-on with Azure Data Factory, Azure Databricks, and Microsoft Fabric.
• Solid understanding of Spark internals and Delta Lake.
• Familiarity with Git, CI/CD, orchestration, and cloud security basics.
• Excellent English communication for technical and stakeholder interactions.

Responsibilities

• Design, develop, and optimize batch and real-time data pipelines on Azure.
• Build a lakehouse on ADLS Gen2 following the medallion pattern (Bronze-Silver-Gold).
• Implement robust ETL/ELT processes for large-scale data in various formats.
• Develop streaming solutions with Azure Event Hubs, Kafka, and Spark Structured Streaming.
• Model data using dimensional modeling and data vault techniques.
• Tune queries, manage partitioning, indexing, caching, and cost optimization.
• Implement data quality, observability, lineage, and monitoring tools.
• Enforce data governance and security: RBAC, KVault, encryption, private endpoints.

Skills

SQL

Python

Azure Data Factory

Azure Databricks

Microsoft Fabric

Delta Lake

Spark

Git

CI/CD

Docker

Kubernetes

Airflow

MLOps

English communication

Tools

Azure Data Factory

Azure Databricks

Microsoft Fabric

Delta Lake

Terraform

Bicep

Azure DevOps

GitHub Actions

Airflow

Docker

Kubernetes/AKS

Job description

SMX Services & Consulting, Inc. is seeking an experienced Azure Data Engineer to design, build, and maintain scalable cloud data platforms.

You will own end-to-end pipelines from ingestion to serving analytics-ready datasets for BI and ML, collaborating with data scientists, analysts, and software engineers. The role emphasizes Azure Data Factory, Azure Databricks, Microsoft Fabric, ADLS Gen2 lakehouse, and governance.