Azure Data Engineer

SMX Services & Consulting, Inc.; United States; Remote; USD 110,000 - 150,000; Full time

Job summary

SMX Services & Consulting, Inc. is seeking an experienced Azure Data Engineer to design, build, and maintain scalable cloud data platforms.

You will own end-to-end pipelines from ingestion to serving analytics-ready datasets for BI and ML, collaborating with data scientists, analysts, and software engineers. The role emphasizes Azure Data Factory, Azure Databricks, Microsoft Fabric, ADLS Gen2 lakehouse, and governance.

Qualifications

• 3+ years of experience as a Data Engineer with 2+ years on Azure cloud.
• Strong SQL (T-SQL, window functions) and Python (PySpark, pandas).
• Hands-on with Azure Data Factory, Azure Databricks, and Microsoft Fabric.
• Solid understanding of Spark internals and Delta Lake.
• Familiarity with Git, CI/CD, orchestration, and cloud security basics.
• Excellent English communication for technical and stakeholder interactions.

Responsibilities

• Design, develop, and optimize batch and real-time data pipelines on Azure.
• Build a lakehouse on ADLS Gen2 following the medallion pattern (Bronze-Silver-Gold).
• Implement robust ETL/ELT processes for large-scale data in various formats.
• Develop streaming solutions with Azure Event Hubs, Kafka, and Spark Structured Streaming.
• Model data using dimensional modeling and data vault techniques.
• Tune queries, manage partitioning, indexing, caching, and cost optimization.
• Implement data quality, observability, lineage, and monitoring tools.
• Enforce data governance and security: RBAC, KVault, encryption, private endpoints.

Skills

SQL

Python

Azure Data Factory

Azure Databricks

Microsoft Fabric

Delta Lake

Spark

Git

CI/CD

Docker

Kubernetes

Airflow

MLOps

English communication

Tools

Azure Data Factory

Azure Databricks

Microsoft Fabric

Delta Lake

Terraform

Bicep

Azure DevOps

GitHub Actions

Airflow

Docker

Kubernetes/AKS

Job description

We are looking for an experienced Azure Data Engineer to design, build, and maintain scalable, secure, and high-performance data platforms in the cloud. You will own end-to-end data pipelines, from ingestion and transformation to serving curated datasets for analytics, BI, and machine learning. You will work closely with data scientists, analysts, and software engineers to turn raw data into reliable, production-grade assets.

Key Responsibilities

• Design, develop, and optimize batch and real-time data pipelines using Azure Data Factory, Microsoft Fabric, and Azure Databricks.
• Build and maintain a lakehouse architecture on Azure Data Lake Storage Gen2 (ADLS) following the medallion pattern (Bronze, Silver, Gold layers).
• Implement ETL/ELT processes that handle large-scale structured and semi-structured data (JSON, Parquet, Avro, Delta).
• Develop streaming solutions with Azure Event Hubs, Kafka, and Spark Structured Streaming.
• Model data using dimensional modeling (star and snowflake schemas) and data vault techniques.
• Tune query performance and manage partitioning, indexing, caching, and cost optimization across the platform.
• Implement data quality checks, observability, lineage, and monitoring using tools such as Azure Monitor, Log Analytics, and Microsoft Purview.
• Apply data governance and security best practices: RBAC, managed identities, Azure Key Vault, private endpoints, encryption at rest and in transit, and data masking.
• Build and maintain CI/CD pipelines with Azure DevOps or GitHub Actions, and manage infrastructure as code with Terraform or Bicep.
• Troubleshoot production incidents, perform root cause analysis, and participate in on-call rotations when needed.
• Document architecture, data flows, and operational runbooks; contribute to code reviews and engineering standards.

Required Qualifications

• 3+ years of professional experience as a Data Engineer, with at least 2 years working on the Azure cloud.
• Strong proficiency in SQL (T-SQL, window functions, CTEs, query optimization) and Python (PySpark, pandas).
• Hands‑on experience with Azure Data Factory, Azure Databricks, and Microsoft Fabric.
• Solid understanding of distributed computing concepts and Apache Spark internals (shuffles, partitions, broadcast joins, skew handling).
• Experience with Delta Lake (ACID transactions, time travel, schema evolution, MERGE operations).
• Knowledge of data warehousing concepts, ETL/ELT design patterns, and orchestration frameworks.
• Familiarity with Git, branching strategies, automated testing, and CI/CD practices.
• Ability to communicate clearly in English, both written and spoken, with technical and non-technical stakeholders.

Preferred Qualifications

• Microsoft certifications such as DP-700 (Fabric Data Engineer Associate)
• Experience with Microsoft Fabric, Azure Functions, Azure Stream Analytics, or Azure Cosmos DB.
• Experience with Airflow, or other orchestration and transformation tools.
• Exposure to MLOps workflows and feature stores.
• Knowledge of containerization ( Docker, Kubernetes/AKS).
• Experience working in Agile/Scrum environments.

Location: LATAM/ Remote |

Employment Type: Full-time/Contract |

Schedule Work: Monday - Friday 9am-6pm (AST) |

Languages: Bilingual with Technical English