Senior Data Engineer (Databricks + Pyspark + Adf + Unity Catalog)

Mps Group; Xico; Presencial; MXN 600.000 - 900.000; Jornada completa

Descripción de la vacante

Mps Group is seeking a Senior Data Engineer to design and maintain scalable data pipelines in Azure, with hands-on Databricks, Unity Catalog, and ADF expertise. You will collaborate with architects and business stakeholders to deliver secure, governed data platforms and cloud-based analytics.

The role involves building Delta Lake architectures, implementing CI/CD for data products, and ensuring data quality, security, and compliance across enterprise workloads, while continuously improving

Formación

• 4–7+ years of data engineering or big data experience.
• Hands-on Databricks, Unity Catalog, and Azure Data Factory experience.
• Proficient with PySpark, Spark SQL, and Delta Lake.
• Solid understanding of data modeling and data warehousing concepts.
• Experience with Azure cloud services and governance.

Responsabilidades

• Design, develop, and optimize ETL/ELT pipelines using Azure Data Factory and Databricks.
• Build scalable data ingestion frameworks for structured and unstructured data sources.
• Develop data transformation processes using PySpark, Spark SQL, and Databricks notebooks.
• Implement and manage Unity Catalog for enterprise data governance, security, and access controls.
• Create and optimize Delta Lake architectures for analytics workloads.
• Monitor pipeline performance and troubleshoot production issues.
• Collaborate with data architects to implement enterprise data models and best practices.
• Ensure data quality, integrity, security, and compliance standards are met.
• Support CI/CD deployment processes and DevOps for data engineering.
• Work with business users to deliver scalable data solutions.

Conocimientos

Azure Databricks

PySpark

Spark SQL

Unity Catalog

Azure Data Factory

Delta Lake

Data modeling

ETL/ELT

Azure cloud

Git / CI/CD

Descripción del empleo

Overview

We are seeking a highly skilled Senior Data Engineer with hands-on experience in Databricks, Unity Catalog, and Azure Data Factory (ADF) to support a large-scale enterprise data modernization initiative. The ideal candidate will be responsible for designing, developing, and maintaining scalable data pipelines, enabling secure data governance, and supporting cloud-based analytics solutions in Azure.

This role will collaborate closely with architects, business stakeholders, and cross-functional engineering teams to build reliable, high-performance data platforms and accelerate data-driven decision-making.

Responsibilities

Your Impact

• Design, develop, and optimize ETL/ELT pipelines using Azure Data Factory and Databricks.
• Build and maintain scalable data ingestion frameworks for structured and unstructured data sources.
• Develop data transformation processes using PySpark, Spark SQL, and Databricks notebooks.
• Implement and manage Unity Catalog for enterprise data governance, security, and access controls.
• Create and optimize Delta Lake architectures for analytics and reporting workloads.
• Monitor pipeline performance and troubleshoot production issues.
• Collaborate with data architects to implement enterprise data models and best practices.
• Ensure data quality, integrity, security, and compliance standards are met.
• Support CI/CD deployment processes and DevOps practices for data engineering solutions.
• Work closely with business and analytics teams to understand requirements and deliver scalable solutions.

Requirements

Your Skills & Experience

• 4-7+ years of experience in Data Engineering or Big Data development.
• Strong hands-on experience with Azure Databricks.
• Proficiency in PySpark, Spark SQL, and distributed data processing.
• Experience implementing and managing Unity Catalog.
• Strong experience with Azure Data Factory (ADF).
• Experience working with Delta Lake and modern Lakehouse architectures.
• Solid understanding of data modeling, ETL/ELT, and data warehousing concepts.
• Experience with Azure cloud services and ecosystem.
• Proficiency with Git, version control, and CI/CD processes.
• Strong analytical and problem-solving skills.
• Excellent communication and stakeholder management abilities.

Preferred Qualifications

• Experience with Azure Synapse Analytics.
• Familiarity with Collibra or other data governance platforms.
• Experience with Infrastructure as Code (Terraform, ARM Templates).
• Databricks Certification (Data Engineer Associate/Professional).
• Microsoft Azure Data Engineer Associate Certification.

Nice to Have

• Experience supporting enterprise-scale data migration projects.
• Knowledge of healthcare, retail, or consumer products data environments.
• Exposure to real-time data processing and streaming technologies.