Senior Data Engineer: Scalable Pipelines on Azure Databricks

Applaudo; Mexico; On-site; MXN 900,000 - 1,200,000; Full time

Job summary

Applaudo is seeking a Senior Data Engineer to design, build, and maintain scalable data pipelines using Azure Databricks and Python. You will ingest data from APIs, databases, cloud storage, and SFTP sources, and integrate data across CRM, ticketing, marketing, and retail platforms into a centralized Data Hub.

The role emphasizes data warehousing, dimensional modeling, and ensuring data quality and consistency across source and target systems for enterprise-scale analytics.

Qualifications

• 5+ years of professional experience in Data Engineering, designing, building, and maintaining production-grade data pipelines.
• Strong hands-on experience with Azure Databricks.
• Advanced Python skills for data processing, transformation, and pipeline development.
• Solid experience with Apache Spark, including PySpark and/or Spark SQL, for processing data at scale.
• Advanced SQL proficiency, including complex transformations, data validation, and query performance optimization.
• Strong knowledge of data warehousing and dimensional modeling, including facts, dimensions, and data marts.
• Experience integrating data from multiple sources, including REST APIs, relational databases, cloud storage, and SFTP.
• Hands-on experience with Microsoft Azure data services for data ingestion, storage, processing, and integration.
• Experience implementing data quality checks and validating consistency between source systems and target data models.

Responsibilities

• Design, build, and maintain reliable, production-grade data pipelines using Azure Databricks and Python.
• Develop scalable data transformations using PySpark or Spark SQL to integrate data from multiple business systems.
• Ingest and process data from APIs, relational databases, cloud storage, and SFTP sources.
• Integrate data from ticketing, CRM, marketing, retail, and other enterprise platforms into a centralized Data Hub.
• Develop and maintain data warehouse structures, dimensional models, and data marts aligned with business requirements.
• Write, optimize, and maintain SQL transformations to ensure data accuracy and processing efficiency.
• Implement data quality validations and consistency checks between source systems and target models.
• Build and maintain outbound data feeds for third-party systems according to defined specifications.
• Investigate data discrepancies, troubleshoot pipeline issues, and implement appropriate corrective actions.
• Document data pipelines, transformation logic, dependencies, and technical processes.
• Collaborate with team members and stakeholders to clarify requirements, identify risks early, and deliver reliable data solutions.
• Communicate progress, blockers, and potential delays proactively, proposing clear next steps to keep deliverables on track.

Skills

Data engineering

Azure Databricks

Python

Apache Spark

SQL

Tools

Azure Databricks

PySpark

Spark SQL

SQL

REST APIs

SFTP

Azure data services

Job description

Applaudo is seeking a Senior Data Engineer to design, build, and maintain scalable data pipelines using Azure Databricks and Python. You will ingest data from APIs, databases, cloud storage, and SFTP sources, and integrate data across CRM, ticketing, marketing, and retail platforms into a centralized Data Hub.

The role emphasizes data warehousing, dimensional modeling, and ensuring data quality and consistency across source and target systems for enterprise-scale analytics.