Senior Data Engineer: Azure Databricks & PySpark Expert

Applaudo; Jalisco; On-site; MXN 900,000 - 1,300,000; Full time

Benefits offered by this job

Remote-possible? not specified

Job summary

Applaudo is seeking a Senior Data Engineer to design, build, and maintain scalable data pipelines using Azure Databricks and Python. You will transform data from multiple sources into reliable datasets and ensure data quality and consistency across the data lifecycle.

You will collaborate with a US-based team, own pipelines end-to-end, and document all processes while advocating for best engineering practices and clear communication.

Qualifications

• Proficiency in Azure Databricks, Python, and Apache Spark.
• Advanced SQL and data warehousing knowledge.
• Experience integrating data from multiple enterprise systems.

Responsibilities

• Design, build, and maintain production-grade data pipelines using Azure Databricks and Python.
• Develop scalable data transformations using PySpark or Spark SQL.
• Ingest data from APIs, relational databases, cloud storage, and SFTP.
• Integrate data from enterprise platforms into a centralized Data Hub.
• Maintain data warehouse structures and dimensional models.
• Write and optimize SQL transformations for accuracy and performance.
• Implement data quality validations and data consistency checks.
• Document data pipelines and technical processes.
• Collaborate with stakeholders to clarify requirements and manage risks.
• Proactively communicate progress and blockers.

Skills

Azure Databricks

Python

Apache Spark

Spark SQL

SQL

Data warehousing

Dimensional modeling

Data integration

Tools

PySpark

Azure Data Services

SFTP

REST APIs

Relational Databases

Cloud Storage

Job description

Applaudo is seeking a Senior Data Engineer to design, build, and maintain scalable data pipelines using Azure Databricks and Python. You will transform data from multiple sources into reliable datasets and ensure data quality and consistency across the data lifecycle.

You will collaborate with a US-based team, own pipelines end-to-end, and document all processes while advocating for best engineering practices and clear communication.