Senior Data Engineer - Azure Databricks & Medallion Pipeline

Papigen; México; On-site; MXN 900,000 - 1,500,000; Full time

Job summary

Papigen is seeking a Senior Data Engineer / Databricks Engineer to design, develop, and optimize enterprise data pipelines using Azure Databricks, ADLS, ADF, PostgreSQL, and Azure DevOps in Mexico City. The role focuses on building ingestion, transformation, and reporting-ready datasets, implementing Medallion architecture, data quality controls, and CI/CD processes in a production environment.

You will collaborate with architects, BI teams, QA, and business analysts to ensure robust pipelines

Qualifications

• 8+ years of IT experience with data engineering focus.
• Azure Databricks notebooks, transformations, and pipeline development.
• ADLS storage and data lake architectures.
• PySpark, Python, and SQL proficiency for data pipelines.
• ETL/ELT development and data integration with strong data quality controls.

Responsibilities

• Design, develop, and maintain ETL/ELT pipelines for data ingestion, transformation, and integration.
• Develop Databricks notebooks and reusable data engineering components using PySpark and SQL.
• Implement Medallion Architecture (Bronze, Silver, Gold).
• Integrate data sources across operational, financial, accounting, portfolio, risk, and enterprise domains.
• Create curated analytical datasets and data models for reporting and analytics.
• Implement data-quality checks, validation rules, and reconciliation controls.
• Tune performance of Databricks workloads and PostgreSQL data processing.
• Monitor scheduled data loads and troubleshoot pipeline issues in production.
• Support data lineage, metadata, and source-to-target mapping; maintain runbooks.

Skills

ETL/ELT development

Data modeling

SQL proficiency

Data quality and reconciliation

Production troubleshooting

Tools

Azure Databricks

Azure Data Lake Storage

PySpark

Python

PostgreSQL

Azure Data Factory

Azure DevOps

Git

Job description

Papigen is seeking a Senior Data Engineer / Databricks Engineer to design, develop, and optimize enterprise data pipelines using Azure Databricks, ADLS, ADF, PostgreSQL, and Azure DevOps in Mexico City. The role focuses on building ingestion, transformation, and reporting-ready datasets, implementing Medallion architecture, data quality controls, and CI/CD processes in a production environment.

You will collaborate with architects, BI teams, QA, and business analysts to ensure robust pipelines