Senior Databricks Data Engineer - Remote & Lakehouse Impact

AgileEngine; Región Centro; Presencial; MXN 900.000 - 1.300.000; Jornada completa

Ventajas ofrecidas por este puesto de trabajo

Growth opportunities

Competitive compensation

Remote work with flexible hours

Modern tech stack

Mentorship & learning budget

Descripción de la vacante

AgileEngine is seeking a Senior Data Engineer to modernize a 15-year-old data warehouse into a governed Databricks Lakehouse. You will build batch and streaming pipelines with PySpark and Delta Lake, following a medallion architecture for analytics, reporting, and ML use cases.

You will also leverage AI tools like Claude and GitHub Copilot to speed up development while ensuring data quality, governance, and secure coding practices across production pipelines.

Formación

• Strong experience with Apache Spark (PySpark) and cloud data architectures.
• Advanced SQL and Python with data modeling across lakehouse patterns.
• Experience migrating ETL workloads to lakehouse environments.

Responsabilidades

• Design, build, and operate batch and streaming data pipelines on Databricks using PySpark, Delta Lake, and Databricks Workflows.
• Model and maintain a medallion architecture serving analytics, reporting, and ML consumers.
• Migrate legacy ETL and data warehouse workloads onto the Lakehouse with validated data parity.
• Use Claude or Github Copilot to accelerate development, tests, and documentation.
• Write clean, tested Python and SQL; review code and maintain documentation.
• Optimize Spark jobs and Delta tables for performance and cost.
• Implement data quality, lineage, and governance with Unity Catalog and validations.
• Debug and resolve pipeline failures and production incidents.
• Collaborate in Agile delivery and with DevOps, platform, and analytics teams.

Conocimientos

PySpark

SQL

Python

Data modeling

Cloud data

Educación

Bachelor's or Master's in CS/SE

Herramientas

Databricks

Delta Lake

Airflow

Azure Data Factory

Unity Catalog

Descripción del empleo

AgileEngine is seeking a Senior Data Engineer to modernize a 15-year-old data warehouse into a governed Databricks Lakehouse. You will build batch and streaming pipelines with PySpark and Delta Lake, following a medallion architecture for analytics, reporting, and ML use cases.

You will also leverage AI tools like Claude and GitHub Copilot to speed up development while ensuring data quality, governance, and secure coding practices across production pipelines.