Senior Databricks Data Engineer - Remote Lakehouse Excellence

AgileEngine, LLC.; Puebla de Zaragoza; Presencial; MXN 900.000 - 1.600.000; Jornada completa

Ventajas ofrecidas por este puesto de trabajo

Growth without limits

Competitive compensation

Remote work with flexible hours

Modern projects with global teams

Collaborative culture

Well-being & support

Descripción de la vacante

AgileEngine seeks a Senior Data Engineer to modernize a 15-year-old data warehouse into a governed Databricks Lakehouse. You will build batch and streaming pipelines with PySpark and Delta Lake, following a medallion architecture across bronze, silver, and gold layers.

This role also uses AI tools like Claude and GitHub Copilot to speed up development. You will collaborate with DevOps, platform, and analytics engineers on observability, security, and governance while mentoring junior engineers

Formación

• 4+ years of professional data engineering experience with Spark and cloud data architectures.
• Hands-on with Databricks, PySpark, and Delta Lake.
• Advanced SQL and Python with strong data modeling skills across lakehouse patterns.
• Experience with streaming ingestion using Structured Streaming, Auto Loader, Kafka, or Event Hubs.
• Experience with workflow orchestration (Databricks Workflows, Airflow, or Azure Data Factory).
• Experience migrating legacy ETL or data warehouse workloads to a Lakehouse.

Responsabilidades

• Design, build, and operate batch and streaming data pipelines on Databricks using PySpark, Delta Lake, and Databricks Workflows.
• Model and maintain a medallion architecture (bronze/silver/gold) for analytics, reporting, and ML consumers.
• Migrate legacy ETL workloads to the Lakehouse with validated data parity and minimal disruption.
• Leverage Claude or Github Copilot to accelerate development and generate tests/docs.
• Write clean Python and SQL and maintain high quality with reviews and tests.
• Optimize Spark jobs and Delta tables for performance and cost (partitioning, clustering, caching).
• Implement data quality, lineage, and governance using Unity Catalog and automated checks.
• Debug and resolve pipeline failures and production incidents.
• Participate in Agile/product delivery and collaborate with DevOps and analytics engineers.

Conocimientos

Apache Spark

PySpark

Delta Lake

Databricks

SQL

Python

Structured Streaming

Kafka

Data modeling

Unity Catalog

Herramientas

Databricks Workflows

Airflow

Azure Data Factory

dbt

Terraform

Descripción del empleo

AgileEngine seeks a Senior Data Engineer to modernize a 15-year-old data warehouse into a governed Databricks Lakehouse. You will build batch and streaming pipelines with PySpark and Delta Lake, following a medallion architecture across bronze, silver, and gold layers.

This role also uses AI tools like Claude and GitHub Copilot to speed up development. You will collaborate with DevOps, platform, and analytics engineers on observability, security, and governance while mentoring junior engineers