Data Engineer - Databricks Lakehouse & Streaming Pipelines (Remote)

AgileEngine, LLC.; León; Presencial; MXN 1.529.000 - 2.209.000; Jornada completa

Ventajas ofrecidas por este puesto de trabajo

Growth without limits

Competitive compensation

Flexibility: 100% remote

Descripción de la vacante

AgileEngine is seeking a Middle Data Engineer to modernize a 15-year-old data warehouse into a governed Databricks Lakehouse. You will design batch and streaming pipelines with PySpark and Delta Lake, following a medallion architecture across bronze, silver, and gold layers.

Tools like Claude and GitHub Copilot speed up development. You will migrate legacy ETL workloads, ensure data parity, and collaborate with DevOps and analytics teams on governance, security, and observability while writing

Formación

• 3+ years of professional data engineering experience with Spark and cloud data architectures.
• Experience building data pipelines with Databricks, PySpark, and Delta Lake.
• Advanced SQL and Python with strong data modeling skills.

Responsabilidades

• Design, build, and operate batch and streaming data pipelines on Databricks using PySpark, Delta Lake, and Databricks Workflows.
• Model and maintain a medallion (bronze/silver/gold) architecture for analytics, reporting, and ML.
• Migrate legacy ETL and data warehouse workloads onto a Lakehouse with validated data parity and minimal disruption.
• Use Claude or Github Copilot to accelerate development, generate scaffolding, tests, docs, and prototypes.
• Write clean, well-tested Python and SQL; enforce coding standards through reviews and docs.
• Optimize Spark jobs and Delta tables for performance and cost (partitioning, caching, clustering, sizing).
• Implement data quality, lineage, and governance using Unity Catalog and automated checks.
• Debug and resolve pipeline failures, data defects, and production incidents.
• Collaborate with DevOps, platform and analytics engineers on observability, security, and compliance.

Conocimientos

Databricks

PySpark

Delta Lake

SQL

Python

Structured Streaming

Kafka

Event Hubs

Databricks Workflows

Airflow

Azure Data Factory

dbt

Unity Catalog

Secure coding

Herramientas

Databricks

PostgreSQL

Descripción del empleo

AgileEngine is seeking a Middle Data Engineer to modernize a 15-year-old data warehouse into a governed Databricks Lakehouse. You will design batch and streaming pipelines with PySpark and Delta Lake, following a medallion architecture across bronze, silver, and gold layers.

Tools like Claude and GitHub Copilot speed up development. You will migrate legacy ETL workloads, ensure data parity, and collaborate with DevOps and analytics teams on governance, security, and observability while writing