Senior Databricks Engineer — Lakehouse & Streaming

AgileEngine, LLC.; Rosarito; Presencial; MXN 2.063.000 - 3.095.000; Jornada completa

Ventajas ofrecidas por este puesto de trabajo

Professional growth

Competitive USD-based compensation

A selection of exciting projects

Flextime

Descripción de la vacante

AgileEngine in Mexico is seeking a Senior Data Engineer to build batch and streaming pipelines on Databricks using PySpark and Delta Lake. You will migrate legacy data workloads to a governed Lakehouse with Unity Catalog governance and model medallion architecture for analytics and ML.

Strong SQL and Python, experience with streaming ingestion, and collaboration with DevOps and analytics teams are essential. Competitive USD compensation and flexible work options are offered.

Formación

• 3+ years of professional data engineering experience.
• Hands-on with Apache Spark, PySpark, and Delta Lake.
• Strong SQL and Python skills with data modeling.
• Experience migrating ETL workloads to Lakehouse.
• Familiarity with Unity Catalog and data governance.
• Experience with streaming ingestion (Structured Streaming, Kafka, or Event Hubs).
• Experience with Databricks Workflows, Airflow, or Azure Data Factory.
• Secure coding practices and compliance awareness.

Responsabilidades

• Design, build, and operate batch and streaming data pipelines on Databricks using PySpark and Delta Lake.
• Model a medallion architecture for analytics and ML.
• Migrate legacy ETL workloads to Lakehouse with data parity.
• Use CI tools to accelerate development, tests, and docs.
• Write clean Python and SQL with code reviews.
• Optimize Spark jobs and Delta tables for performance and cost.
• Implement data quality, lineage, and governance using Unity Catalog.
• Debug pipeline issues and production incidents.
• Collaborate with DevOps and analytics engineers on observability and security.

Conocimientos

Databricks

PySpark

Delta Lake

SQL

Python

Lakehouse

Data Modeling

Structured Streaming

Kafka

Databricks Workflows

Unity Catalog

Governance

dbt

Herramientas

Airflow

Azure Data Factory

PostgreSQL

Descripción del empleo

AgileEngine in Mexico is seeking a Senior Data Engineer to build batch and streaming pipelines on Databricks using PySpark and Delta Lake. You will migrate legacy data workloads to a governed Lakehouse with Unity Catalog governance and model medallion architecture for analytics and ML.

Strong SQL and Python, experience with streaming ingestion, and collaboration with DevOps and analytics teams are essential. Competitive USD compensation and flexible work options are offered.