Data Engineer - Databricks Lakehouse & Streaming (Remote)

AgileEngine; Ciudad de México; Presencial; MXN 600.000 - 850.000; Jornada completa

Ventajas ofrecidas por este puesto de trabajo

Growth opportunities

Competitive pay

Remote work

Modern projects

Collaborative culture

Well-being programs

Descripción de la vacante

AgileEngine is seeking a Middle Data Engineer to modernize a data warehouse to a Databricks Lakehouse. You will design batch and streaming pipelines with PySpark and Delta Lake across bronze, silver, and gold layers, leveraging AI tools to accelerate development.

The role emphasizes data governance, production-grade pipelines, and collaboration with DevOps and analytics teams. Remote work options are available.

Formación

• 3+ years of data engineering with Spark and cloud architectures.
• Strong hands-on with Databricks, Spark (PySpark) and Delta Lake.
• Advanced SQL and Python with Lakehouse data modeling.
• Streaming ingestion using Structured Streaming, Auto Loader, Kafka or Event Hubs.
• Workflow orchestration with Databricks Workflows, Airflow or Azure Data Factory.
• Experience migrating legacy ETL and modernizing data warehouses.
• Strong problem-solving, collaboration, and communication.
• Familiarity with Unity Catalog, data governance and PII handling.
• Experience with dbt or comparable transformation framework.
• Secure coding practices and security standards.
• Experience delivering production data platforms at scale.
• Upper-intermediate English level.

Responsabilidades

• Design, build, and operate batch and streaming pipelines on Databricks.
• Model and maintain a medallion architecture for analytics and ML.
• Migrate workloads to Lakehouse with data parity and minimal disruption.
• Utilize Claude or Copilot to accelerate development and testing.
• Write clean Python and SQL with thorough code reviews.
• Optimize Spark jobs and Delta tables for performance and cost.
• Implement data quality, lineage, and governance using Unity Catalog.
• Debug and resolve pipeline failures and data defects.
• Collaborate with DevOps, platform, and analytics teams on observability and security.

Conocimientos

Databricks

PySpark

Spark

Delta Lake

SQL

Python

Structured Streaming

Kafka

Event Hubs

Airflow

Databricks Workflows

Azure Data Factory

Unity Catalog

dbt

Security practices

English Proficiency

Production data platforms

Herramientas

Terraform

CloudWatch

PostgreSQL

Descripción del empleo

AgileEngine is seeking a Middle Data Engineer to modernize a data warehouse to a Databricks Lakehouse. You will design batch and streaming pipelines with PySpark and Delta Lake across bronze, silver, and gold layers, leveraging AI tools to accelerate development.

The role emphasizes data governance, production-grade pipelines, and collaboration with DevOps and analytics teams. Remote work options are available.