Data Engineer: Databricks Lakehouse & AI-Driven Pipelines

AgileEngine; Monterrey; Presencial; MXN 500.000 - 700.000; Jornada completa

Ventajas ofrecidas por este puesto de trabajo

Growth opportunities

Remote flexibility

Competitive compensation

Modern projects

Collaborative culture

Well-being programs

Descripción de la vacante

AgileEngine is seeking a Middle Data Engineer to modernize a data warehouse into a governed Databricks Lakehouse. You will build batch and streaming pipelines with PySpark and Delta Lake across bronze, silver, and gold layers.

You will leverage AI tools like Claude and Github Copilot to speed up development, write clean Python/SQL, and collaborate with DevOps and analytics teams on governance and security.

Formación

• 3+ years of data engineering experience with Spark and cloud data architectures.
• Expertise with Databricks, Spark (PySpark), and Delta Lake.
• Advanced SQL and Python and data modeling skills.
• Experience with streaming ingestion and structured streaming.
• Experience with Databricks Workflows, Airflow, or Azure Data Factory.
• Experience migrating legacy ETL to Lakehouse and data hygiene.
• Familiarity with Unity Catalog and data governance.
• dbt or equivalent transformation framework experience.
• Knowledge of secure coding standards and security best practices.
• Experience delivering production data platforms at scale.
• Upper-intermediate English level.

Responsabilidades

• Design, build, and operate batch and streaming pipelines on Databricks.
• Migrate legacy data workloads to a Databricks Lakehouse with minimal disruption.
• Model a medallion architecture for analytics, reporting, and ML consumers.
• Use Claude or Copilot to accelerate development, tests, and docs.
• Write clean Python and SQL, with strong code reviews and docs.
• Optimize Spark jobs and Delta tables for performance and cost.
• Implement data quality, lineage, and governance with Unity Catalog.
• Debug and resolve pipeline failures and production incidents.
• Collaborate with DevOps, platform, and analytics engineers on observability and security.

Conocimientos

Apache Spark (PySpark)

Delta Lake

SQL

Python

Structured Streaming

Databricks Workflows

Airflow

Azure Data Factory

Unity Catalog

dbt

Terraform

Azure DevOps

PostgreSQL

Herramientas

PostgreSQL

Descripción del empleo

AgileEngine is seeking a Middle Data Engineer to modernize a data warehouse into a governed Databricks Lakehouse. You will build batch and streaming pipelines with PySpark and Delta Lake across bronze, silver, and gold layers.

You will leverage AI tools like Claude and Github Copilot to speed up development, write clean Python/SQL, and collaborate with DevOps and analytics teams on governance and security.