Senior Databricks Engineer: Remote Flex & Lakehouse Lead

AgileEngine, LLC.; Ciudad de México; Híbrido; MXN 1.547.000 - 2.579.000; Jornada completa

Ventajas ofrecidas por este puesto de trabajo

Professional growth

Competitive compensation (USD)

A selection of exciting projects

Flextime

Descripción de la vacante

AgileEngine is seeking a Senior Data Engineer to design and operate batch and streaming data pipelines on Databricks using PySpark and Delta Lake. You will migrate legacy ETL workloads to a governed Lakehouse supporting analytics and AI use cases.

You will collaborate with DevOps and analytics teams, implement data quality, governance with Unity Catalog, and optimize Spark jobs for performance and cost while maintaining strong tests and documentation.

Formación

• 3+ years of professional experience in data engineering with Spark and cloud data architectures.
• Strong SQL and Python, and experience with Databricks, Delta Lake and PySpark.

Responsabilidades

• Design, build, and operate batch and streaming data pipelines on Databricks using PySpark, Delta Lake, and Databricks Workflows.
• Model and maintain a medallion architecture serving analytics, reporting, and ML consumers.
• Migrate legacy ETL and data warehouse workloads onto the Lakehouse with validated data parity and minimal disruption.
• Write clean, tested Python and SQL; ensure quality through reviews and documentation.
• Collaborate with DevOps, platform, and analytics engineers on observability, security, and governance.

Conocimientos

3+ years data engineering

Apache Spark

PySpark

Delta Lake

SQL

Python

Structured Streaming

Kafka

Airflow

Databricks Workflows

Unity Catalog

dbt

Herramientas

Databricks

Airflow

Terraform

Azure Data Factory

PostgreSQL

Descripción del empleo

AgileEngine is seeking a Senior Data Engineer to design and operate batch and streaming data pipelines on Databricks using PySpark and Delta Lake. You will migrate legacy ETL workloads to a governed Lakehouse supporting analytics and AI use cases.

You will collaborate with DevOps and analytics teams, implement data quality, governance with Unity Catalog, and optimize Spark jobs for performance and cost while maintaining strong tests and documentation.