Senior Databricks Data Engineer - Lakehouse & ML Pipelines

INGEPSY; Metropolitana; Presencial; COP 280.260.000 - 435.961.000; Jornada completa

Ventajas ofrecidas por este puesto de trabajo

Professional growth

Competitive compensation

Curated projects

Flextime

Descripción de la vacante

AgileEngine is seeking a Senior Data Engineer to design, build, and operate batch and streaming data pipelines on Databricks, leveraging PySpark and Delta Lake to migrate legacy workloads to a governed Lakehouse. You will model a medallion architecture powering analytics, dashboards, and AI use cases.

The role requires strong SQL, Python, and experience with Unity Catalog governance, plus collaboration with DevOps and analytics teams to ensure performance, reliability, and data quality.

Formación

• 3+ years of professional data engineering experience with Spark and cloud data architectures.
• Strong SQL, Python, and experience with Delta Lake and Databricks.
• Experience migrating ETL workloads to Lakehouse with validated data parity.
• Familiarity with Unity Catalog governance and data quality practices.
• Ability to troubleshoot pipelines and collaborate with DevOps and analytics teams.

Responsabilidades

• Design, build, and operate batch and streaming data pipelines on Databricks (PySpark, Delta Lake).
• Model and maintain a medallion architecture serving analytics, reporting, and ML consumers.
• Migrate legacy ETL and data warehouse workloads onto the Lakehouse with minimal disruption.
• Use Claude or GitHub Copilot to generate scaffolding, tests, and documentation.
• Write clean Python and SQL and maintain high coding standards through reviews.
• Optimize Spark jobs and Delta tables for performance and cost.
• Implement data quality, lineage, and governance using Unity Catalog.
• Debug and resolve pipeline failures and production incidents.
• Collaborate with DevOps, platform, and analytics engineers on observability and security.

Conocimientos

Databricks

Apache Spark

PySpark

Delta Lake

SQL

Python

Data pipelines

Batch processing

Streaming processing

GitHub Copilot

Unity Catalog

Herramientas

Databricks

Airflow

Azure Data Factory

dbt

Descripción del empleo

AgileEngine is seeking a Senior Data Engineer to design, build, and operate batch and streaming data pipelines on Databricks, leveraging PySpark and Delta Lake to migrate legacy workloads to a governed Lakehouse. You will model a medallion architecture powering analytics, dashboards, and AI use cases.

The role requires strong SQL, Python, and experience with Unity Catalog governance, plus collaboration with DevOps and analytics teams to ensure performance, reliability, and data quality.