Senior Data Engineer: Databricks, Lakehouse & Streaming

INGEPSY; Medellín; Presencial; COP 280.260.000 - 435.961.000; Jornada completa

Ventajas ofrecidas por este puesto de trabajo

Professional growth

Competitive compensation

A selection of exciting projects

Flextime

Descripción de la vacante

AgileEngine in Medellín, Colombia is seeking a Senior Data Engineer to design and operate batch and streaming data pipelines on Databricks using PySpark and Delta Lake. You will migrate legacy ETL workloads to a governed Lakehouse and model a medallion architecture powering analytics and AI use cases.

The role requires strong SQL, Python, and experience with Unity Catalog governance, with a focus on performance optimization and data quality. Join a people-first culture and impactful projects.

Formación

• 3+ years of professional experience in data engineering with Apache Spark and cloud-based data architectures.
• Strong hands-on experience building data pipelines with Databricks, Apache Spark (PySpark), and Delta Lake.
• Advanced SQL and Python with data modeling across dimensional and Lakehouse patterns.
• Experience with streaming ingestion using Structured Streaming, Auto Loader, Kafka, or Event Hubs.
• Experience with workflow orchestration (Databricks Workflows, Airflow, or Azure Data Factory).

Responsabilidades

• Design, build, and operate batch and streaming data pipelines on Databricks using PySpark, Delta Lake, and Databricks Workflows.
• Model and maintain a medallion architecture serving analytics, reporting, and ML consumers.
• Migrate legacy ETL and data warehouse workloads onto the Lakehouse with validated parity.
• Write clean, tested Python and SQL; maintain high standards through reviews.
• Optimize Spark jobs and Delta tables for performance and cost.
• Implement data quality, lineage, and governance controls with Unity Catalog.

Conocimientos

Apache Spark

Databricks

PySpark

SQL

Python

Data Modeling

Analytical Skills

Herramientas

Databricks Workflows

Airflow

Azure Data Factory

Descripción del empleo

AgileEngine in Medellín, Colombia is seeking a Senior Data Engineer to design and operate batch and streaming data pipelines on Databricks using PySpark and Delta Lake. You will migrate legacy ETL workloads to a governed Lakehouse and model a medallion architecture powering analytics and AI use cases.

The role requires strong SQL, Python, and experience with Unity Catalog governance, with a focus on performance optimization and data quality. Join a people-first culture and impactful projects.