Senior Databricks & Lakehouse Engineer

INGEPSY; Capital; Híbrido; COP 280.260.000 - 404.820.000; Jornada completa

Ventajas ofrecidas por este puesto de trabajo

Professional growth

Competitive USD compensation

A selection of exciting projects

Flextime

Descripción de la vacante

AgileEngine is seeking a Senior Data Engineer to design and operate batch and streaming data pipelines on Databricks using PySpark, Delta Lake, and Databricks Workflows. You will migrate legacy ETL workloads to a governed Lakehouse, modeling a medallion architecture that powers analytics and AI use cases.

The role requires strong SQL and Python, with experience in Unity Catalog governance and collaboration with DevOps and analytics teams to ensure performance, security, and data quality.

Formación

• 3+ years of professional experience in data engineering.
• Strong SQL and Python skills with data modeling experience.
• Experience with Databricks, Spark (PySpark), and Delta Lake.
• Familiarity with Unity Catalog governance and data quality controls.
• Experience with streaming and batch data pipelines.
• Experience with workflow orchestration tools (Airflow, Databricks Workflows, or Azure Data Factory).
• Knowledge of secure coding and data privacy practices.

Responsabilidades

• Design, build, and operate batch and streaming pipelines on Databricks (PySpark, Delta Lake).
• Model medallion architecture (bronze/silver/gold) for analytics and ML.
• Migrate legacy ETL workloads to Lakehouse with minimal disruption.
• Write clean Python and SQL with tests and docs.
• Improve performance and cost of Spark jobs and Delta tables.
• Implement data quality, lineage, and governance using Unity Catalog.
• Collaborate with DevOps and analytics teams on observability and security.

Conocimientos

Databricks

Apache Spark

Python

SQL

Delta Lake

Unity Catalog

PySpark

Cloud architectures

Herramientas

Airflow

Azure Data Factory

Databricks Workflows

Descripción del empleo

AgileEngine is seeking a Senior Data Engineer to design and operate batch and streaming data pipelines on Databricks using PySpark, Delta Lake, and Databricks Workflows. You will migrate legacy ETL workloads to a governed Lakehouse, modeling a medallion architecture that powers analytics and AI use cases.

The role requires strong SQL and Python, with experience in Unity Catalog governance and collaboration with DevOps and analytics teams to ensure performance, security, and data quality.