Databricks Data Engineer: Lakehouse & AI Analytics

INGEPSY; Perímetro Urbano Barranquilla; Híbrido; COP 280.260.000 - 404.820.000; Jornada completa

Ventajas ofrecidas por este puesto de trabajo

Professional growth

Competitive USD-based compensation

A selection of exciting projects

Flextime

Descripción de la vacante

AgileEngine is seeking a Data Engineer to build batch and streaming pipelines on Databricks using PySpark and Delta Lake. You will migrate legacy data warehouse and ETL workloads onto a governed Lakehouse, modeling a medallion architecture that powers analytics and AI use cases.

Strong SQL, Python, and experience with Unity Catalog governance round out the role. The role involves designing and operating pipelines, optimizing Spark and Delta tables for performance, and collaborating with DevOps

Formación

• 4+ years of professional experience in data engineering with Spark and cloud-based data architectures.
• Strong SQL and Python with data modeling skills across dimensional and Lakehouse patterns.
• Experience with streaming ingestion using Structured Streaming, Auto Loader, Kafka, or Event Hubs.
• Experience with workflow orchestration (Databricks Workflows, Airflow, or Azure Data Factory).
• Experience with legacy platform migrations, ETL modernization, or managing data hygiene when porting old systems.
• Familiarity with Unity Catalog, data governance, access control, and PII handling.
• Experience delivering production data platforms at scale.

Responsabilidades

• Design, build, and operate batch and streaming data pipelines on Databricks using PySpark, Delta Lake, and Databricks Workflows.
• Model and maintain a medallion (bronze/silver/gold) architecture serving analytics, reporting, and machine learning consumers.
• Migrate legacy ETL and data warehouse workloads onto the Lakehouse with validated data parity and minimal business disruption.
• Write clean, well-tested Python and SQL; maintain high standards through code review and documentation.
• Optimize Spark jobs and Delta tables for performance and cost, including partitioning, clustering, caching, and cluster sizing.
• Implement data quality, lineage, and governance controls using Unity Catalog and automated validation checks.
• Debug, troubleshoot, and resolve pipeline failures, data defects, and production incidents.
• Collaborate with DevOps, platform, and analytics engineers on observability, security, and compliance best practices.

Conocimientos

Apache Spark

Python

SQL

Data modeling

Distributed systems

Mentoring

Herramientas

Databricks

Delta Lake

PySpark

Unity Catalog

Airflow

Azure Data Factory

dbt

Descripción del empleo

AgileEngine is seeking a Data Engineer to build batch and streaming pipelines on Databricks using PySpark and Delta Lake. You will migrate legacy data warehouse and ETL workloads onto a governed Lakehouse, modeling a medallion architecture that powers analytics and AI use cases.

Strong SQL, Python, and experience with Unity Catalog governance round out the role. The role involves designing and operating pipelines, optimizing Spark and Delta tables for performance, and collaborating with DevOps