Senior Databricks Engineer - Lakehouse & Data Platforms

INGEPSY; Sur; Presencial; COP 280.260.000 - 467.101.000; Jornada completa

Ventajas ofrecidas por este puesto de trabajo

Competitive USD-based compensation

Flextime

Professional growth

A selection of exciting projects

Descripción de la vacante

AgileEngine is seeking a Senior Data Engineer to design, build, and operate batch and streaming data pipelines on Databricks using PySpark and Delta Lake. You will migrate legacy ETL workloads to a governed Lakehouse and implement data quality, governance, and testing practices.

The role emphasizes strong SQL/Python skills, Unity Catalog governance, and collaboration with DevOps and analytics teams to ensure scalable data platforms for analytics and AI use cases.

Formación

• 3+ years of professional experience in data engineering with Spark and cloud data architectures.
• Hands-on experience with Databricks, Spark (PySpark), and Delta Lake.
• Advanced SQL and Python with strong data modeling skills.
• Experience with Structured Streaming, Auto Loader, Kafka, or Event Hubs.
• Experience with Databricks Workflows, Airflow, or Azure Data Factory.
• Experience with legacy migrations and ETL modernization.
• Strong problem-solving, collaboration, and communication skills.
• Familiarity with Unity Catalog, data governance, and PII handling.
• Experience with dbt or equivalent transformation framework.
• Familiarity with secure coding standards and security practices.
• Experience delivering production data platforms at scale.

Responsabilidades

• Design, build, and operate batch and streaming data pipelines on Databricks using PySpark, Delta Lake, and Databricks Workflows.
• Model and maintain a medallion architecture serving analytics, reporting, and ML consumers.
• Migrate legacy ETL and data warehouse workloads onto the Lakehouse with validated data parity and minimal business disruption.
• Use Claude or GitHub Copilot to accelerate development, generate tests, document, and prototype solutions.
• Write clean, well-tested Python and SQL; ensure high quality via code reviews and docs.
• Optimize Spark jobs and Delta tables for performance and cost (partitioning, clustering, caching).
• Implement data quality, lineage, and governance using Unity Catalog and automated checks.
• Debug and resolve pipeline failures, data defects, and production incidents.
• Collaborate with DevOps, platform, and analytics engineers on observability and security.

Conocimientos

SQL

Python

Data modeling

Problem-solving

Communication

Herramientas

Databricks

Apache Spark (PySpark)

Delta Lake

Unity Catalog

Airflow

Azure Data Factory

dbt

Descripción del empleo

AgileEngine is seeking a Senior Data Engineer to design, build, and operate batch and streaming data pipelines on Databricks using PySpark and Delta Lake. You will migrate legacy ETL workloads to a governed Lakehouse and implement data quality, governance, and testing practices.

The role emphasizes strong SQL/Python skills, Unity Catalog governance, and collaboration with DevOps and analytics teams to ensure scalable data platforms for analytics and AI use cases.