Senior Databricks Engineer — Flextime Hybrid Data Pipelines

INGEPSY; Risaralda; Presencial; COP 280.260.000 - 404.820.000; Jornada completa

Ventajas ofrecidas por este puesto de trabajo

Professional growth

Competitive USD-based compensation

A selection of exciting projects

Flextime

Descripción de la vacante

AgileEngine in Pereira, RIS, Colombia, is seeking a Senior Data Engineer to design and operate batch and streaming data pipelines on Databricks using PySpark and Delta Lake. The role focuses on migrating legacy ETL workloads to a governed Lakehouse architecture to power analytics and AI use cases.

You'll model a bronze/silver/gold medallion structure for analytics, reporting, and ML consumers, and collaborate with DevOps and analytics teams to ensure data quality and governance across the stack.

Formación

• 3+ years of professional experience in data engineering.
• Advanced SQL and Python for data modeling.
• Experience with Apache Spark and cloud-based data architectures.
• Strong experience building data pipelines with Databricks, PySpark, and Delta Lake.
• Experience with streaming ingestion using Structured Streaming, Auto Loader, Kafka, or Event Hubs.
• Experience with workflow orchestration (Databricks Workflows, Airflow, or Azure Data Factory).
• Experience with legacy platform migrations, ETL modernization, or data hygiene when porting old systems.
• Familiarity with Unity Catalog, data governance, access control, and PII handling.
• Experience with dbt or an equivalent transformation framework.
• Familiarity with secure coding standards and industry security best practices.
• Strong problem-solving, collaboration, and communication skills.

Responsabilidades

• Design, build, and operate batch and streaming data pipelines on Databricks using PySpark, Delta Lake, and Databricks Workflows.
• Model and maintain a medallion (bronze/silver/gold) architecture serving analytics, reporting, and ML consumers.
• Migrate legacy ETL and data warehouse workloads onto the Lakehouse with validated data parity and minimal disruption.
• Write clean, well-tested Python and SQL; ensure quality via tests and documentation.
• Optimize Spark jobs and Delta tables for performance and cost (partitioning, clustering, caching, sizing).
• Implement data quality, lineage, and governance controls using Unity Catalog and automated checks.
• Collaborate with DevOps, platform, and analytics engineers on observability, security, and compliance.
• Debug, troubleshoot, and resolve pipeline failures and production incidents.
• Use Claude or GitHub Copilot to accelerate development, generate code scaffolding, and prototyping solutions.

Conocimientos

Apache Spark

Cloud data architectures

Databricks

PySpark

Delta Lake

SQL

Python

Streaming pipelines

Data governance

PII handling

dbt

Problem solving

Communication

Databricks Workflows

Airflow

Azure Data Factory

Herramientas

Databricks Workflows

Airflow

Azure Data Factory

Descripción del empleo

AgileEngine in Pereira, RIS, Colombia, is seeking a Senior Data Engineer to design and operate batch and streaming data pipelines on Databricks using PySpark and Delta Lake. The role focuses on migrating legacy ETL workloads to a governed Lakehouse architecture to power analytics and AI use cases.

You'll model a bronze/silver/gold medallion structure for analytics, reporting, and ML consumers, and collaborate with DevOps and analytics teams to ensure data quality and governance across the stack.