Databricks Data Engineer - Lakehouse Pipelines & AI

AgileEngine; Colombia; Presencial; COP 248.463.000 - 372.694.000; Jornada completa

Ventajas ofrecidas por este puesto de trabajo

Growth opportunities

Competitive compensation

100% remote

Modern projects

Collaborative culture

Well-being programs

Descripción de la vacante

AgileEngine is seeking a Middle Data Engineer to modernize a 15-year-old data warehouse into a governed Databricks Lakehouse. You will design batch and streaming pipelines with PySpark and Delta Lake to serve analytics and ML workloads, while applying a medallion architecture.

Lead migrations from legacy ETL, optimize performance and cost, and collaborate with DevOps and analytics teams to ensure security and observability across the platform.

Formación

• 3+ years of professional experience in data engineering.
• Experience with Databricks, Spark (PySpark), and Delta Lake.
• Advanced SQL and Python with data modeling skills.
• Experience with streaming ingestion (Structured Streaming, Auto Loader, Kafka, Event Hubs).
• Experience with Databricks Workflows, Airflow, or Azure Data Factory.
• Familiarity with Unity Catalog and data governance.
• Experience with dbt or similar transformation framework.
• Upper-intermediate English level.

Responsabilidades

• Design, build, and operate batch and streaming data pipelines on Databricks.
• Model and maintain medallion (bronze/silver/gold) architecture for analytics and ML.
• Migrate legacy ETL workloads to Lakehouse with minimal disruption.
• Use Claude or Copilot to accelerate development, tests, and docs.
• Write clean Python and SQL with thorough code reviews and documentation.
• Optimize Spark jobs and Delta tables for performance and cost.
• Implement data quality, lineage, and governance with Unity Catalog.
• Debug and resolve production data pipeline issues.
• Collaborate with DevOps, platform, and analytics engineers on observability and security.

Conocimientos

Databricks

Apache Spark (PySpark)

Delta Lake

SQL

Python

Structured Streaming

Auto Loader

Kafka

Event Hubs

Databricks Workflows

Airflow

Azure Data Factory

dbt

Unity Catalog

Data governance

PII handling

Security best practices

Collaboration

English (Upper-intermediate)

Herramientas

Terraform

CloudWatch

Dynatrace

Databricks system tables

Descripción del empleo

AgileEngine is seeking a Middle Data Engineer to modernize a 15-year-old data warehouse into a governed Databricks Lakehouse. You will design batch and streaming pipelines with PySpark and Delta Lake to serve analytics and ML workloads, while applying a medallion architecture.

Lead migrations from legacy ETL, optimize performance and cost, and collaborate with DevOps and analytics teams to ensure security and observability across the platform.