Data Engineer - Databricks Lakehouse Pipelines (Remote)

AgileEngine; Sur; Presencial; COP 144.000.000 - 252.000.000; Jornada completa

Ventajas ofrecidas por este puesto de trabajo

Growth without limits

Competitive compensation

Remote-friendly with flexible hours

Modern projects with global teams

Collaborative culture

Descripción de la vacante

AgileEngine in Colombia is seeking a Middle Data Engineer to modernize a 15-year‑old data warehouse into a governed Databricks Lakehouse. You will build batch and streaming pipelines with PySpark and Delta Lake, following a medallion architecture for analytics, reporting and ML consumers.

You’ll use Claude and Github Copilot to speed development, write tests, and create documentation while maintaining high quality Python and SQL.

Formación

• 3+ years professional data engineering with Spark and cloud data architectures.
• Experience with Databricks, PySpark, and Delta Lake.
• Advanced SQL and Python with data modeling skills.
• Experience with streaming ingestion using Structured Streaming, Kafka, or Event Hubs.
• Workflow orchestration with Databricks Workflows, Airflow, or Azure Data Factory.
• Experience with legacy migrations, ETL modernization, or data hygiene.
• Strong problem-solving and collaboration skills.
• Upper-intermediate English proficiency.

Responsabilidades

• Design, build, and operate batch and streaming data pipelines on Databricks using PySpark and Delta Lake.
• Model and maintain medallion architecture (bronze/silver/gold) for analytics and ML users.
• Migrate legacy ETL workloads to Lakehouse with minimal disruption.
• Use Claude or Copilot to accelerate development, generate tests, and document solutions.
• Write clean Python and SQL; ensure code quality through reviews.
• Optimize Spark jobs and Delta tables for performance and cost.
• Implement data quality, lineage, and governance using Unity Catalog.
• Debug, troubleshoot, and resolve production pipeline incidents.
• Collaborate with DevOps, platform, and analytics engineers on observability and security.

Conocimientos

Databricks

PySpark

Delta Lake

SQL

Python

Structured Streaming

Airflow

dbt

Unity Catalog

Azure Data Factory

Herramientas

Databricks Workflows

Kafka

Delta Lake

Azure Data Factory

GitHub Copilot

Descripción del empleo

AgileEngine in Colombia is seeking a Middle Data Engineer to modernize a 15-year‑old data warehouse into a governed Databricks Lakehouse. You will build batch and streaming pipelines with PySpark and Delta Lake, following a medallion architecture for analytics, reporting and ML consumers.

You’ll use Claude and Github Copilot to speed development, write tests, and create documentation while maintaining high quality Python and SQL.