Data Engineer — Databricks Lakehouse (Remote)

AgileEngine; Cartagena de Indias; Presencial; COP 205.918.000 - 300.957.000; Jornada completa

Ventajas ofrecidas por este puesto de trabajo

Growth opportunities

Competitive compensation

100% remote

Annual learning budget

Flexible hours

Modern projects

Collaborative culture

Well-being programs

Descripción de la vacante

AgileEngine is seeking a Middle Data Engineer to modernize a 15-year-old data warehouse into a governed Databricks Lakehouse. You will design batch and streaming pipelines with PySpark and Delta Lake, and implement a medallion architecture across bronze, silver, and gold layers.

This role also uses AI tools like Claude and GitHub Copilot to speed up development, emphasizes data quality, governance with Unity Catalog, and collaboration with DevOps, platform, and analytics engineers.

Formación

• 3+ years of professional data engineering experience
• Proficient with Spark and cloud-based data architectures
• Strong SQL and Python skills
• Experience with streaming using Structured Streaming, Auto Loader, Kafka or Event Hubs
• Experience with Databricks Workflows, Airflow or Azure Data Factory
• Experience with legacy platform migrations and ETL modernization
• Strong problem-solving and communication skills
• Familiarity with Unity Catalog and data governance
• Experience with dbt or equivalent transformation framework
• Upper-intermediate English

Responsabilidades

• Design, build, and operate batch and streaming data pipelines on Databricks using PySpark, Delta Lake, and Databricks Workflows.
• Model and maintain a medallion (bronze/silver/gold) architecture for analytics, reporting, and ML consumers.
• Migrate legacy ETL and data warehouse workloads onto the Lakehouse with validated data parity and minimal disruption.
• Use Claude or GitHub Copilot to accelerate development, write tests, and create documentation.
• Write clean Python and SQL; ensure quality via code reviews and tests.
• Optimize Spark jobs and Delta tables for performance and cost.
• Implement data quality, lineage, and governance with Unity Catalog.
• Debug and resolve pipeline failures and production incidents.
• Collaborate with DevOps, platform, and analytics engineers on observability, security, and compliance.

Conocimientos

Data engineering

Apache Spark

Databricks

Python

SQL

Herramientas

Databricks

Apache Spark (PySpark)

Delta Lake

Descripción del empleo

AgileEngine is seeking a Middle Data Engineer to modernize a 15-year-old data warehouse into a governed Databricks Lakehouse. You will design batch and streaming pipelines with PySpark and Delta Lake, and implement a medallion architecture across bronze, silver, and gold layers.

This role also uses AI tools like Claude and GitHub Copilot to speed up development, emphasizes data quality, governance with Unity Catalog, and collaboration with DevOps, platform, and analytics engineers.