Remote Data Engineer: Databricks Lakehouse & Pipelines

AgileEngine; Metropolitana; Presencial; COP 283.751.000 - 409.862.000; Jornada completa

Ventajas ofrecidas por este puesto de trabajo

Growth without limits

Competitive compensation

Flexibility

Meaningful, modern projects

Collaborative culture

Well-being & support

Descripción de la vacante

AgileEngine is seeking a Middle Data Engineer to modernize a 15-year-old data warehouse into a governed Databricks Lakehouse. You will build batch and streaming data pipelines with PySpark and Delta Lake, following a medallion architecture across bronze, silver, and gold layers.

This role also uses AI tools like Claude and Github Copilot to speed up development. Responsibilities include migrating legacy workloads, optimizing Spark jobs, and implementing data quality and governance with Unity

Formación

• 3+ years of professional experience in data engineering.
• Expertise with Databricks, Apache Spark (PySpark), and Delta Lake.
• Advanced SQL and Python with strong data modeling skills.
• Experience with streaming ingestion (Structured Streaming, Kafka, Event Hubs).
• Experience with Databricks Workflows, Airflow or Azure Data Factory.
• Experience with legacy migrations, data hygiene, or ETL modernization.
• Familiarity with Unity Catalog, data governance, and PII handling.
• Experience with dbt or equivalent transformation framework.
• Upper-intermediate English level.

Responsabilidades

• Design, build, and operate batch and streaming data pipelines on Databricks using PySpark and Delta Lake.
• Maintain a medallion architecture (bronze/silver/gold) for analytics and ML.
• Migrate legacy ETL workloads to the Lakehouse with minimal disruption.
• Leverage Claude or GitHub Copilot to speed development and generate tests.
• Write clean Python and SQL; ensure code quality via reviews and docs.
• Optimize Spark jobs and Delta tables for performance and cost.
• Implement data quality, lineage, and governance using Unity Catalog.
• Troubleshoot pipeline failures and production incidents.
• Collaborate with DevOps, platform, and analytics teams on observability and security.

Conocimientos

Databricks

Apache Spark (PySpark)

Delta Lake

SQL

Python

Data Modeling

Unity Catalog

Kafka / Event Hubs

Airflow

Herramientas

Databricks Workflows

dbt

Azure Data Factory

Terraform

GitHub Copilot

Descripción del empleo

AgileEngine is seeking a Middle Data Engineer to modernize a 15-year-old data warehouse into a governed Databricks Lakehouse. You will build batch and streaming data pipelines with PySpark and Delta Lake, following a medallion architecture across bronze, silver, and gold layers.

This role also uses AI tools like Claude and Github Copilot to speed up development. Responsibilities include migrating legacy workloads, optimizing Spark jobs, and implementing data quality and governance with Unity