Remote Data Engineer - Databricks Lakehouse & PySpark

AgileEngine; Centrosur; Presencial; COP 100.440.000 - 156.240.000; Jornada completa

Ventajas ofrecidas por este puesto de trabajo

Growth without limits

Competitive compensation

Flexibility

Meaningful, modern projects

Collaborative culture

Well-being & support

Descripción de la vacante

AgileEngine is seeking a Middle Data Engineer to modernize a 15-year-old data warehouse into a governed Databricks Lakehouse. You will build batch and streaming pipelines with PySpark and Delta Lake, following a medallion architecture across bronze, silver, and gold layers.

This role also uses AI tools like Claude and Github Copilot to speed up development. Responsibilities include designing end-to-end data pipelines, migrating legacy workloads, and collaborating with DevOps and analytics teams

Formación

• 3+ years of professional experience in data engineering.
• Strong hands-on experience with Databricks, Spark (PySpark), and Delta Lake.
• Advanced SQL and Python skills with data modeling expertise.
• Experience with streaming ingestion using Structured Streaming, Kafka, or Event Hubs.
• Experience with workflow orchestration (Databricks Workflows, Airflow, or Azure Data Factory).
• Experience with legacy migrations, ETL modernization, or data hygiene projects.
• Strong problem-solving, collaboration, and communication skills.
• Familiarity with Unity Catalog, data governance, and PII handling.
• Experience with dbt or equivalent transformation framework.
• Familiarity with secure coding standards and security practices.
• Experience delivering production data platforms at scale.
• Upper-intermediate English level.

Responsabilidades

• Design, build, and operate batch and streaming data pipelines on Databricks using PySpark, Delta Lake, and Databricks Workflows.
• Model and maintain a medallion (bronze/silver/gold) architecture serving analytics, reporting, and ML consumers.
• Migrate legacy ETL and data warehouse workloads onto the Lakehouse with minimal business disruption.
• Use Claude or Copilot to accelerate development, generate scaffolding, and review tests.
• Write clean, well-tested Python and SQL; document and review code.
• Optimize Spark jobs and Delta tables for performance and cost through partitioning, clustering, and caching.
• Implement data quality, lineage, and governance with Unity Catalog and validation checks.
• Debug production pipelines, resolve data defects, and troubleshoot incidents.
• Collaborate with DevOps, platform, and analytics engineers on observability and security best practices.

Conocimientos

Apache Spark

PySpark

Delta Lake

SQL

Python

Structured Streaming

Kafka

Event Hubs

Databricks Workflows

Unity Catalog

dbt

Security best practices

Production data platforms

Herramientas

Databricks

Airflow

Azure Data Factory

dbt

Terraform

Dynatrace

CloudWatch

PostgreSQL

Descripción del empleo

AgileEngine is seeking a Middle Data Engineer to modernize a 15-year-old data warehouse into a governed Databricks Lakehouse. You will build batch and streaming pipelines with PySpark and Delta Lake, following a medallion architecture across bronze, silver, and gold layers.

This role also uses AI tools like Claude and Github Copilot to speed up development. Responsibilities include designing end-to-end data pipelines, migrating legacy workloads, and collaborating with DevOps and analytics teams