Remote Databricks Data Engineer — Lakehouse & AI Pipelines

AgileEngine; Bogotá ciudad; Presencial; COP 90.000.000 - 150.000.000; Jornada completa

Ventajas ofrecidas por este puesto de trabajo

Flexible remote work

Annual learning budget

Growth opportunities

Competitive compensation

Supportive culture

Remote collaboration with global teams

Descripción de la vacante

AgileEngine is seeking a Middle Data Engineer to modernize a 15-year-old data warehouse into a governed Databricks Lakehouse. Build batch and streaming data pipelines with PySpark and Delta Lake, following a medallion architecture across bronze, silver, and gold layers.

Tools like Claude and Copilot speed up development. You will migrate legacy ETL workloads, optimize Spark jobs, and implement data quality and governance with Unity Catalog, while collaborating with DevOps and analytics teams.

Formación

• 3+ years of professional data engineering experience.
• Strong hands-on experience with Databricks, Spark (PySpark), and Delta Lake.
• Advanced SQL and Python with data modeling skills.
• Experience with streaming ingestion (Structured Streaming, Auto Loader, Kafka, Event Hubs).
• Experience with workflow orchestration (Databricks Workflows, Airflow, or Azure Data Factory).
• Experience with legacy platform migrations or ETL modernization.
• Familiarity with Unity Catalog, data governance, and PII handling.
• Experience with dbt or equivalent transformation framework.
• Upper-intermediate English level.

Responsabilidades

• Design, build, and operate batch and streaming data pipelines on Databricks using PySpark, Delta Lake, and Databricks Workflows.
• Model and maintain a medallion architecture serving analytics, reporting, and ML consumers.
• Migrate legacy ETL and data warehouse workloads onto the Lakehouse with validated data parity and minimal disruption.
• Use Claude or Copilot to accelerate development and generate tests and docs.
• Write clean, well-tested Python and SQL with code reviews and documentation.
• Optimize Spark jobs and Delta tables for performance and cost; manage partitioning and clustering.
• Implement data quality, lineage, governance via Unity Catalog and automated checks.
• Debug and resolve production data pipeline incidents; collaborate on observability and security.

Conocimientos

Apache Spark

PySpark

Delta Lake

SQL

Python

Structured Streaming

Databricks Workflows

Data modeling

Data warehouse migration

Herramientas

Databricks

Airflow

Azure Data Factory

Unity Catalog

dbt

Descripción del empleo

AgileEngine is seeking a Middle Data Engineer to modernize a 15-year-old data warehouse into a governed Databricks Lakehouse. Build batch and streaming data pipelines with PySpark and Delta Lake, following a medallion architecture across bronze, silver, and gold layers.

Tools like Claude and Copilot speed up development. You will migrate legacy ETL workloads, optimize Spark jobs, and implement data quality and governance with Unity Catalog, while collaborating with DevOps and analytics teams.