Senior Databricks Data Engineer — Remote & Flexible Hours

AgileEngine; Cartagena de Indias; Presencial; COP 90.000.000 - 150.000.000; Jornada completa

Ventajas ofrecidas por este puesto de trabajo

Growth without limits

Competitive compensation

Flexibility

Meaningful modern projects

Collaborative culture

Well-being & support

Descripción de la vacante

AgileEngine is seeking a Senior Data Engineer to modernize a 15-year-old data warehouse into a governed Databricks Lakehouse. You will build batch and streaming pipelines using PySpark, Delta Lake, and Databricks Workflows.

You will migrate legacy ETL workloads with validated data parity and support analytics, reporting, and ML consumers. Expect to leverage AI tools like Claude and GitHub Copilot to accelerate development, write clean Python and SQL, and optimize Spark jobs for performance and

Formación

• 4+ years of professional data engineering experience with Spark and cloud data architectures.
• Strong hands-on data pipelines using Databricks, Spark (PySpark) and Delta Lake.
• Advanced SQL and Python with data modeling across Lakehouse patterns.
• Experience with Structured Streaming using Kafka or Event Hubs.
• Experience with Databricks Workflows, Airflow or Azure Data Factory.
• Experience migrating legacy ETL and data warehouses to Lakehouse.
• Familiar with Unity Catalog, data governance, and PII handling.
• Experience with dbt or similar transformation frameworks.
• Knowledge of secure coding standards and production-grade data platforms.

Responsabilidades

• Design, build, and operate batch and streaming pipelines on Databricks with PySpark and Delta Lake.
• Maintain medallion architecture supporting analytics, reporting, and ML consumers.
• Migrate legacy workloads to Lakehouse with validated data parity.
• Use Claude or Copilot to accelerate development and testing.
• Write clean Python and SQL and conduct code reviews.
• Optimize Spark jobs and Delta tables for performance and cost.
• Implement data quality, lineage, and governance with Unity Catalog.
• Debug and resolve pipeline issues in production.
• Collaborate in Agile or product-centric sprints with DevOps and analytics teams.

Conocimientos

Databricks PySpark

Delta Lake

SQL

Python

Structured Streaming

Kafka / Event Hubs

Databricks Workflows

Airflow

Azure Data Factory

Unity Catalog

dbt

Cloud data architectures

Herramientas

Databricks

PostgreSQL

Terraform

Azure DevOps

Descripción del empleo

AgileEngine is seeking a Senior Data Engineer to modernize a 15-year-old data warehouse into a governed Databricks Lakehouse. You will build batch and streaming pipelines using PySpark, Delta Lake, and Databricks Workflows.

You will migrate legacy ETL workloads with validated data parity and support analytics, reporting, and ML consumers. Expect to leverage AI tools like Claude and GitHub Copilot to accelerate development, write clean Python and SQL, and optimize Spark jobs for performance and