Mid Data Engineer - Databricks Lakehouse (Remote)

INGEPSY; Cartagena de Indias; Presencial; COP 279.616.000 - 372.821.000; Jornada completa

Ventajas ofrecidas por este puesto de trabajo

Growth opportunities

Competitive compensation

100% remote with flexible hours

Modern tech stack

Collaborative culture

Well-being & support

Descripción de la vacante

AgileEngine is seeking a Middle Data Engineer to modernize a 15-year data warehouse into a governed Databricks Lakehouse. You will build batch and streaming pipelines with PySpark and Delta Lake, using a medallion architecture for analytics, reporting and ML consumption.

Responsibilities include migrating legacy workloads, implementing data quality and governance, and collaborating with DevOps and analytics teams. Strong SQL/Python, Spark, and cloud architecture experience are required.

Formación

• 3+ years of professional experience in data engineering.
• Hands-on with Databricks, Spark (PySpark), and Delta Lake.
• Advanced SQL and Python, data modeling skills.
• Experience with Structured Streaming and Kafka.
• Experience with Databricks Workflows, Airflow, or Azure Data Factory.
• Experience with legacy migrations and data hygiene.
• Familiarity with Unity Catalog and data governance.
• Knowledge of dbt or similar transformation framework.
• Secure coding and production data platforms at scale.
• Upper-intermediate English level.

Responsabilidades

• Design, build, and operate batch and streaming pipelines on Databricks.
• Model a medallion architecture (bronze/silver/gold) for analytics and ML.
• Migrate legacy ETL workloads to Lakehouse with minimal disruption.
• Leverage Claude or Copilot to accelerate development, tests, and docs.
• Write clean Python and SQL; ensure code quality via reviews.
• Optimize Spark jobs and Delta tables for performance and cost.
• Implement data quality, lineage, and governance controls.
• Troubleshoot pipeline failures and production incidents.
• Collaborate with DevOps, platform, and analytics teams on observability and security.

Conocimientos

Databricks

Apache Spark

PySpark

Delta Lake

SQL

Python

Structured Streaming

Kafka

Airflow

Azure Data Factory

dbt

Unity Catalog

Terraform

Cloud data architectures

Herramientas

Databricks

Airflow

Azure Data Factory

Terraform

Descripción del empleo

AgileEngine is seeking a Middle Data Engineer to modernize a 15-year data warehouse into a governed Databricks Lakehouse. You will build batch and streaming pipelines with PySpark and Delta Lake, using a medallion architecture for analytics, reporting and ML consumption.

Responsibilities include migrating legacy workloads, implementing data quality and governance, and collaborating with DevOps and analytics teams. Strong SQL/Python, Spark, and cloud architecture experience are required.