Data Engineer - Databricks Lakehouse & AI-Driven Pipelines

AgileEngine; Pereira; Presencial; COP 84.000.000 - 126.000.000; Jornada completa

Ventajas ofrecidas por este puesto de trabajo

Growth budget

Competitive pay

Remote work options

Modern projects with global teams

Collaborative culture

Well-being programs

Descripción de la vacante

AgileEngine is seeking a Middle Data Engineer to modernize a 15-year-old data warehouse into a governed Databricks Lakehouse. You will design and operate batch and streaming pipelines using PySpark and Delta Lake while implementing a bronze/silver/gold data model to serve analytics and ML workloads.

The role requires strong SQL and Python skills, experience with Spark and Databricks, and familiarity with data governance and Unity Catalog.

Formación

• 3+ years of professional data engineering experience.
• Strong SQL and Python with data modeling skills.
• Experience with Apache Spark, Databricks, Delta Lake, and PySpark.
• Experience with Structured Streaming, Auto Loader, Kafka, or Event Hubs.
• Experience with Databricks Workflows, Airflow, or Azure Data Factory.
• Knowledge of Unity Catalog and data governance.

Responsabilidades

• Design, build, and operate batch and streaming data pipelines on Databricks with PySpark, Delta Lake, and Workflows.
• Maintain a medallion bronze/silver/gold architecture for analytics and ML consumers.
• Migrate legacy ETL workloads to Lakehouse with minimal business disruption.
• Leverage Claude or Copilot for development acceleration and testing.
• Write clean Python and SQL; ensure code quality via reviews and docs.
• Optimize Spark jobs and Delta tables for performance and cost.
• Implement data quality and governance checks with Unity Catalog.

Conocimientos

Apache Spark

Python

SQL

Data Modeling

English Proficiency

Herramientas

Databricks

Delta Lake

PySpark

dbt

Airflow

Azure Data Factory

Descripción del empleo

AgileEngine is seeking a Middle Data Engineer to modernize a 15-year-old data warehouse into a governed Databricks Lakehouse. You will design and operate batch and streaming pipelines using PySpark and Delta Lake while implementing a bronze/silver/gold data model to serve analytics and ML workloads.

The role requires strong SQL and Python skills, experience with Spark and Databricks, and familiarity with data governance and Unity Catalog.