Senior Databricks Data Engineer - Remote Lakehouse & AI

AgileEngine; Sur; Presencial; COP 378.334.000 - 599.029.000; Jornada completa

Ventajas ofrecidas por este puesto de trabajo

Growth opportunities

Competitive compensation

100% remote with flexible hours

Challenging projects with leading tech

Collaborative culture

Well-being programs and support

Descripción de la vacante

AgileEngine is seeking a Senior Data Engineer to modernize a 15-year data warehouse into a governed Databricks Lakehouse. You will design and operate batch and streaming pipelines with PySpark and Delta Lake, applying a medallion architecture across bronze, silver, and gold layers for analytics and ML.

You will leverage AI tools like Claude and Github Copilot to accelerate development, mentor junior engineers, and collaborate with DevOps and analytics teams to ensure performance, security, and

Formación

• 4+ years of professional data engineering experience with Spark.
• Strong SQL and Python skills with data modeling expertise.
• Experience migrating ETL workloads to Lakehouse architectures.

Responsabilidades

• Design, build, and operate batch and streaming data pipelines on Databricks using PySpark and Delta Lake.
• Maintain a medallion architecture (bronze/silver/gold) for analytics and ML.
• Migrate legacy ETL workloads to Lakehouse with minimal disruption.
• Use AI tools to accelerate development, testing, and documentation.
• Write clean code and perform reviews; optimize Spark jobs for cost and performance.
• Implement data governance, lineage, and quality controls with Unity Catalog.

Conocimientos

Problem solving

Communication

Mentoring

Collaboration

English proficiency

Herramientas

Databricks

Apache Spark (PySpark)

Delta Lake

Airflow

Azure Data Factory

dbt

Unity Catalog

Structured Streaming

Descripción del empleo

AgileEngine is seeking a Senior Data Engineer to modernize a 15-year data warehouse into a governed Databricks Lakehouse. You will design and operate batch and streaming pipelines with PySpark and Delta Lake, applying a medallion architecture across bronze, silver, and gold layers for analytics and ML.

You will leverage AI tools like Claude and Github Copilot to accelerate development, mentor junior engineers, and collaborate with DevOps and analytics teams to ensure performance, security, and