Senior Databricks Lakehouse Engineer — Remote

AgileEngine, LLC.; Bogotá; Presencial; COP 289.119.000 - 481.866.000; Jornada completa

Ventajas ofrecidas por este puesto de trabajo

Growth without limits

Competitive compensation

Remote work 100%

Meaningful, modern projects

Collaborative culture

Well-being & support

Descripción de la vacante

AgileEngine is seeking a Senior Data Engineer to modernize a 15-year-old data warehouse into a governed Databricks Lakehouse, building batch and streaming pipelines with PySpark and Delta Lake. You will design a medallion architecture and migrate legacy workloads with minimal disruption, leveraging AI tools like Claude and GitHub Copilot to speed development.

Join a people-first team delivering analytics and ML-ready data products in a collaborative, Agile environment with remote-friendly

Formación

• 4+ years of professional data engineering experience.
• Experience with Databricks, Spark (PySpark), and Delta Lake.
• Advanced SQL and Python with data modeling skills.
• Experience with Structured Streaming, Auto Loader, Kafka or Event Hubs.
• Experience with Databricks Workflows, Airflow, or Azure Data Factory.
• Experience with legacy migration, ETL modernization, or data hygiene.
• Strong problem-solving, collaboration, and communication; mentoring juniors.
• Familiarity with Unity Catalog, data governance and PII handling.
• Experience with dbt or equivalent transformation framework.
• Familiarity with secure coding standards and security best practices.

Responsabilidades

• Design, build, and operate batch and streaming data pipelines on Databricks using PySpark, Delta Lake, and Databricks Workflows.
• Model and maintain a medallion architecture (bronze/silver/gold) for analytics, reporting, and ML consumption.
• Migrate legacy ETL and data warehouse workloads onto the Lakehouse with minimal disruption.
• Use Claude or Github Copilot to accelerate development, write tests, and document solutions.
• Write clean Python and SQL; enforce high-quality standards via reviews and docs.
• Optimize Spark jobs and Delta tables for performance and cost.
• Implement data quality, lineage, and governance via Unity Catalog and automated checks.
• Debug, troubleshoot, and resolve pipeline failures and data defects.
• Participate in Agile/product delivery with sprint planning and retrospectives.
• Collaborate with DevOps, platform, and analytics engineers on observability and security.

Conocimientos

Problem solving

Collaboration

Communication

Mentoring juniors

Explaining data concepts

Herramientas

Databricks

Apache Spark (PySpark)

Delta Lake

dbt

Azure Data Factory

Airflow

Terraform

Descripción del empleo

AgileEngine is seeking a Senior Data Engineer to modernize a 15-year-old data warehouse into a governed Databricks Lakehouse, building batch and streaming pipelines with PySpark and Delta Lake. You will design a medallion architecture and migrate legacy workloads with minimal disruption, leveraging AI tools like Claude and GitHub Copilot to speed development.

Join a people-first team delivering analytics and ML-ready data products in a collaborative, Agile environment with remote-friendly