Senior Databricks Data Engineer — Remote Lakehouse Builder

AgileEngine, LLC.; Pereira; Presencial; COP 256.995.000 - 385.493.000; Jornada completa

Ventajas ofrecidas por este puesto de trabajo

Growth without limits

Competitive compensation

Flexibility (remote)

Meaningful, modern projects

Collaborative culture

Well-being & support

Descripción de la vacante

AgileEngine, LLC. is seeking an experienced Senior Data Engineer to help modernize a 15-year-old data warehouse into a governed Databricks Lakehouse. You will build batch and streaming pipelines with PySpark and Delta Lake, following a medallion architecture across bronze, silver, and gold layers.

You will use Claude and GitHub Copilot to accelerate development, write tests, and create documentation, while collaborating in Agile teams and with DevOps for observability and security.

Formación

• 4+ years of professional data engineering experience with Spark.
• Experience with Databricks, PySpark, and Delta Lake.
• Advanced SQL and Python with data modeling.
• Experience with Structured Streaming, Auto Loader, Kafka.
• Experience with Databricks Workflows, Airflow, or Azure Data Factory.
• Familiarity with Unity Catalog and data governance.

Responsabilidades

• Design, build, and operate batch and streaming data pipelines on Databricks using PySpark, Delta Lake, and Databricks Workflows.
• Model and maintain a medallion architecture serving analytics, reporting, and ML consumers.
• Migrate legacy ETL and data warehouse workloads onto the Lakehouse with validated data parity and minimal business disruption.
• Write clean, well-tested Python and SQL; maintain high standards through code review and documentation.
• Optimize Spark jobs and Delta tables for performance and cost; include partitioning, clustering, and caching.
• Collaborate with DevOps, platform, and analytics engineers on observability, security, and governance.

Conocimientos

Apache Spark

PySpark

SQL

Python

Data Modeling

Problem solving

Mentoring

Herramientas

Databricks

Delta Lake

Airflow

Azure Data Factory

dbt

Terraform

Descripción del empleo

AgileEngine, LLC. is seeking an experienced Senior Data Engineer to help modernize a 15-year-old data warehouse into a governed Databricks Lakehouse. You will build batch and streaming pipelines with PySpark and Delta Lake, following a medallion architecture across bronze, silver, and gold layers.

You will use Claude and GitHub Copilot to accelerate development, write tests, and create documentation, while collaborating in Agile teams and with DevOps for observability and security.