Senior Databricks Data Engineer — Remote, Growth

AgileEngine; Perímetro Urbano Barranquilla; Presencial; COP 120.000.000 - 180.000.000; Jornada completa

Ventajas ofrecidas por este puesto de trabajo

Growth without limits

Competitive compensation

Remote work

Modern projects

Collaborative culture

Well-being programs

Descripción de la vacante

AgileEngine is seeking a Senior Data Engineer to modernize a 15-year-old data warehouse into a governed Databricks Lakehouse. You will build batch and streaming pipelines with PySpark and Delta Lake, following a medallion architecture across bronze, silver, and gold layers.

This role also uses AI tools like Claude and GitHub Copilot to speed up development. You will mentor engineers and collaborate across DevOps, analytics, and platform teams.

Formación

• 4+ years of professional data engineering experience.
• Expertise with Apache Spark and cloud data architectures.
• Proficiency in SQL and Python with data modeling skills.
• Experience with streaming ingestion (Structured Streaming, Kafka, Auto Loader).
• Familiarity with Unity Catalog and data governance.
• dbt or equivalent transformation framework.
• Experience migrating ETL workloads to lakehouse platforms.
• Strong problem-solving and collaboration skills, with ability to mentor.

Responsabilidades

• Design, build, and operate batch and streaming data pipelines on Databricks using PySpark, Delta Lake, and Databricks Workflows.
• Model and maintain a medallion architecture (bronze/silver/gold) for analytics, reporting, and ML.
• Migrate legacy ETL and data warehouse workloads onto the Lakehouse with minimal disruption.
• Leverage Claude or GitHub Copilot to accelerate development, testing, and documentation.
• Write clean Python and SQL with thorough code reviews and documentation.
• Optimize Spark jobs and Delta tables for performance and cost.
• Implement data quality, lineage, and governance using Unity Catalog and validations.
• Debug and resolve pipeline failures and production incidents.
• Participate in Agile delivery practices and collaborate with cross-functional teams.
• Work with DevOps, platform, and analytics engineers on observability, security, and compliance.

Conocimientos

Databricks

Apache Spark

PySpark

SQL

Python

Structured Streaming

Kafka

Unity Catalog

dbt

Security best practices

Herramientas

Databricks Workflows

Airflow

Azure Data Factory

Terraform

PostgreSQL

Descripción del empleo

AgileEngine is seeking a Senior Data Engineer to modernize a 15-year-old data warehouse into a governed Databricks Lakehouse. You will build batch and streaming pipelines with PySpark and Delta Lake, following a medallion architecture across bronze, silver, and gold layers.

This role also uses AI tools like Claude and GitHub Copilot to speed up development. You will mentor engineers and collaborate across DevOps, analytics, and platform teams.