Senior Data Engineer – Databricks Lakehouse (Remote)

INGEPSY; Bogotá; Presencial; COP 217.479.000 - 341.753.000; Jornada completa

Ventajas ofrecidas por este puesto de trabajo

Growth without limits

Competitive compensation

Flexibility: 100% remote

Descripción de la vacante

AgileEngine is seeking a Senior Data Engineer to modernize a 15-year-old data warehouse into a governed Databricks Lakehouse. You will build batch and streaming pipelines with PySpark and Delta Lake, using a medallion architecture while leveraging AI tools like Claude and GitHub Copilot.

You will migrate legacy workloads to the Lakehouse with data parity and work in an Agile environment, collaborating across DevOps, platform, and analytics teams to ensure security and observability.

Formación

• 4+ years of professional experience in data engineering with Spark and cloud data architectures.
• Advanced SQL and Python with data modeling skills across Lakehouse patterns.
• Experience with streaming ingestion using Structured Streaming, Auto Loader, Kafka, or Event Hubs.
• Experience with Databricks Workflows, Airflow, or Azure Data Factory.

Responsabilidades

• Design, build, and operate batch and streaming data pipelines on Databricks using PySpark, Delta Lake, and Databricks Workflows.
• Model and maintain a medallion (bronze/silver/gold) architecture serving analytics, reporting, and ML consumers.
• Migrate legacy ETL workloads to the Lakehouse with data parity and minimal disruption.
• Use Claude or Github Copilot to accelerate development, generate scaffolding, tests, and documentation.
• Write clean Python and SQL with emphasis on code reviews and testing.
• Optimize Spark jobs and Delta tables for performance and cost.
• Implement data quality, lineage, governance with Unity Catalog and validation checks.
• Debug and resolve pipeline failures and production incidents.
• Participate in Agile or product-centric delivery practices.
• Collaborate with DevOps, platform, and analytics engineers on observability and security.

Conocimientos

Databricks

Apache Spark (PySpark)

SQL

Python

Delta Lake

Unity Catalog

dbt

Airflow

Herramientas

Databricks Workflows

Azure Data Factory

Claude

Descripción del empleo

AgileEngine is seeking a Senior Data Engineer to modernize a 15-year-old data warehouse into a governed Databricks Lakehouse. You will build batch and streaming pipelines with PySpark and Delta Lake, using a medallion architecture while leveraging AI tools like Claude and GitHub Copilot.

You will migrate legacy workloads to the Lakehouse with data parity and work in an Agile environment, collaborating across DevOps, platform, and analytics teams to ensure security and observability.