Remote Senior Data Engineer - Databricks Lakehouse

INGEPSY; Perímetro Urbano Barranquilla; Presencial; COP 100.440.000 - 200.880.000; Jornada completa

Ventajas ofrecidas por este puesto de trabajo

Growth without limits

Competitive compensation

Flexibility: 100% remote

Descripción de la vacante

AgileEngine is seeking a Senior Data Engineer to modernize a 15-year data warehouse into a governed Databricks Lakehouse. You will design batch and streaming pipelines with PySpark and Delta Lake, applying a medallion architecture for analytics and ML workflows.

You will also leverage AI tools like Claude and GitHub Copilot to accelerate development, while enforcing data governance, quality checks, and scalable best practices in a distributed, remote-friendly environment.

Formación

• 4+ years of professional data engineering experience.
• Experience with Apache Spark, PySpark, and Delta Lake on cloud data warehouses.
• Advanced SQL and Python with strong data modeling skills.
• Experience with streaming ingestion using Structured Streaming, Kafka, or Event Hubs.
• Experience with workflow orchestration (Databricks Workflows, Airflow, or Azure Data Factory).
• Experience with legacy platform migrations, ETL modernization, or managing data hygiene during porting of old systems.
• Strong problem-solving, collaboration, and communication skills, including mentoring junior engineers.
• Familiarity with Unity Catalog, data governance, access control, and PII handling.
• Experience with dbt or an equivalent transformation framework.

Responsabilidades

• Design, build, and operate batch and streaming data pipelines on Databricks using PySpark, Delta Lake, and Databricks Workflows.
• Model and maintain a medallion (bronze/silver/gold) architecture for analytics, reporting, and ML consumers.
• Migrate legacy ETL and data warehouse workloads onto the Lakehouse with minimal business disruption.
• Use Claude or GitHub Copilot to speed up development, write tests, and create documentation.
• Write clean Python and SQL and maintain high coding standards through reviews.
• Optimize Spark jobs and Delta tables for performance and cost.
• Implement data quality, lineage, and governance controls using Unity Catalog.

Conocimientos

Apache Spark

PySpark

Delta Lake

SQL

Python

Structured Streaming

Kafka

Databricks

Databricks Workflows

Airflow

Azure Data Factory

Unity Catalog

dbt

Data governance

Herramientas

Databricks Workflows

Airflow

Azure Data Factory

PostgreSQL

Descripción del empleo

AgileEngine is seeking a Senior Data Engineer to modernize a 15-year data warehouse into a governed Databricks Lakehouse. You will design batch and streaming pipelines with PySpark and Delta Lake, applying a medallion architecture for analytics and ML workflows.

You will also leverage AI tools like Claude and GitHub Copilot to accelerate development, while enforcing data governance, quality checks, and scalable best practices in a distributed, remote-friendly environment.