Databricks Engineer: Lakehouse & Streaming Architect

INGEPSY; Risaralda; Presencial; COP 280.260.000 - 435.961.000; Jornada completa

Ventajas ofrecidas por este puesto de trabajo

Professional growth

Competitive USD-based compensation

A selection of exciting projects

Flextime

Descripción de la vacante

INGEPSY seeks a Data Engineer to design, build, and operate batch and streaming data pipelines on Databricks using PySpark and Delta Lake. You will migrate legacy ETL workloads to a governed Lakehouse, modeling a medallion architecture for analytics and AI use cases.

The role requires strong SQL and Python, experience with Unity Catalog governance, and collaboration with DevOps and analytics engineers in an Agile environment. Col/work in Pereira, RIS.

Formación

• 4+ years of professional experience in data engineering with Spark and cloud data architectures.
• Hands-on experience with Databricks, PySpark, and Delta Lake for pipelines.
• Advanced SQL and Python with data modeling across Lakehouse patterns.
• Experience with Structured Streaming, Auto Loader, Kafka, or Event Hubs.

Responsabilidades

• Design, build, and operate batch and streaming data pipelines on Databricks using PySpark, Delta Lake, and Databricks Workflows.
• Model and maintain a medallion architecture serving analytics, reporting, and ML users.
• Migrate legacy ETL and data warehouse workloads onto the Lakehouse with data parity.
• Write clean, well-tested Python and SQL; review code and document solutions.
• Optimize Spark jobs and Delta tables for performance and cost; tune partitions and caching.
• Implement data quality, lineage, and governance using Unity Catalog and tests.
• Collaborate with DevOps, platform, and analytics teams on security and observability.

Conocimientos

Apache Spark

Databricks

PySpark

Delta Lake

SQL

Python

Structured Streaming

Kafka

Event Hubs

Databricks Workflows

Airflow

Azure Data Factory

Unity Catalog

Data governance

PII handling

dbt

Security best practices

Terraform

Dynatrace

Herramientas

Terraform

Dynatrace

CloudWatch

PostgreSQL

Descripción del empleo

INGEPSY seeks a Data Engineer to design, build, and operate batch and streaming data pipelines on Databricks using PySpark and Delta Lake. You will migrate legacy ETL workloads to a governed Lakehouse, modeling a medallion architecture for analytics and AI use cases.

The role requires strong SQL and Python, experience with Unity Catalog governance, and collaboration with DevOps and analytics engineers in an Agile environment. Col/work in Pereira, RIS.