Data Engineer: Real-Time Pipelines on Databricks

blanclabs; Colombia; A distancia; COP 224.531.691 - 304.721.581; Jornada completa

Descripción de la vacante

Blanc Labs is seeking a skilled Data Engineer to design, build, and maintain scalable data pipelines powering enterprise analytics and BI. The role focuses on real-time data integration, CDC, and Python-based processing, with collaboration across data architects and analysts.

You will implement scalable ETL/ELT processes, optimize data flows, and ensure data quality across ingestion, transformation, and storage layers while leveraging distributed processing frameworks.

Formación

• 3+ years of data engineering experience.
• Strong Python and SQL skills for data processing.
• Experience with Databricks and Spark for large-scale workloads.
• Experience implementing CDC solutions (Debezium or similar).
• Experience building and maintaining ETL/ELT pipelines in production.
• Understanding of data modeling and data warehousing concepts.

Responsabilidades

• Design, build, and maintain scalable data pipelines using Databricks and Apache Spark.
• Develop and optimize ETL/ELT processes to ingest, transform, and load data from diverse sources.
• Implement Change Data Capture (CDC) for real-time data synchronization.
• Write clean, efficient Python code for data processing and automation.
• Collaborate with data architects and analysts to translate business requirements into pipeline designs.
• Monitor, troubleshoot, and optimize pipeline performance and cost.

Conocimientos

Python

SQL

Data engineering

ETL/ELT

CDC

Communication

Herramientas

Databricks

Apache Spark

Debezium

Airflow

Kafka

AWS

Azure

Descripción del empleo

Blanc Labs is seeking a skilled Data Engineer to design, build, and maintain scalable data pipelines powering enterprise analytics and BI. The role focuses on real-time data integration, CDC, and Python-based processing, with collaboration across data architects and analysts.

You will implement scalable ETL/ELT processes, optimize data flows, and ensure data quality across ingestion, transformation, and storage layers while leveraging distributed processing frameworks.