Real-Time Data Engineer - Azure Databricks & Spark

Blanc Labs; Bogotá ciudad; On-site; COP 90,000,000 - 140,000,000; Full time

Job summary

Blanc Labs is seeking a Data Engineer to design, build, and maintain scalable data pipelines on Azure Databricks and Synapse. You will implement CDC for real-time data synchronization and write Python code to process data, transform it, and load it into Delta Lake and ADLS Gen2.

You will collaborate with architects and analysts to translate business needs into robust solutions, monitor performance, and ensure data quality across ingestion, transformation, and storage layers.

Qualifications

• 3+ years of experience in data engineering or a related field.
• Hands-on with Azure Databricks for data processing and analytics workloads.
• Proficiency with Apache Spark for large‑scale distributed data processing.
• Experience implementing CDC solutions (e.g., Debezium, Azure SQL CDC, or Delta Live Tables).
• Strong programming skills in Python for data engineering tasks.
• Solid understanding of SQL and relational databases (Azure SQL).
• Experience building ETL/ELT pipelines using Azure Data Factory or Databricks Workflows.
• Knowledge of data modeling and data warehousing concepts (Azure Synapse).
• Strong analytical and problem‑solving skills.
• Good communication across tech and business teams.

Responsibilities

• Design, build, and maintain scalable data pipelines using Azure Databricks and Apache Spark.
• Develop and optimize ETL/ELT processes to ingest, transform, and load data from diverse sources.
• Implement CDC solutions for real-time data synchronization using Debezium, Azure SQL CDC, or Delta Live Tables.
• Write clean, efficient Python code for data processing and automation.
• Collaborate with data architects and analysts to translate requirements into robust pipelines on Azure Synapse/Databricks.
• Monitor and optimize pipeline performance, reliability, and cost using Azure Monitor and Databricks observability.
• Ensure data quality and integrity across ingestion, transformation, and storage (ADLS Gen2, Delta Lake).
• Support integration of structured/unstructured data into Microsoft Fabric / Azure Synapse platforms.
• Participate in code reviews and promote best practices in data engineering.
• Document pipelines, transformations, and architecture for team visibility.

Skills

Azure Databricks

Apache Spark

Python

SQL

CDC implementation

Azure Data Factory

Azure Synapse Analytics

Data modeling

Tools

Debezium

Delta Live Tables

Databricks Workflows

Azure Data Lake Storage

Job description

Blanc Labs is seeking a Data Engineer to design, build, and maintain scalable data pipelines on Azure Databricks and Synapse. You will implement CDC for real-time data synchronization and write Python code to process data, transform it, and load it into Delta Lake and ADLS Gen2.

You will collaborate with architects and analysts to translate business needs into robust solutions, monitor performance, and ensure data quality across ingestion, transformation, and storage layers.