Data Engineer - Azure Databricks & Delta Lake

Brightgrove Ltd.; Medellín; Híbrido; COP 60.000.000 - 90.000.000; Jornada completa

Ventajas ofrecidas por este puesto de trabajo

Interview process respectful of time

IT community

Knowledge sharing events

Recovery time and relaxation

Online and offline events

Volunteer community

Descripción de la vacante

Brightgrove Ltd. is seeking a data engineer to design, build, and optimize scalable data pipelines on Microsoft Azure. You will work with Azure Databricks, Data Factory, and Delta Lake to deliver reliable data transformations.

The role emphasizes data quality, governance, and secure, scalable processing of large datasets, collaborating with cross-functional teams to enable enterprise analytics.

Formación

• Hands-on experience building scalable data pipelines on Azure.
• Proficient in PySpark, Spark SQL and SQL data modeling.
• Experience with Delta Lake and Databricks notebooks/workflows.

Responsabilidades

• Design, develop and maintain scalable data pipelines using Azure Databricks and Azure Data Factory.
• Develop ETL/ELT solutions for ingesting and transforming data from multiple sources.
• Build and optimize PySpark applications and Spark SQL transformations.
• Create and manage Databricks notebooks, workflows, jobs and clusters.
• Implement Delta Lake-based data solutions ensuring data quality and performance.
• Collaborate with data architects and stakeholders to deliver enterprise data solutions.
• Establish CICD pipelines for Databricks artifacts using Azure DevOps or GitHub.
• Ensure security, governance and compliance using Unity Catalog and Azure services.

Conocimientos

Python

PySpark

Spark SQL

SQL

ETL/ELT

Databricks

Git/GitHub

Spark tuning

Herramientas

Azure Databricks

Azure Data Factory

Delta Lake

Azure Data Lake Storage Gen2

Descripción del empleo

Brightgrove Ltd. is seeking a data engineer to design, build, and optimize scalable data pipelines on Microsoft Azure. You will work with Azure Databricks, Data Factory, and Delta Lake to deliver reliable data transformations.

The role emphasizes data quality, governance, and secure, scalable processing of large datasets, collaborating with cross-functional teams to enable enterprise analytics.