Junior Data Engineer Medellin, Colombia, Remote

Brightgrove Ltd.; Medellín; Híbrido; COP 60.000.000 - 90.000.000; Jornada completa

Ventajas ofrecidas por este puesto de trabajo

Interview process respectful of time

IT community

Knowledge sharing events

Recovery time and relaxation

Online and offline events

Volunteer community

Descripción de la vacante

Brightgrove Ltd. is seeking a data engineer to design, build, and optimize scalable data pipelines on Microsoft Azure. You will work with Azure Databricks, Data Factory, and Delta Lake to deliver reliable data transformations.

The role emphasizes data quality, governance, and secure, scalable processing of large datasets, collaborating with cross-functional teams to enable enterprise analytics.

Formación

• Hands-on experience building scalable data pipelines on Azure.
• Proficient in PySpark, Spark SQL and SQL data modeling.
• Experience with Delta Lake and Databricks notebooks/workflows.

Responsabilidades

• Design, develop and maintain scalable data pipelines using Azure Databricks and Azure Data Factory.
• Develop ETL/ELT solutions for ingesting and transforming data from multiple sources.
• Build and optimize PySpark applications and Spark SQL transformations.
• Create and manage Databricks notebooks, workflows, jobs and clusters.
• Implement Delta Lake-based data solutions ensuring data quality and performance.
• Collaborate with data architects and stakeholders to deliver enterprise data solutions.
• Establish CICD pipelines for Databricks artifacts using Azure DevOps or GitHub.
• Ensure security, governance and compliance using Unity Catalog and Azure services.

Conocimientos

Python

PySpark

Spark SQL

SQL

ETL/ELT

Databricks

Git/GitHub

Spark tuning

Herramientas

Azure Databricks

Azure Data Factory

Delta Lake

Azure Data Lake Storage Gen2

Descripción del empleo

Refer a Friend

About the Client

Our customer is a global energy and technology company operating across multiple markets worldwide. With a strong focus on digital transformation, engineering, and innovation, the company uses advanced data and cloud technologies to improve business operations and decision-making at scale.

The project focuses on building and scaling a modern cloud-based data platform on Microsoft Azure.

The platform brings together data from multiple sources and uses Azure Databricks, Data Factory, Data Lake Storage Gen2, PySpark, Spark SQL, and Delta Lake to create reliable, scalable data pipelines and transformation processes.

The solution is designed for large-scale data processing, with a strong focus on performance, data quality, security, governance, and automation.

Your Team

You will work as part of a cross-functional technology and client team, interacting with technical consultants, application specialists, senior IT professionals, engineers, account managers, business stakeholders, and customer leadership.

What's in it for you

• Interview process that respects people and their time
• Professional and open IT community
• Internal meet-ups and resources for knowledge sharing
• Time for recovery and relaxation
• Bright online and offline events
• Opportunity to become part of our internal volunteer community

Responsibilities

• Design develop and maintain scalable data pipelines using Azure Databricks and Azure Data Factory.
• Develop ETLELT solutions for ingesting transforming and loading data from multiple sources.
• Build and optimize PySpark applications and Spark SQL transformations for largescale data processing.
• Create and manage Databricks notebooks workflows jobs and clusters.
• Implement Delta Lakebased data solutions ensuring data quality reliability and performance.
• Develop data ingestion frameworks using Azure Data Lake Storage ADLS Gen2.
• Monitor troubleshoot and optimize data pipelines and Databricks workloads.
• Implement CICD pipelines for Databricks artifacts using Azure DevOps or GitHub.
• Collaborate with data architects business analysts and stakeholders to deliver enterprise data solutions.
• Ensure security governance and compliance using Unity Catalog Azure Key Vault and related Azure services.

Skills

Must-Have Skills

• Python — 1–3 years of hands-on experience
• Strong PySpark and Spark SQL
• Strong SQL and data modeling
• ETL/ELT development and optimization
• Databricks Workflows, Notebooks, and Delta Lake
• Git/GitHub
• Spark performance tuning and troubleshooting

Nice to Have

• Unity Catalog
• Snowflake
• Data governance and data quality
• Azure monitoring and logging
• CDC and real-time data processing