Senior Data Engineer: Scale Pipelines, Spark & Data Lakes

Capgemini Engineering; Colombia; Presencial; COP 90.000.000 - 130.000.000; Jornada completa

Ventajas ofrecidas por este puesto de trabajo

Stable Employment

Learning & Development

Language Training

Health Coverage

Insurance Protection

Wellness Perks

Descripción de la vacante

Capgemini Engineering Colombia is seeking a data engineer to design, build, and optimize data pipelines for analytics. You will work with Databricks, Spark, and Delta Lake to ensure data quality and scalable architectures, collaborating with data scientists and analysts on ML workloads.

The role emphasizes data governance, security, and production support. Responsibilities include integrating data from diverse sources, supporting CI/CD workflows, and contributing to data migration and

Formación

• Hands-on experience with data engineering or big data development.
• Strong knowledge of SQL (Oracle/SQL Server) and Python.
• Experience with Databricks, Spark, Delta Lake, and data lake architectures.
• Familiarity with cloud platforms (Azure, AWS, or GCP) and data orchestration tools.
• Experience with CI/CD pipelines and version control (Git).

Responsabilidades

• Design, develop, and support data replication and integration solutions using HVR.
• Build scalable data pipelines using Databricks (Spark, Delta Lake).
• Develop and optimize ETL/ELT processes for structured and unstructured data.
• Ensure data quality, integrity, and performance optimization on large datasets.
• Collaborate with data scientists and analysts to enable advanced analytics and ML workloads.
• Integrate data from various sources including databases and APIs.
• Monitor and troubleshoot data pipelines and production issues.
• Support CI/CD pipelines and DevOps practices for data engineering workflows.

Conocimientos

Data engineering

Big data development

SQL

Python

Spark (PySpark/Scala)

CI/CD

Herramientas

Databricks Platform

Apache Spark

Delta Lake

Git

Cloud platforms (Azure/AWS/GCP)

Descripción del empleo

Capgemini Engineering Colombia is seeking a data engineer to design, build, and optimize data pipelines for analytics. You will work with Databricks, Spark, and Delta Lake to ensure data quality and scalable architectures, collaborating with data scientists and analysts on ML workloads.

The role emphasizes data governance, security, and production support. Responsibilities include integrating data from diverse sources, supporting CI/CD workflows, and contributing to data migration and