Remote Senior Data Engineer - Lakehouse & Spark

Capgemini Engineering; Aguascalientes; Presencial; MXN 600.000 - 900.000; Jornada completa

Ventajas ofrecidas por este puesto de trabajo

Flexible work location

Learning opportunities

Descripción de la vacante

Capgemini Engineering is seeking a Senior Data Engineer for a Sr role with remote work options. You will design, build, and optimize data pipelines using Databricks, Spark, and Lakehouse architecture while coordinating with data scientists and analysts on ML workloads.

You will ensure data quality, governance, and security across multiple sources and cloud platforms. The role involves monitoring Databricks clusters, implementing orchestration with ADF, and documenting standards and best

Formación

• Experience with Delta Lake and Lakehouse architecture.
• Knowledge of Azure Data Factory (ADF) for orchestration.
• Exposure to CI/CD pipelines for data engineering.
• Experience with SQL and Spark SQL.
• Azure certifications related to data engineering.

Responsabilidades

• Design, build, and maintain data pipelines and ETL with Databricks and Spark.
• Optimize data workflows for performance, scalability, and cost.
• Implement data lakehouse architecture and data ingestion from multiple sources.
• Collaborate with data scientists and analysts for analytics and ML workloads.
• Ensure data quality, governance, and security across assets.
• Monitor and troubleshoot Databricks clusters, jobs, and workflows.
• Integrate Databricks with AWS, Azure, or GCP and other systems.
• Document processes, standards, and best practices.

Conocimientos

Delta Lake

Lakehouse architecture

CI/CD pipelines

SQL

Spark SQL

Data governance

Herramientas

Databricks

Apache Spark

Azure Data Factory

Descripción del empleo

Capgemini Engineering is seeking a Senior Data Engineer for a Sr role with remote work options. You will design, build, and optimize data pipelines using Databricks, Spark, and Lakehouse architecture while coordinating with data scientists and analysts on ML workloads.

You will ensure data quality, governance, and security across multiple sources and cloud platforms. The role involves monitoring Databricks clusters, implementing orchestration with ADF, and documenting standards and best