Azure Data Engineer - Databricks, PySpark & TB-Scale ETL

Pyramid Consulting, Inc; Colombia; Presencial; COP 226.218.753 - 301.625.004; A tiempo parcial

Descripción de la vacante

A consulting firm in Colombia is seeking a skilled Data Engineer at the mid-senior level. The ideal candidate will have over 5 years of hands-on experience with Azure Databricks, Spark, and proficient coding in Python and SQL. Responsibilities include developing data processing pipelines, maintaining data lake architecture, and ensuring data quality. This role is a contract position that requires excellent technical writing skills and the ability to manage large-scale data effectively.

Formación

• 5+ years of hands-on experience with Azure Databricks and Spark.
• Proficiency in Python (PySpark) and SQL coding.
• Ability to optimize performance for TB-scale data processing.

Responsabilidades

• Develop and implement data processing pipelines using Databricks and Spark.
• Write efficient and optimized code for TB-scale data processing.
• Design and maintain data lake architecture and ETL processes.

Conocimientos

Azure Databricks

Spark

Python (PySpark)

SQL

Data quality management

Git

Herramientas

Azure data analytics services

Descripción del empleo

A consulting firm in Colombia is seeking a skilled Data Engineer at the mid-senior level. The ideal candidate will have over 5 years of hands-on experience with Azure Databricks, Spark, and proficient coding in Python and SQL. Responsibilities include developing data processing pipelines, maintaining data lake architecture, and ensuring data quality. This role is a contract position that requires excellent technical writing skills and the ability to manage large-scale data effectively.