Azure Data Engineer

Pyramid Consulting, Inc; Colombia; Presencial; COP 226.218.753 - 301.625.004; A tiempo parcial

Descripción de la vacante

A consulting firm in Colombia is seeking a skilled Data Engineer at the mid-senior level. The ideal candidate will have over 5 years of hands-on experience with Azure Databricks, Spark, and proficient coding in Python and SQL. Responsibilities include developing data processing pipelines, maintaining data lake architecture, and ensuring data quality. This role is a contract position that requires excellent technical writing skills and the ability to manage large-scale data effectively.

Formación

• 5+ years of hands-on experience with Azure Databricks and Spark.
• Proficiency in Python (PySpark) and SQL coding.
• Ability to optimize performance for TB-scale data processing.

Responsabilidades

• Develop and implement data processing pipelines using Databricks and Spark.
• Write efficient and optimized code for TB-scale data processing.
• Design and maintain data lake architecture and ETL processes.

Conocimientos

Azure Databricks

Spark

Python (PySpark)

SQL

Data quality management

Git

Herramientas

Azure data analytics services

Descripción del empleo

About the Role

We are searching for a skilled Data Engineer to join our team. The ideal candidate will have hands‑on experience in development and production implementation with Azure Databricks and Spark, proficiency in Python (PySpark) and SQL coding, and a strong understanding of data lake architecture, ETL processes, and data quality management. Experience with Azure data analytics services and excellent technical writing skills are also required.

Responsibilities

• Develop and implement data processing pipelines using Databricks and Spark.
• Write efficient and optimized code in Python (PySpark) and SQL for TB-scale data processing.
• Design and maintain data lake architecture and ETL processes.
• Ensure data quality and integrity through robust data quality management practices.
• Utilize Azure data analytics services for data storage, processing, and analysis.
• Create comprehensive technical documentation, including requirements and testing documents.
• Collaborate with cross‑functional teams to understand data requirements and deliver solutions.
• Maintain version control using Git.

Requirements

• 5+ years of hands‑on experience in development and production implementation with Azure Databricks and Spark.
• Proficiency in Python (PySpark) and SQL coding.
• Proven ability to optimize performance for TB-scale data processing.
• Strong understanding of data lake architecture, ETL processes, and data quality management.
• Experience with Azure data analytics services.
• Excellent technical writing skills for creating requirements and testing documents.
• Ability to communicate technical concepts clearly and concisely.
• Knowledge of Git.

Additional Details

Seniority level: Mid‑Senior level

Employment type: Contract

Job function: Information Technology

Industries: IT Services and IT Consulting

Location: Bogota, D.C., Capital District, Colombia