Azure Data Engineer - Databricks, PySpark & TB-Scale ETL

Pyramid Consulting, Inc; Colombia; On-site; COP 226,218,753 - 301,625,004; Part time

Job summary

A consulting firm in Colombia is seeking a skilled Data Engineer at the mid-senior level. The ideal candidate will have over 5 years of hands-on experience with Azure Databricks, Spark, and proficient coding in Python and SQL. Responsibilities include developing data processing pipelines, maintaining data lake architecture, and ensuring data quality. This role is a contract position that requires excellent technical writing skills and the ability to manage large-scale data effectively.

Qualifications

• 5+ years of hands-on experience with Azure Databricks and Spark.
• Proficiency in Python (PySpark) and SQL coding.
• Ability to optimize performance for TB-scale data processing.

Responsibilities

• Develop and implement data processing pipelines using Databricks and Spark.
• Write efficient and optimized code for TB-scale data processing.
• Design and maintain data lake architecture and ETL processes.

Skills

Azure Databricks

Spark

Python (PySpark)

SQL

Data quality management

Git

Tools

Azure data analytics services

Job description

A consulting firm in Colombia is seeking a skilled Data Engineer at the mid-senior level. The ideal candidate will have over 5 years of hands-on experience with Azure Databricks, Spark, and proficient coding in Python and SQL. Responsibilities include developing data processing pipelines, maintaining data lake architecture, and ensuring data quality. This role is a contract position that requires excellent technical writing skills and the ability to manage large-scale data effectively.