Senior Data Engineer - Azure Databricks Pipelines

Applaudo; Medellín; On-site; COP 100,000,000 - 140,000,000; Full time

Job summary

Applaudo is seeking a Senior Data Engineer in Medellín to design, build, and maintain production-grade data pipelines using Azure Databricks and Python. You will develop scalable PySpark/Spark SQL transformations and integrate data from APIs, databases, cloud storage, and SFTP sources into a centralized Data Hub.

Strong SQL, data warehousing knowledge, and experience with Azure data services are required. You will own data quality validations, troubleshoot issues, and collaborate with a US-based

Qualifications

• 5+ years in Data Engineering, building production-grade pipelines.
• Azure Databricks, Python, Spark experience required.
• Strong SQL and data warehousing/dimensional modeling knowledge.
• Experience integrating data from APIs, databases, cloud storage, and SFTP.
• English communication with a US-based team.

Responsibilities

• Design, build, and maintain production data pipelines using Azure Databricks and Python.
• Develop scalable transformations with PySpark or Spark SQL.
• Ingest data from APIs, relational databases, cloud storage, and SFTP sources.
• Create and maintain centralized data hub integrating multiple enterprise platforms.
• Develop and optimize data warehouse structures and data marts.
• Write and optimize SQL transformations for accuracy and performance.
• Implement data quality checks and validate source-target consistency.
• Investigate data issues and implement corrective actions.
• Document pipelines, logic, dependencies, and processes.

Skills

Azure Databricks

Python

Apache Spark

PySpark / Spark SQL

SQL

Data warehousing & dimensional model

Data integration across sources

Azure data services

Data quality & validation

Troubleshooting & debugging

English communication

Job description

Applaudo is seeking a Senior Data Engineer in Medellín to design, build, and maintain production-grade data pipelines using Azure Databricks and Python. You will develop scalable PySpark/Spark SQL transformations and integrate data from APIs, databases, cloud storage, and SFTP sources into a centralized Data Hub.

Strong SQL, data warehousing knowledge, and experience with Azure data services are required. You will own data quality validations, troubleshoot issues, and collaborate with a US-based