Senior Data Engineer: Azure Databricks Data Pipelines

Applaudo; Región Metropolitana de Santiago; On-site; CLP 24,000,000 - 42,000,000; Full time

Job summary

Applaudo is seeking a Senior Data Engineer to design and maintain production-grade data pipelines using Azure Databricks and Python. You will transform data from multiple sources into reliable datasets for business insights and own the data lifecycle.

You will implement data quality checks, collaborate with a US-based team, and document processes. Strong SQL, Spark, and Azure data services are essential, as is autonomous work style.

Qualifications

• 5+ years of professional experience in Data Engineering
• Strong hands-on experience with Azure Databricks
• Advanced Python skills for data processing, transformation, and pipeline development
• Solid experience with Apache Spark, including PySpark and/or Spark SQL
• Advanced SQL proficiency, including complex transformations, data validation, and query performance optimization
• Strong knowledge of data warehousing and dimensional modeling, including facts, dimensions, and data marts
• Experience integrating data from multiple sources, including REST APIs, relational databases, cloud storage, and SFTP
• Hands-on experience with Microsoft Azure data services for data ingestion, storage, processing, and integration
• Experience implementing data quality checks and validating consistency between source systems and target data models
• Ability to troubleshoot pipeline failures, investigate data discrepancies, and implement reliable solutions
• Strong problem-solving, analytical, and organizational skills
• Ability to work autonomously, take ownership, and follow established engineering practices and documentation standards
• Strong English communication skills, with the ability to collaborate effectively with a US-based team

Responsibilities

• Design, build, and maintain reliable, production-grade data pipelines using Azure Databricks and Python
• Develop scalable data transformations using PySpark or Spark SQL to integrate data from multiple business systems
• Ingest and process data from APIs, relational databases, cloud storage, and SFTP sources
• Integrate data from ticketing, CRM, marketing, retail, and other enterprise platforms into a centralized Data Hub
• Develop and maintain data warehouse structures, dimensional models, and data marts aligned with business requirements
• Write, optimize, and maintain SQL transformations to ensure data accuracy and processing efficiency
• Implement data quality validations and consistency checks between source systems and target models
• Build and maintain outbound data feeds for third-party systems according to defined specifications
• Investigate data discrepancies, troubleshoot pipeline issues, and implement appropriate corrective actions
• Document data pipelines, transformation logic, dependencies, and technical processes
• Collaborate with team members and stakeholders to clarify requirements, identify risks early, and deliver reliable data solutions
• Communicate progress, blockers, and potential delays proactively, proposing clear next steps to keep deliverables on track

Skills

Azure Databricks

Python

Apache Spark

SQL

Data warehousing

Dimensional modeling

Data integration

Azure data services

Data quality

Troubleshooting

Documentation

English communication

Autonomy

Ownership

Tools

REST APIs

SFTP

Job description

Applaudo is seeking a Senior Data Engineer to design and maintain production-grade data pipelines using Azure Databricks and Python. You will transform data from multiple sources into reliable datasets for business insights and own the data lifecycle.

You will implement data quality checks, collaborate with a US-based team, and document processes. Strong SQL, Spark, and Azure data services are essential, as is autonomous work style.