Lead Data Engineer (Azure) — ETL, Spark & Architecture

SoftServe; Chile; On-site; CLP 40,000,000 - 70,000,000; Full time

Job summary

SoftServe is seeking a senior Data Engineer to design and deliver scalable data solutions in Azure. You will combine hands-on ETL development with technical leadership, working with clients and architects to translate business needs into reliable data platforms.

You will build pipelines with SQL, Python, Spark, PySpark, Azure Data Factory, and Airflow; develop data models; ensure data quality; optimize performance; guide the team and support cross-functional delivery.

Qualifications

• 5+ years of Data Engineering experience with ETL pipelines and data integration solutions.
• Advanced proficiency in SQL and Python for data processing, transformation, and optimization.
• Strong experience with Spark and PySpark for distributed data processing.
• Hands-on experience with Azure data technologies and cloud-based data platforms.
• Advanced proficiency with Azure Data Factory and/or Apache Airflow for workflow orchestration and scheduling.
• Strong understanding of Data Architecture, Data Modeling, Data Quality, Data Integration, and SQL/ETL performance optimization.
• Proven ability to provide technical guidance to engineering teams and serve as a technical reference.
• Strong working proficiency in English for daily communication with international clients and teams.
• Client-facing, communication, analytical, and problem-solving skills.
• Exposure to Microsoft Fabric, Hadoop, Hive, Kafka, Power BI, or Tableau is a plus.
• Experience with healthcare or pharmaceutical data environments is advantageous.

Responsibilities

• Design and develop scalable ETL and data integration pipelines across Azure data environments.
• Own technical implementation and solution quality, ensuring reliable, scalable, and maintainable data processing.
• Collaborate with clients, Solution Architects, and engineering teams to translate business requirements into effective data solutions.
• Build and optimize data processing workflows using SQL, Python, Spark, PySpark, Azure Data Factory, and Apache Airflow.
• Develop data models and schemas aligned with business requirements, architecture standards, and downstream consumption needs.
• Implement data quality and validation controls to maintain data integrity and consistency across platforms.
• Optimize SQL queries, ETL processes, and data workflows to improve performance and reliability.
• Guide the Data Engineering team on technical decisions, engineering practices, troubleshooting, and successful solution delivery.

Skills

Data engineering

Analytical thinking

Problem solving

Client communication

English proficiency

Tools

SQL

Python

Spark

PySpark

Azure Data Factory

Apache Airflow

Job description

SoftServe is seeking a senior Data Engineer to design and deliver scalable data solutions in Azure. You will combine hands-on ETL development with technical leadership, working with clients and architects to translate business needs into reliable data platforms.

You will build pipelines with SQL, Python, Spark, PySpark, Azure Data Factory, and Airflow; develop data models; ensure data quality; optimize performance; guide the team and support cross-functional delivery.