Data Engineer: Azure Pipelines & Spark Expert

SMASH; Dallas (TX); On-site; USD 110,000 - 150,000; Full time

Job summary

SMASH in the United States is seeking an experienced Data Engineer to design, build, and maintain scalable data pipelines and cloud-based data solutions. The role emphasizes hands-on work with SQL, Python, Spark, Spark SQL, PySpark, and Microsoft Azure, with a focus on end-to-end pipeline development, reliability, and scalable data platforms.

Knowledge of Microsoft Fabric is a plus, and experience in pharmaceutical, life sciences, or insurance industries is valued for collaboration with domain

Qualifications

• 5-6+ years of professional Data Engineering experience.
• Proven hands‑on experience designing, building, and maintaining production data pipelines.
• Strong coding and software development capabilities.
• Hands‑on Python experience.
• Advanced SQL skills.
• Hands‑on experience with Apache Spark.
• Strong experience with Spark SQL.
• Strong experience developing data solutions using PySpark.
• Experience building data solutions within Microsoft Azure.
• Experience developing automated workflows for data ingestion, transformation, and delivery.
• Experience processing and transforming large and complex datasets.
• Strong understanding of data integration, ETL/ELT, and data pipeline architecture.
• Ability to troubleshoot and optimize data pipelines and processing workloads.
• Strong understanding of data quality and validation practices.
• Strong analytical and problem‑solving skills.
• Ability to work independently while collaborating effectively with cross‑functional technical teams.

Responsibilities

• Design, build, and maintain automated data pipelines that move and transform data across systems.
• Develop scalable data workflows to support analytics, reporting, and machine learning use cases.
• Build data ingestion, transformation, processing, and integration solutions within Microsoft Azure.
• Develop and maintain production-quality code using Python and SQL.
• Use Apache Spark, Spark SQL, and PySpark to process and transform large-scale datasets.
• Design data transformations that convert raw data into reliable and usable formats for downstream consumers.
• Develop efficient data integration processes across multiple data sources and destinations.
• Monitor and troubleshoot data pipelines to ensure reliability, accuracy, and performance.
• Identify and resolve data quality, pipeline, and processing issues.
• Optimize data workflows and code for performance, scalability, and maintainability.
• Collaborate with Data Analysts, Data Scientists, engineering teams, and business stakeholders to understand data requirements.
• Support the implementation and continuous improvement of cloud-based data engineering solutions.
• Document pipeline architecture, transformations, dependencies, and technical processes.
• Follow software engineering best practices for coding, testing, version control, and deployment.

Skills

Data engineering

Python

SQL

Apache Spark

Spark SQL

PySpark

Microsoft Azure

Experience 5-6+ years

Job description

SMASH in the United States is seeking an experienced Data Engineer to design, build, and maintain scalable data pipelines and cloud-based data solutions. The role emphasizes hands-on work with SQL, Python, Spark, Spark SQL, PySpark, and Microsoft Azure, with a focus on end-to-end pipeline development, reliability, and scalable data platforms.

Knowledge of Microsoft Fabric is a plus, and experience in pharmaceutical, life sciences, or insurance industries is valued for collaboration with domain