Senior Azure Data Engineer — Spark, PySpark & Pipelines

Smash CR; Dallas (TX); On-site; USD 110,000 - 140,000; Full time

Job summary

SMASH is seeking an experienced Data Engineer to design, build, and maintain scalable data pipelines and cloud-based data solutions. The role emphasizes coding quality and end-to-end pipeline development in Microsoft Azure.

Candidates should have hands-on experience with SQL, Python, Spark, Spark SQL, PySpark, and Azure, with additional value for Microsoft Fabric and experience in related industries. Collaborate with analysts and data scientists to deliver reliable data for analytics,

Qualifications

• 5-6+ years of professional Data Engineering experience.
• Proven hands-on experience designing, building, and maintaining production data pipelines.
• Strong coding and software development capabilities.
• Strong hands-on Python experience.
• Advanced SQL skills.
• Hands-on experience with Apache Spark.
• Strong experience with Spark SQL.
• Strong experience developing data solutions using PySpark.
• Experience building data solutions within Microsoft Azure.
• Experience developing automated workflows for data ingestion, transformation, and delivery.
• Experience processing and transforming large and complex datasets.
• Strong understanding of data integration, ETL/ELT, and data pipeline architecture.
• Ability to troubleshoot and optimize data pipelines and processing workloads.
• Strong understanding of data quality and validation practices.
• Strong analytical and problem-solving skills.

Responsibilities

• Design, build, and maintain automated data pipelines that move and transform data across systems.
• Develop scalable data workflows to support analytics, reporting, and machine learning use cases.
• Build data ingestion, transformation, processing, and integration solutions within Microsoft Azure.
• Develop and maintain production-quality code using Python and SQL.
• Use Apache Spark, Spark SQL, and PySpark to process and transform large-scale datasets.
• Design data transformations that convert raw data into reliable and usable formats for downstream consumers.
• Develop efficient data integration processes across multiple data sources and destinations.
• Monitor and troubleshoot data pipelines to ensure reliability, accuracy, and performance.
• Identify and resolve data quality, pipeline, and processing issues.
• Optimize data workflows and code for performance, scalability, and maintainability.
• Collaborate with Data Analysts, Data Scientists, engineering teams, and business stakeholders to understand data requirements.
• Support the implementation and continuous improvement of cloud-based data engineering solutions.
• Document pipeline architecture, transformations, dependencies, and technical processes.
• Follow software engineering best practices for coding, testing, version control, and deployment.

Skills

Python

SQL

Apache Spark

Spark SQL

PySpark

Azure

Data Pipelines

Data Integration

Tools

Microsoft Fabric

Job description

SMASH is seeking an experienced Data Engineer to design, build, and maintain scalable data pipelines and cloud-based data solutions. The role emphasizes coding quality and end-to-end pipeline development in Microsoft Azure.

Candidates should have hands-on experience with SQL, Python, Spark, Spark SQL, PySpark, and Azure, with additional value for Microsoft Fabric and experience in related industries. Collaborate with analysts and data scientists to deliver reliable data for analytics,