Senior Data Engineer: Azure Data Pipelines & Spark

smashcr; Town of Texas (WI); On-site; USD 110,000 - 140,000; Full time

Job summary

SMASH is seeking an experienced Data Engineer to design, build, and maintain scalable data pipelines and cloud-based data solutions. The role emphasizes coding excellence with Python, SQL, Spark, Spark SQL, PySpark, and Microsoft Azure.

Ideal candidates will bring hands-on experience with Microsoft Fabric and have exposure to Pharmaceutical, Life Sciences, or Insurance industries. You will collaborate with analysts, scientists, and engineers to deliver reliable data capabilities.

Qualifications

• 5–6+ years of professional Data Engineering experience.
• Proven hands-on experience designing, building, and maintaining production data pipelines.
• Strong coding and software development capabilities.
• Strong hands-on Python experience.
• Advanced SQL skills.
• Hands-on experience with Apache Spark.
• Strong experience with Spark SQL.
• Strong experience developing data solutions using PySpark.
• Experience building data solutions within Microsoft Azure.
• Experience developing automated workflows for data ingestion, transformation, and delivery.

Responsibilities

• Design, build, and maintain automated data pipelines that move and transform data across systems.
• Develop scalable data workflows to support analytics, reporting, and machine learning use cases.
• Build data ingestion, transformation, processing, and integration solutions within Microsoft Azure.
• Develop and maintain production-quality code using Python and SQL.
• Use Apache Spark, Spark SQL, and PySpark to process and transform large-scale datasets.
• Design data transformations that convert raw data into reliable and usable formats for downstream consumers.
• Develop efficient data integration processes across multiple data sources and destinations.
• Monitor and troubleshoot data pipelines to ensure reliability, accuracy, and performance.
• Identify and resolve data quality, pipeline, and processing issues.
• Optimize data workflows and code for performance, scalability, and maintainability.
• Collaborate with Data Analysts, Data Scientists, engineering teams, and business stakeholders to understand data requirements.
• Support the implementation and continuous improvement of cloud-based data engineering solutions.
• Document pipeline architecture, transformations, dependencies, and technical processes.
• Follow software engineering best practices for coding, testing, version control, and deployment.

Skills

Python

SQL

Apache Spark

Spark SQL

PySpark

Azure

Data pipelines

ETL/ELT

Data integration

Analytical thinking

Tools

Microsoft Fabric

Job description

SMASH is seeking an experienced Data Engineer to design, build, and maintain scalable data pipelines and cloud-based data solutions. The role emphasizes coding excellence with Python, SQL, Spark, Spark SQL, PySpark, and Microsoft Azure.

Ideal candidates will bring hands-on experience with Microsoft Fabric and have exposure to Pharmaceutical, Life Sciences, or Insurance industries. You will collaborate with analysts, scientists, and engineers to deliver reliable data capabilities.