Data Engineer: Spark, Databricks & Azure Pipelines (Onsite)

SFE; Chicago (IL); On-site; USD 100,000 - 140,000; Full time

Job summary

SFE is seeking an experienced Data Engineer to design, develop, and maintain scalable data pipelines and data platforms in an onsite, multi-city environment including Chicago IL, St. Louis MO and Richardson TX.

The ideal candidate will have hands-on experience with Databricks, Azure, Apache Spark, and strong Python/Scala programming skills to build robust data processing solutions for healthcare-related datasets.

Qualifications

• Hands-on experience with Databricks and Spark in a data engineering role.
• Strong knowledge of Microsoft Azure cloud services for data ingestion, storage, processing, and integration.
• Experience building scalable data pipelines and ETL/ELT workflows in large-scale environments.

Responsibilities

• Design and develop scalable data pipelines and ETL/ELT workflows using Databricks and Apache Spark.
• Develop data engineering solutions using Python and Scala.
• Work with Azure cloud services for data ingestion, storage, processing, and integration.
• Build and optimize Spark-based data processing applications for large-scale datasets.
• Develop data transformation, cleansing, and validation processes.
• Collaborate with data architects, analysts, and application teams to deliver reliable data solutions.
• Troubleshoot performance, data quality, and pipeline issues.
• Follow best practices for data security, scalability, monitoring, and deployment.

Skills

Databricks

Azure

Apache Spark

Python

Scala

ETL/ELT

Data pipelines

Distributed computing

Cloud data

Communication

Job description

SFE is seeking an experienced Data Engineer to design, develop, and maintain scalable data pipelines and data platforms in an onsite, multi-city environment including Chicago IL, St. Louis MO and Richardson TX.

The ideal candidate will have hands-on experience with Databricks, Azure, Apache Spark, and strong Python/Scala programming skills to build robust data processing solutions for healthcare-related datasets.