Remote Data Engineer II: Scalable Pipelines & Real-Time

Essentia Health; Duluth (MN); Remote; USD 90,000 - 134,000; Full time

Benefits offered by this job

Medical insurance

Dental insurance

Vision insurance

Life insurance

Disability insurance

401(k) with employer contributions

Flexible scheduling

Paid time off

Wellness resources

Job summary

Essentia Health in Duluth, MN is seeking a Data Engineer II to design and optimize scalable data pipelines and workflows for large datasets. You will implement processing using Apache Spark, Airflow, and AWS Glue, and ensure data quality, governance, and security.

Collaborate with data scientists and analysts to deliver actionable data products for business-critical tools, and mentor junior engineers while building real-time and batch processing capabilities.

Qualifications

• Bachelor’s degree in computer science, data analytics, statistics, or related field.
• 2-5 years of experience in data engineering or related role.
• Proficiency in SQL and experience with database management systems.
• Strong programming skills in Python, Java, or Scala.
• Experience with Apache Airflow or AWS Glue for data ingestion.
• Experience with data lakes, lakehouse, and data catalogs.
• Experience with batch and streaming data for near real-time analytics.

Responsibilities

• Designs and optimizes scalable data pipelines to handle large datasets.
• Implements data processing using Spark, Airflow, and AWS Glue.
• Write and optimize advanced SQL queries for reporting.
• Ensure data quality, governance and security via automated testing.
• Collaborate with data scientists and analysts to understand data requirements.
• Provide mentorship to junior data engineers.
• Develop infrastructure for real-time and batch processing.
• Deliver actionable data products and insights for business-critical tools.

Skills

SQL

Python

Java

Scala

Education

Bachelor's degree in computer science, data analytics, statistics, or related field

Tools

Apache Spark

Airflow

AWS Glue

DBT

Microsoft Fabric

Job description

Essentia Health in Duluth, MN is seeking a Data Engineer II to design and optimize scalable data pipelines and workflows for large datasets. You will implement processing using Apache Spark, Airflow, and AWS Glue, and ensure data quality, governance, and security.

Collaborate with data scientists and analysts to deliver actionable data products for business-critical tools, and mentor junior engineers while building real-time and batch processing capabilities.