Lead Data Engineer (Onsite)

US staffing Inc; Eagan (MN); On-site; USD 120,000 - 170,000; Full time

Job summary

US staffing Inc. is seeking an experienced Data Engineer to modernize and own critical data pipelines, focusing on Python/PySpark on Databricks. The role emphasizes leading pipeline development and collaborating with SMEs during a knowledge transfer period.

You will advance SAS-to-Python modernization efforts and contribute to metrics used to assess BCBS performance across care dimensions. The candidate will transition to full pipeline ownership over time.

Qualifications

• 10+ years data engineering experience with leadership responsibilities.
• Proficient in data manipulation and pipeline development.
• Experience with large datasets in distributed environments.
• Hands-on Databricks notebooks, workflows, and cluster management.
• Strong SQL for complex data extraction/transformation.

Responsibilities

• Modernize data pipeline from SAS to Python/PySpark on Databricks.
• Own and evolve our critical HEDIS data pipeline.
• HEDIS data is delivered to our analytics vendor to measure BCBS performance.
• Partner with SMEs during knowledge transfer.
• Eventually take on pipeline ownership.

Skills

Lead experience

Data manipulation

Distributed datasets

SQL

Pipeline scheduling

Python scripting

AI/Automation

Version control

Issue tracking (epics)

Tools

Databricks notebooks

Databricks workflows

Databricks cluster management

AWS S3

AWS Lambda

AWS Glue

AWS EC2

Git / GitHub

Jira

Job description

Job Description:

What we are looking for:

Need local candidates to Minnesota for this position

Linked profile should be old with solid connections and match with resume

Dev Engineer capabilities (this is the core of the role)

App dev experience (scripting, especially in python)

Expert in AI or Automation (this would be a huge plus)

Skills

• 10+ years of experience with lead experience
• Proficient in data manipulation and pipeline development
• Experience processing large datasets in a distributed environment
• Hands-on experience with Databricks notebooks, workflows, and cluster management
• Moderate hands-on experience with S3, Lambda, Glue, and EC2
• Experience scheduling and automating pipelines
• Strong SQL skills for complex data extraction and transformation queries
• Comfortable with version control in a collaborative, shared-repo environment
• Comfortable creating and managing issues and epics

Summary

• Modernize data pipeline from SAS to Python/PySpark on Databricks.
• This Data Engineer will own and evolve our critical HEDIS data pipeline.
• HEDIS data is delivered to our analytics vendor and used to measure BCBS’s performance across key dimensions of care.
• This role will initially partner with SMEs during a knowledge transfer period.
• They will eventually take on pipeline ownership.