Lead Data Engineer — AI-Driven Pipelines on Databricks (MN)

US staffing Inc; Eagan (MN); On-site; USD 120,000 - 170,000; Full time

Job summary

US staffing Inc. is seeking an experienced Data Engineer to modernize and own critical data pipelines, focusing on Python/PySpark on Databricks. The role emphasizes leading pipeline development and collaborating with SMEs during a knowledge transfer period.

You will advance SAS-to-Python modernization efforts and contribute to metrics used to assess BCBS performance across care dimensions. The candidate will transition to full pipeline ownership over time.

Qualifications

• 10+ years data engineering experience with leadership responsibilities.
• Proficient in data manipulation and pipeline development.
• Experience with large datasets in distributed environments.
• Hands-on Databricks notebooks, workflows, and cluster management.
• Strong SQL for complex data extraction/transformation.

Responsibilities

• Modernize data pipeline from SAS to Python/PySpark on Databricks.
• Own and evolve our critical HEDIS data pipeline.
• HEDIS data is delivered to our analytics vendor to measure BCBS performance.
• Partner with SMEs during knowledge transfer.
• Eventually take on pipeline ownership.

Skills

Lead experience

Data manipulation

Distributed datasets

SQL

Pipeline scheduling

Python scripting

AI/Automation

Version control

Issue tracking (epics)

Tools

Databricks notebooks

Databricks workflows

Databricks cluster management

AWS S3

AWS Lambda

AWS Glue

AWS EC2

Git / GitHub

Jira

Job description

US staffing Inc. is seeking an experienced Data Engineer to modernize and own critical data pipelines, focusing on Python/PySpark on Databricks. The role emphasizes leading pipeline development and collaborating with SMEs during a knowledge transfer period.

You will advance SAS-to-Python modernization efforts and contribute to metrics used to assess BCBS performance across care dimensions. The candidate will transition to full pipeline ownership over time.