Databricks Data Engineer- In MN, IA, WI, SD or ND - Insight Global

OpenTalent; Bloomington (IN); On-site; USD 90,000 - 120,000; Full time

Job summary

Insight Global is seeking a Data Engineer to join a large payor/provider based in Bloomington, Indiana. You will design and implement end-to-end data solutions using Azure Databricks, convert SQL to Python in Databricks, and maintain scalable data pipelines. The role requires strong PySpark and SQL skills and a focus on building robust, high-volume data workflows.

You'll work with cross-functional teams to optimize data models, ensure data quality, and deliver timely insights for decision making.

Qualifications

• Experience designing and implementing end-to-end data solutions using Azure Databricks.
• Strong PySpark and SQL scripting skills for transforming data at scale.
• Experience building and maintaining robust ETL pipelines and data workflows.

Responsibilities

• Architect and build enriched data objects in the data lake's gold zone.
• Convert current SQL to Python code in Databricks.
• Modify and maintain data pipelines to handle high volumes of data.
• Write, test, and optimize PySpark and SQL scripts for transforms and loads.

Skills

Data engineering

PySpark

SQL

Azure Databricks

Python

Job description

Job Description Insight Global is seeking a Data Engineer to join a large payor/provider. This person will be responsible to:

• Solution, architect and build enriched objects in the gold zone
• Design, develop, and implement end-to-end data solutions using Azure Databricks.
• Convert current SQL to Python code in Databricks.
• Modify and maintain data pipelines.
• Write, test, and optimize PySpark and SQL scripts to transform and load high volumes of structured data.
• Update or maintain existi…