Databricks Lead

Healthcare Triangle, Inc.; Minneapolis (MN); On-site; USD 130,000 - 180,000; Full time

Job summary

Subash US IT Consulting seeks a Databricks Lead in Minneapolis to architect scalable data engineering solutions on Databricks. You will drive PySpark, Spark SQL, and Delta Lake-based pipelines and guide the data team.

You will collaborate with architects, analysts, and business stakeholders, mentoring engineers and delivering robust, cost-efficient data lakehouse architectures in a long-term, on-site role.

Qualifications

• Proficient in Databricks platform and PySpark programming.
• Strong SQL skills and experience with Spark SQL and Delta Lake.
• Experience building ETL/ELT pipelines and data lakehouse architectures.

Responsibilities

• Lead the design and development of scalable data engineering solutions using Databricks
• Develop and optimize PySpark, Spark SQL, and Delta Lake pipelines
• Build and manage ETL/ELT workflows and data lakehouse architecture
• Work with Azure/AWS, Databricks Workflows, Unity Catalog, and cloud storage
• Optimize performance, data quality, and cost of Databricks workloads
• Mentor engineers and provide technical leadership
• Collaborate with architects, analysts, and business teams

Skills

Databricks

PySpark

Spark SQL

Delta Lake

Python

SQL

ETL/ELT

Data Lakehouse

Azure/AWS

Unity Catalog

CI/CD

Git

Job description

We have a job opening for the position of Databricks Lead.

Job Title

Databricks Lead

Location

Minneapolis, MN

Duration

Long Term

Job Responsibilities

• Lead the design and development of scalable data engineering solutions using Databricks.
• Develop and optimize PySpark, Spark SQL, and Delta Lake pipelines.
• Build and manage ETL/ELT workflows and data lakehouse architecture.
• Work with Azure/AWS, Databricks Workflows, Unity Catalog, and cloud storage.
• Optimize performance, data quality, and cost of Databricks workloads.
• Mentor engineers and provide technical leadership.
• Collaborate with architects, analysts, and business teams.

Required Skills

Databricks, PySpark, Spark SQL, Delta Lake, Python, SQL, ETL/ELT, Data Lakehouse, Azure/AWS, Unity Catalog, CI/CD, Git.

Subash US IT Consulting