Lead Databricks Data Engineer

Centene Corporation; United States; Remote; USD 120,000 - 150,000; Full time

Job summary

A healthcare services company is looking for a Lead Databricks Data Engineer to design and implement data management procedures and ensure data quality. The ideal candidate has a Bachelor's degree in a quantitative field, 5-7 years of experience in data engineering, and advanced proficiency in Databricks Lakehouse architecture. This role focuses on overseeing data structures for business intelligence analytics.

Qualifications

• 5 - 7 years of related experience in data engineering.
• Advanced proficiency in Databricks Lakehouse Medallion architecture and Unity Catalog.
• Experience with scalable data pipelines on AWS Databricks.

Responsibilities

• Lead the design and implementation of standardized data management procedures.
• Oversee the development of large-scale data structures for business intelligence analytics.
• Implement improvements for data reliability, efficiency, and quality.

Skills

Databricks Lakehouse Medallion architecture

PySpark

SQL

Python

Data reliability

Data efficiency

Cloud storage platforms (S3, Azure Blob)

Education

Bachelor's degree in a quantitative or business field

Job description

A company is looking for a Lead Databricks Data Engineer.

Key Responsibilities

• Lead the design and implementation of standardized data management procedures
• Oversee the development and operation of large-scale data structures for business intelligence analytics
• Implement improvements for data reliability, efficiency, and quality while ensuring compliance with policies and standards

Required Qualifications

• Bachelor's degree in a quantitative or business field or equivalent experience
• 5 - 7 years of related experience in data engineering
• Advanced proficiency in Databricks Lakehouse Medallion architecture and Unity Catalog
• Experience with scalable data pipelines using PySpark, SQL, and/or Python on AWS Databricks
• Familiarity with cloud storage platforms such as S3 or Azure Blob