Lead Data Engineer: Azure & Databricks, Scale Personalization

84.51˚; Chicago (IL); On-site; USD 125,000 - 207,000; Full time

Benefits offered by this job

Health insurance

401(k) with Roth option and matching

Paid time off and holidays

Job summary

84.51° Lead Data Engineer designs and implements ingestion, storage, transformation, and distribution of big data on Azure Cloud. Use Databricks, Spark, PySpark, Python, JSON, and SQL to power customer personalization at scale.

Strong collaboration with Product and Data Science is essential for success. Take ownership of features, mentor engineers, and lead Databricks-based solution development while guiding REST API work hosted in Kubernetes is expected.

Qualifications

• Bachelor’s degree in CS or related field.
• 5+ years data development experience.
• 5+ years Spark (Databricks, PySpark) experience.
• 3+ years Python experience.
• 1+ year REST API development.
• 1+ year Kubernetes experience.

Responsibilities

• Own features across the SDLC for Databricks-based solutions.
• Lead design and development of Databricks-based solutions.
• Develop REST APIs hosted in Kubernetes.
• Mentor junior engineers and engineers across teams.
• Collaborate with Product and Data Science teams.
• Set technical strategy with Data Science and Product partners.
• Oversee teams including Offshore/Onshore resources.
• Drive continuous improvement in development practices.

Skills

Spark (Databricks/PySpark)

Python

REST APIs

Kubernetes

Agile SCRUM

ETL concepts

Education

Bachelor’s degree in Computer Science or related field

Tools

Databricks

SQL

Kubernetes

Job description

84.51° Lead Data Engineer designs and implements ingestion, storage, transformation, and distribution of big data on Azure Cloud. Use Databricks, Spark, PySpark, Python, JSON, and SQL to power customer personalization at scale.

Strong collaboration with Product and Data Science is essential for success. Take ownership of features, mentor engineers, and lead Databricks-based solution development while guiding REST API work hosted in Kubernetes is expected.