Senior Data Engineer - Databricks & Cloud Pipelines

Retail Insight; Chicago (IL); Hybrid; USD 113,000 - 138,000; Full time

Benefits offered by this job

Hybrid & Flexible Working

Summer hours and extra days off

6% retirement match

Health, dental, vision benefits

Phone reimbursement

Onsite gym

Learning platform access

Job summary

Retail Insight is seeking a Senior Data Engineer to build and support scalable data systems powering our US retail data solutions. You will work with a Chicago-based team to translate business needs into practical data engineering solutions, developing robust pipelines and data products with a focus on reliability and performance.

You’ll collaborate with data scientists, analysts, and product teams to improve platform quality, contribute to architecture decisions, and help drive best practices

Qualifications

• Databricks production pipelines expertise.
• Experience with Unity Catalog & Asset Bundles.
• Strong Spark, Delta Lake, and Databricks Workflows knowledge.
• Proficient in Python and SQL; data modelling and ETL/ELT patterns.
• Experience with cloud data platforms (Azure/AWS/GCP).
• Ability to implement data quality checks and monitor pipelines.
• Excellent communication and collaboration with technical and non-technical colleagues.
• CS/related degree or equivalent practical experience; willingness to travel.

Responsibilities

• Build and maintain scalable data pipelines (batch and streaming).
• Design and implement data architecture components.
• Implement data quality checks, monitoring, observability.
• Apply security, access control, and governance standards.
• Collaborate with data scientists and analysts to deliver data products.
• Work with Lead Data Engineer to translate business requirements into tech solutions.
• Participate in code reviews and uphold engineering standards.
• Identify opportunities to optimize cloud costs and performance.
• Contribute to platform documentation and tooling decisions.

Skills

Databricks pipelines

Spark & Delta Lake

Python

SQL

Data modelling

ETL/ELT

Distributed processing

Azure/AWS/GCP

Communication

Team collaboration

Willing to travel

Education

Bachelor’s degree in CS/Data Engineering or related field

Tools

Databricks Unity Catalog

Databricks Asset Bundles

Databricks Workflows

Azure Data Factory

ADLS Gen2

Azure DevOps

dbt

Terraform/Bicep

Job description

Retail Insight is seeking a Senior Data Engineer to build and support scalable data systems powering our US retail data solutions. You will work with a Chicago-based team to translate business needs into practical data engineering solutions, developing robust pipelines and data products with a focus on reliability and performance.

You’ll collaborate with data scientists, analysts, and product teams to improve platform quality, contribute to architecture decisions, and help drive best practices