Senior Databricks Data Engineer — Production Pipelines

Medium; Denver (CO); Hybrid; USD 155,000 - 185,000; Full time

Benefits offered by this job

Healthcare package

401(k) match

Unlimited Paid Time Off

Paid Parental Leave

Online Legal Services

Financial Planning Services

Pet Insurance discount

Corporate Benefit Program

Health and Wellness Reimbursement

Travel Discounts

Career & Personal Development Program

Employee Referral Bonus Program

Job summary

SugarAI in Denver, CO, is seeking a Senior Data Engineer to own the Databricks pipelines powering Sugar Predict. You’ll drive production reliability, cost efficiency, and platform growth through onboarding and legacy modernization, working with ML engineers, product teams, and EA.

The role requires in‑person interaction in Denver with a hybrid work model. The ideal candidate has 4+ years in data engineering, 2+ years on Databricks/AWS/Azure, and strong Python/SQL skills.

Qualifications

• 4+ years of data engineering experience.
• At least 2 years on Databricks or the Apache Spark ecosystem across Azure and/or AWS.
• Proficiency in PySpark, SQL, and Python with production‑grade pipelines under SLA constraints.
• Hands‑on experience with Delta Lake including schema evolution, ACID transactions, optimize/vacuum lifecycle, and both incremental and streaming processing patterns.
• Hands‑on experience with pipeline performance tuning and compute optimization in production Databricks environments.
• Solid working knowledge of PostgreSQL including query optimization, schema design, and use as a source or sink in production data pipelines.
• Experience supporting and maintaining legacy ETL tooling (SSIS, Informatica, custom Python/SQL pipelines, or similar) in production.
• Experience supporting large‑scale multi‑tenant architectures with a focus on tenant isolation, per‑tenant performance, and data privacy.

Responsibilities

• Own Databricks production data pipelines and ensure reliability, cost efficiency, and scalability.
• Monitor SLA performance metrics and platform health across data flows.
• Migrate legacy ETL/ELT pipelines to Databricks and automate transitions.
• Onboard new customers by provisioning and hardening tenant data pipelines.
• Design high‑performance Databricks pipelines for ERP/CRM data at scale across Azure and AWS.
• Maintain Delta Lake architecture and data quality enforcement.
• Enforce data security practices including RBAC, secrets management, and Unity Catalog controls.
• Collaborate with Enterprise Architecture to integrate pipelines with SugarAI ecosystem.

Skills

Data engineering

Databricks

PySpark

SQL

Python

PostgreSQL

Data governance

Security/compliance

Cross-functional collaboration

CI/CD pipelines

Tools

Databricks

Microsoft SQL Server

SSIS

Informatica

Job description

SugarAI in Denver, CO, is seeking a Senior Data Engineer to own the Databricks pipelines powering Sugar Predict. You’ll drive production reliability, cost efficiency, and platform growth through onboarding and legacy modernization, working with ML engineers, product teams, and EA.

The role requires in‑person interaction in Denver with a hybrid work model. The ideal candidate has 4+ years in data engineering, 2+ years on Databricks/AWS/Azure, and strong Python/SQL skills.