Hybrid Senior Databricks Data Engineer - Scale Pipelines

SugarAI; Denver (CO); Hybrid; USD 155,000 - 185,000; Full time

Benefits offered by this job

Healthcare

401(k) match

Unlimited PTO

Parental Leave

Legal services

Financial planning

Pet insurance

Education resources

Employee referral bonus

Job summary

SugarAI is redefining CRM with AI‑driven data platforms. We seek a Senior Data Engineer to own the Databricks pipelines behind Sugar Predict, delivering fast, clean data at scale across Azure and AWS.

You will work with ML, product, and Enterprise Architecture teams to ensure reliable data delivery for enterprise CRM/ERP insights. Hybrid role based in Denver, CO with in‑office work 2‑3 days weekly; you will help modernize legacy pipelines while onboarding new customers and maintaining strong

Qualifications

• 4+ years of data engineering experience.
• At least 2 years on Databricks or the Apache Spark ecosystem across Azure and/or AWS.
• Proficiency in PySpark, SQL, and Python with a strong track record building and operating production‑grade pipelines under SLA constraints.
• Hands‑on experience with Delta Lake including schema evolution, ACID transactions, optimize/vacuum lifecycle, and both incremental and streaming processing patterns.
• Hands‑on experience with pipeline performance tuning and compute optimization in production Databricks environments.
• Solid working knowledge of PostgreSQL including query optimization, schema design, and use as a source or sink in production data pipelines.
• Experience supporting and maintaining legacy ETL tooling (SSIS, Informatica, custom Python/SQL pipelines, or similar) in production.
• Experience supporting large‑scale multi‑tenant architectures with a focus on tenant isolation, per‑tenant performance, and data privacy.

Responsibilities

• Own Databricks production support for the Sugar Predict data platform, including monitoring, alerting, and incident response across all production data flows.
• Maintain and report on SLA performance metrics for data pipeline delivery, ensuring visibility into platform health and accountability across internal and external stakeholders.
• Identify and implement pipeline optimizations that reduce Databricks compute costs, improve throughput, and reduce processing windows while tracking impact across measurable KPIs.
• Migrate legacy ETL/ELT pipelines to Databricks, building automation tooling to reduce manual intervention and ensure uninterrupted data delivery during transitions.
• Support new customer onboarding by provisioning, validating, and hardening tenant data pipelines that deliver reliable, isolated data from day one.
• Design and build high‑performance Databricks pipelines that ingest, transform, and serve ERP and CRM data at scale across both Azure and AWS environments.
• Own the Delta Lake architecture including schema design, partitioning strategies, data quality enforcement, and incremental processing patterns.

Skills

PySpark

SQL

Python

Data pipelines

Cross‑functional collaboration

Tools

Databricks

Azure

AWS

PostgreSQL

SQL Server

SSIS

Informatica

Delta Lake

Unity Catalog

Job description

SugarAI is redefining CRM with AI‑driven data platforms. We seek a Senior Data Engineer to own the Databricks pipelines behind Sugar Predict, delivering fast, clean data at scale across Azure and AWS.

You will work with ML, product, and Enterprise Architecture teams to ensure reliable data delivery for enterprise CRM/ERP insights. Hybrid role based in Denver, CO with in‑office work 2‑3 days weekly; you will help modernize legacy pipelines while onboarding new customers and maintaining strong