Senior Databricks Engineer – Delta Lake & Multitenant

Sugar AI Limited; Denver (CO); Hybrid; USD 155,000 - 185,000; Full time

Benefits offered by this job

Healthcare package

401(k) match

Unlimited PTO

Parental leave

Education resources

Employee referral bonus

Travel discounts

Job summary

SugarAI is hiring a Senior Data Engineer to own the Databricks pipelines behind Sugar Predict, ensuring fast, clean data for ERP and CRM systems. You will work with ML engineers, product teams, and Enterprise Architecture to scale data delivery globally and onboard new customers with robust data pipes.

The role requires 4+ years of data engineering, strong Databricks experience on Azure/AWS, and expertise in PySpark, SQL, and Delta Lake.

Qualifications

• 4+ years of data engineering experience.
• Experience with Databricks on Azure and/or AWS.
• Strong proficiency in PySpark, SQL, and Python in production pipelines.
• Hands-on Delta Lake, including schema evolution and ACID transactions.
• Experience with multi-tenant data architectures and data privacy.

Responsibilities

• Own Databricks production support and data pipelines.
• Monitor SLA performance and platform health.
• Migrate legacy ETL/ELT to Databricks and automate transitions.
• Provision and validate tenant data pipelines for onboarding.
• Design high-performance Databricks pipelines across Azure and AWS.
• Enforce security, RBAC, and Unity Catalog controls across environments.
• Collaborate with ML, product, and infrastructure teams on end-to-end delivery.
• Apply CI/CD practices to data pipelines to ensure reliable deployments.

Skills

Databricks

PySpark

SQL

Python

Delta Lake

PostgreSQL

Data governance

Multi-tenant architecture

Tools

Azure

AWS

Unity Catalog

SSIS

Informatica

SQL Server

Job description

SugarAI is hiring a Senior Data Engineer to own the Databricks pipelines behind Sugar Predict, ensuring fast, clean data for ERP and CRM systems. You will work with ML engineers, product teams, and Enterprise Architecture to scale data delivery globally and onboard new customers with robust data pipes.

The role requires 4+ years of data engineering, strong Databricks experience on Azure/AWS, and expertise in PySpark, SQL, and Delta Lake.