Senior Data Engineer (Snowflake / Databricks)

NCompas Business Solutions Inc.; Salt Lake City (UT); On-site; USD 124,000 - 207,000; Full time

Job summary

NCompas Business Solutions Inc. is seeking a senior data engineering consultant to review and optimize a fragmented enterprise data landscape spanning Snowflake, Databricks, and Microsoft Fabric.

You'll assess data flows from SAP and Salesforce, design integration approaches, and outline workloads across platforms. The engagement runs ~124 hours from mid-November to mid-December 2026, with one on-site week for a technical deep dive, and you will build light prototypes to validate recommendations

Qualifications

• 7+ years in data engineering with Snowflake and Databricks.
• Proven experience designing and optimizing ETL/ELT pipelines at enterprise scale.
• Working knowledge of Microsoft Fabric and the Azure data stack (ADF, OneLake, Synapse).
• Experience integrating ERP/CRM data (SAP, Salesforce) into cloud analytics.
• Strong SQL and Python; experience with orchestration (Airflow, dbt, ADF, or Databricks Workflows).
• Understanding how ML pipelines are fed and deployed: feature stores, data versioning, and lineage.
• Ability to assess architecture and cost trade-offs across platforms and explain them clearly to technical and business audiences.
• Strong communication skills and comfort working on-site with client teams.

Responsibilities

• Review the client's data infrastructure and cloud ecosystem: connectivity, integration, data flows, and technical redundancy across platforms.
• Assess Snowflake and Databricks integration, and how Microsoft Fabric fits in as the emerging enterprise data foundation.
• Map data flows from source systems (SAP ERP, Salesforce, planning platforms, and third-party feeds) through to the forecasting and analytics models.
• Evaluate ETL/ELT pipelines for reliability, refresh latency, data quality, and cost — and identify paths to fresher data.
• Assess the data engineering and MLOps foundations behind the custom AI forecasting platform, working alongside the Data Scientist.
• Recommend workload placement (Snowflake for governed data marts and serving, Databricks for ML and feature engineering, Fabric for integration and BI), backed by total cost of ownership (TCO) and performance evidence.
• Build light prototypes or proof-of-concept integrations where needed to validate recommendations.
• Document findings and contribute to the architecture blueprint and final roadmap.

Skills

Snowflake & Databricks

ETL/ELT pipelines

Microsoft Fabric & Azure stack

ERP/CRM data integration

SQL & Python

Orchestration tools

ML pipelines basics

Cost & architecture trade-offs

On-site collaboration

Data governance & quality

Tools

FastAPI

PostgreSQL

Docker

React/Next.js

Job description

About the engagement

Our client runs a modern but fragmented data landscape across Snowflake, Databricks, and a growing Microsoft Fabric footprint. Much of its enterprise data (SAP, Salesforce, and partner systems) arrives in batches with 3-24 hour refresh delays. As part of an independent data science and planning assessment, you'll review the client's data infrastructure and recommend how the platforms should integrate and where workloads should run.

This is a fixed, phased engagement (~124 hours, mid-November to mid-December 2026) with one concentrated on-site week in mid-November for a technical deep dive. Industry: large U.S. beverage manufacturing & distribution.

What you'll do

• Review the client's data infrastructure and cloud ecosystem: connectivity, integration, data flows, and technical redundancy across platforms.
• Assess Snowflake and Databricks integration, and how Microsoft Fabric fits in as the emerging enterprise data foundation.
• Map data flows from source systems (SAP ERP, Salesforce, planning platforms, and third-party feeds) through to the forecasting and analytics models.
• Evaluate ETL/ELT pipelines for reliability, refresh latency, data quality, and cost — and identify paths to fresher data.
• Assess the data engineering and MLOps foundations behind the custom AI forecasting platform, working alongside the Data Scientist.
• Recommend workload placement (Snowflake for governed data marts and serving, Databricks for ML and feature engineering, Fabric for integration and BI), backed by total cost of ownership (TCO) and performance evidence.
• Build light prototypes or proof-of-concept integrations where needed to validate recommendations.
• Document findings and contribute to the architecture blueprint and final roadmap.

What we're looking for

• 7+ years in data engineering, with strong hands-on Snowflake and Databricks experience (Delta Lake, Spark, Unity Catalog, Snowpark, data sharing).
• Proven experience designing and optimizing ETL/ELT pipelines at enterprise scale.
• Working knowledge of Microsoft Fabric and the Azure data stack (ADF, OneLake, Synapse, or similar).
• Experience integrating ERP and CRM data (SAP, Salesforce) into cloud analytics platforms.
• Strong SQL and Python; experience with orchestration (Airflow, dbt, ADF, or Databricks Workflows).
• Understanding of how ML pipelines are fed and deployed: feature stores, data versioning, and lineage.
• Ability to assess architecture and cost trade-offs across platforms and explain them clearly to technical and business audiences.
• Strong communication skills and comfort working on-site with client teams.

Preferred

• Experience with FastAPI, PostgreSQL, Docker, and React/Next.js for rapid prototyping.
• Background in supply chain, manufacturing, CPG, or distribution data.
• Experience with data governance, lineage, and data quality frameworks.
• Databricks and/or Snowflake certifications.

Out of scope

Building production pipelines. The engagement uses read-only access and makes no production changes.