Senior Databricks Engineer: Azure Delta Lake ERP Pipelines

TradeWind Services; Richland (WA); On-site; USD 125,000 - 200,000; Full time

Benefits offered by this job

Disability Insurance

Job summary

TradeWind Services is seeking a Senior Databricks Engineer to augment the client’s enterprise data team during the ERP platform rollout. This hands-on role will develop and optimize PySpark/SQL pipelines across the Medallion Architecture in Azure lakehouse environments.

You will lead data modeling, automated testing, ADR documentation, and CI/CD automation while aligning with ERP milestones and security standards. Mentorship of engineers is expected.

Qualifications

• U.S. Citizenship required.
• Active DOE Q Clearance or ability to obtain.
• 7+ years of data engineering/platform engineering experience, with 3-5+ years on production cloud data architectures.
• 5+ years of production experience with Azure Databricks, Delta Lake, PySpark, Spark SQL, Workflows, and Unity Catalog.
• Strong Python/PySpark and SQL coding in production in an agile environment.
• Expertise in dimensional data modeling (Kimball), star schemas, and platinum-level analytics.
• Experience ingesting ERP and large transactional data into modern lakes.
• Proficient in CI/CD pipelines, automated testing, and ADR documentation.
• Ability to mentor and upskill engineers.

Responsibilities

• Deliver hands-on technical execution across five core areas: Medallion Pipeline Development & Ingestion/Distribution.
• Ingest data from ERP and legacy data warehouse into ADLS Gen2 and Delta Lake.
• Implement data distribution patterns for downstream analytics and Power BI models.
• Lead enterprise dimensional modeling with star schemas and SCDs.
• Design automated data quality tests, monitoring, and SLA-focused pipelines.
• Operationalize Databricks Workflows and CI/CD using Databricks Asset Bundles and GitHub Actions.
• Document designs and ADRs in GitHub; mentor new Databricks engineers.

Skills

Azure Databricks

Delta Lake

PySpark

Spark SQL

Workflows

Unity Catalog

Python

SQL

CI/CD

GitHub Actions

Tools

Databricks Asset Bundles

GitHub Actions

Job description

TradeWind Services is seeking a Senior Databricks Engineer to augment the client’s enterprise data team during the ERP platform rollout. This hands-on role will develop and optimize PySpark/SQL pipelines across the Medallion Architecture in Azure lakehouse environments.

You will lead data modeling, automated testing, ADR documentation, and CI/CD automation while aligning with ERP milestones and security standards. Mentorship of engineers is expected.