Databricks Solution Expert

Planned Systems International, Inc.; United States; Remote; USD 180,000 - 240,000; Full time

Benefits offered by this job

Paid leave

Health insurance

401(k) with match

Professional courses

Tuition reimbursement

Job summary

Planned Systems International, Inc. is seeking a Databricks Solution Expert to define platform patterns, build reference implementations, and drive enablement across product, analytics, and data science teams.

You will implement security, governance, and cost controls while delivering scalable, compliant insights at enterprise scale. You will blueprint landing zones, design lakehouse patterns, establish multi-workspace tenancy, and partner with Security, FinOps, and Compliance to codify

Qualifications

• 12+ years of experience in data engineering/analytics; 5+ years building on cloud data platforms (Azure preferred).
• 3+ years hands-on Azure Databricks (platform + pipelines) and Delta Lake.
• Proven experience setting up Unity Catalog with granular governance (RLS/CLS).
• Deep knowledge of Azure networking (VNets, Private Link, NSGs), identity (Entra ID), Key Vault, ADLS Gen2, Event Hubs, Azure SQL/MI, and Data Factory.
• Strong Spark expertise (PySpark/SQL), Structured Streaming, performance tuning, partitioning and storage optimization.
• Practical Terraform experience for Databricks/Azure resources; CI/CD with Azure DevOps or GitHub Actions.
• Security-first mindset; track record implementing audit logging, policy-as-code, and compliance controls.
• Excellent communication skills; ability to standardize, teach, and influence at enterprise scale.
• Active Public Trust clearance

Responsibilities

• Blueprint the Azure Databricks landing zone: workspace topology (prod/non-prod), network architecture, secure connectivity to data sources.
• Governance with Unity Catalog: multi-region metastores, catalog/schema/table design, data classification tiers, row/column-level security, data lineage, and cross-domain sharing patterns.
• Lakehouse foundations: Delta Lake storage design, medallion flow, partitioning, OPTIMIZE/VACUUM, and performance practices (Photon, AQE).
• Scalability for thousands of teams: multi-workspace strategy, tenancy model, workspace baselines, cluster policy tiers, guardrails for cost and isolation.
• Reliability: HA/DR strategy, regional deployments, backup/restore, versioning, and Terraform-based provisioning.

Skills

Azure Databricks

Unity Catalog

Terraform

Azure DevOps

PySpark

SQL

Security

CI/CD

Education

Bachelor's degree in Information Technology or related field

Tools

Terraform

Azure DevOps

GitHub Actions

dbx

Unity Catalog

Job description

Databricks Solution Expert

Location: US-Remote

Job ID: 2026-12115

Overview

Planned Systems International, Inc. is hiring a Databricks Solutions Expert that will define platform patterns, build reference implementations, set standards (cluster policies, Unity Catalog governance, CI/CD), and drive enablement so product, analytics, and data science teams can deliver reliable, compliant, and performant insights at scale.

Essential Functions and Job Responsibilities

• Platform Architecture & Design
  • Blueprint the Azure Databricks landing zone: workspace topology (prod/non-prod), network architecture (VNet injection, Private Link, NAT), secure connectivity to ADLS Gen2, Azure SQL/MI, Event Hubs, and other data sources.
  • Governance with Unity Catalog: multi-region metastores, catalog/schema/table design, data classification tiers, row/column-level security, data lineage, and cross-domain data sharing patterns (Delta Sharing).
  • Lakehouse foundations: Delta Lake storage design (bronze/silver/gold), medallion data flow standards, partitioning, Z ordering, OPTIMIZE/VACUUM policies, and performance best practices (Photon, AQE).
  • Scalability for "thousands of teams": multi-workspace strategy, tenancy model (shared vs. dedicated), workspace baselines, cluster policy tiers, and guardrails to prevent noisy-neighbor and cost runaways.
  • Reliability: HA/DR strategy, regional deployments, backup/restore, versioning, and repeatable environment provisioning via Terraform.
• Security, Compliance & Access Control
  • Identity & access: Entra ID (Azure AD) SSO, SCIM user/group provisioning, service principals/managed identities, attribute-based/role-based access controls mapped to Unity Catalog.
  • Secrets & credentials: Key Vault-backed secret scopes, credential passthrough patterns, token hygiene (PAT governance).
  • Data protection: encryption at rest/in transit, private endpoints, data exfil and egress controls, policy-as-code, and audit logging to Log Analytics or secure storage.
  • Compliance: enforce enterprise policies (PII/PHI handling), data retention, legal hold, and regulatory reporting with auditable lineage.
• Engineering & Enablement
  • Pipelines: build DLT (Delta Live Tables) pipelines and jobs for batch and streaming (Structured Streaming) with CDC (e.g., via Auto Loader) from enterprise sources.
  • Performance engineering: optimize notebooks/SQL/ETL (Photon, caching, skew mitigation), tune cluster sizing, and set standards for reliable, fast jobs.
  • MLOps: integrate MLflow for experiment tracking, model registry, feature store (Unity Catalog), and serving patterns where appropriate.
  • Observability: end-to-end monitoring (jobs, clusters, UC audits), dashboarding, alerting, and SLOs; integrate with Azure Monitor/Log Analytics.
  • Enablement: build reusable reference accelerators (templates, example notebooks, data products), run playbooks, and conduct office hours to uplevel 1000s of teams.
• DevOps, Automation & Cost Management
  • Infrastructure-as-Code: provision workspaces, catalogs, cluster policies, and permissions via Terraform (Databricks provider), with pipelines in Azure DevOps/GitHub.
  • CI/CD: notebook/package deployment, testing harnesses (dbx/pytest), environment promotion, and artifact versioning.
  • FinOps: cost modeling, budgets/alerts, instance pools, auto-termination, serverless SQL, tagging/chargeback, and usage analytics for executive reporting.
• Stakeholder Management & Governance
  • Partner with Security, Networking, Compliance, and FinOps to codify enterprise standards.
  • Establish a Lakehouse Platform Council to ratify patterns and review exceptions.
  • Create adoption metrics, business case narratives, TCO models, and executive updates.

Minimum Requirements

• Bachelor's degree in Information Technology or a related field. (or commensurate experience)
• 12+ years of experience in data engineering/analytics; 5+ years building on cloud data platforms (Azure preferred).
• 3+ years hands-on Azure Databricks (platform + pipelines) and Delta Lake.
• Proven experience setting up Unity Catalog with granular governance (RLS/CLS).
• Deep knowledge of Azure networking (VNets, Private Link, NSGs), identity (Entra ID), Key Vault, ADLS Gen2, Event Hubs, Azure SQL/MI, and Data Factory.
• Strong Spark expertise (PySpark/SQL), Structured Streaming, performance tuning, partitioning and storage optimization.
• Practical Terraform experience for Databricks/Azure resources; CI/CD with Azure DevOps or GitHub Actions.
• Security-first mindset; track record implementing audit logging, policy-as-code, and compliance controls.
• Excellent communication skills; ability to standardize, teach, and influence at enterprise scale.
• Active Public Trust clearance

Desired Qualifications

• Experience operating platforms for >500 concurrent users and 1000s of analytics teams.
• Knowledge of Photon, DLT, Workflows, Lakehouse ML (MLflow, feature store), Delta Sharing.
• Exposure to FinOps and chargeback models for data platforms.
• Experience with Synapse/Fabric interoperability, and data virtualization patterns.
• Background in data modeling (medallion, dimensional, domain driven design) and data quality (expectations, SLAs).

Additional Information

Professional Certifications

• Databricks: Databricks Certified Data Engineer Professional, Lakehouse Fundamentals, Machine Learning Associate/Professional.
• Microsoft: Azure Solutions Architect Expert (AZ 305), Azure Data Engineer (DP 203), Azure Security Engineer (AZ 500).
• HashiCorp: Terraform Associate.

Required Technical/Business Tools Experience

• Core: Azure Databricks (Unity Catalog, Delta Lake, DLT, Workflows, Photon), ADLS Gen2, Azure Key Vault, Entra ID, Private Link.
• Data Integration: Auto Loader, ADF/Synapse pipelines, Event Hubs/Kafka, JDBC/ODBC.
• DevOps/Infra: Terraform (Databricks & Azure providers), Azure DevOps/GitHub Actions, dbx.
• Observability: Databricks audit logs, Azure Monitor, Log Analytics, custom usage analytics.
• Languages: Python (PySpark), SQL, Scala (optional).
• ML: MLflow, Feature Store (Unity Catalog), model serving patterns.

Company Benefits

• paid leave
• employer sponsored group medical
• dental
• vision
• short-term disability
• long-term disability
• life insurance
• AD&D coverage
• legal services
• identity theft
• accident insurance
• Flexible spending account and health saving account options offer pre-tax savings for qualified medical, dental, and vision expenses
• The company sponsored 401(k) retirement plan with employer contribution match that is immediately vested
• professional courses, certifications, and tuition reimbursement programs

EEO Commitment

It is company policy to promote equal employment opportunities. All personnel decisions, including, but not limited to, recruiting, hiring, training, promotion, compensation, benefits, and termination, are made without regard to race, color, religion, age, sex, sexual orientation, pregnancy, gender identity, genetic information, national origin, citizenship status, veteran status, protected veteran status, disability, or any other characteristic protected by applicable federal, state, or local law.

Reasonable accommodations for applicants and employees with disabilities will be provided. If a reasonable accommodation is needed to participate in the job application or interview process, to perform essential job functions, and/or to receive other benefits and privileges of employment, or to receive an update regarding the status of your application, please contact Talent Acquisition by emailing [e-mail], or by dialing 703-575-8400.