Databricks Solutions Architect: Lead Data & ML Platform

Centraprise; San Francisco (CA); On-site; USD 180,000 - 260,000; Full time

Job summary

Centraprise is seeking a senior data architecture leader to design and govern data platforms on Databricks in a multi-account AWS environment. The role concentrates on ingest, storage layers (Bronze/Silver/Gold), and ML workloads, with a strong emphasis on governance and security.

You will translate business needs into scalable architectures, drive roadmaps, and lead architecture reviews with stakeholders and vendors. This is a high-impact, on-site SF position with competitive compensation.

Qualifications

• 12+ years in data engineering or architecture with 5+ years on Databricks.
• Strong expertise in Spark, Delta Lake, SQL, medallion architecture, and dimensional modeling.
• Deep AWS knowledge: networking, IAM, encryption, multi-account design.
• Experience with Unity Catalog, data security, governance, and PII/NPI protection.
• Proven track record migrating legacy Hadoop/Hive and warehouse workloads.
• Knowledge of Kafka, streaming and batch processing, DR, and high availability.
• Experience with Databricks cost optimization, FinOps, and architecture docs.

Responsibilities

• Define target-state architecture for data ingestion, medallion layers, data serving, and ML workloads on Databricks.
• Lead stakeholder discovery, translate requirements into solution designs and roadmaps.
• Design marketing data solutions for identity resolution, segmentation, and measurement.
• Architect collections data solutions for delinquency monitoring and regulatory compliance.
• Define batch/streaming patterns, AWS networking, IAM, and multi-account architecture.
• Establish security and governance using Unity Catalog, encryption, audit logs, and lineage.
• Plan migration from legacy Hadoop/Hive/EMR and data warehouses.
• Develop Databricks cost models, FinOps controls, and platform investment cases.
• Lead architecture reviews, vendor evaluations, PoCs, and architecture standards.

Skills

Databricks

Apache Spark

Delta Lake

Medallion Architecture

Dimensional Data Modeling

AWS Architecture (multi-account)

Data Governance

Unity Catalog

KSQL / SQL proficiency

Stakeholder Management

Tools

Databricks

Kafka

Spark Structured Streaming

MLflow

Job description

Centraprise is seeking a senior data architecture leader to design and govern data platforms on Databricks in a multi-account AWS environment. The role concentrates on ingest, storage layers (Bronze/Silver/Gold), and ML workloads, with a strong emphasis on governance and security.

You will translate business needs into scalable architectures, drive roadmaps, and lead architecture reviews with stakeholders and vendors. This is a high-impact, on-site SF position with competitive compensation.