Databricks Solution Architect

Centraprise; San Francisco (CA); On-site; USD 180,000 - 260,000; Full time

Job summary

Centraprise is seeking a senior data architecture leader to design and govern data platforms on Databricks in a multi-account AWS environment. The role concentrates on ingest, storage layers (Bronze/Silver/Gold), and ML workloads, with a strong emphasis on governance and security.

You will translate business needs into scalable architectures, drive roadmaps, and lead architecture reviews with stakeholders and vendors. This is a high-impact, on-site SF position with competitive compensation.

Qualifications

• 12+ years in data engineering or architecture with 5+ years on Databricks.
• Strong expertise in Spark, Delta Lake, SQL, medallion architecture, and dimensional modeling.
• Deep AWS knowledge: networking, IAM, encryption, multi-account design.
• Experience with Unity Catalog, data security, governance, and PII/NPI protection.
• Proven track record migrating legacy Hadoop/Hive and warehouse workloads.
• Knowledge of Kafka, streaming and batch processing, DR, and high availability.
• Experience with Databricks cost optimization, FinOps, and architecture docs.

Responsibilities

• Define target-state architecture for data ingestion, medallion layers, data serving, and ML workloads on Databricks.
• Lead stakeholder discovery, translate requirements into solution designs and roadmaps.
• Design marketing data solutions for identity resolution, segmentation, and measurement.
• Architect collections data solutions for delinquency monitoring and regulatory compliance.
• Define batch/streaming patterns, AWS networking, IAM, and multi-account architecture.
• Establish security and governance using Unity Catalog, encryption, audit logs, and lineage.
• Plan migration from legacy Hadoop/Hive/EMR and data warehouses.
• Develop Databricks cost models, FinOps controls, and platform investment cases.
• Lead architecture reviews, vendor evaluations, PoCs, and architecture standards.

Skills

Databricks

Apache Spark

Delta Lake

Medallion Architecture

Dimensional Data Modeling

AWS Architecture (multi-account)

Data Governance

Unity Catalog

KSQL / SQL proficiency

Stakeholder Management

Tools

Databricks

Kafka

Spark Structured Streaming

MLflow

Job description

• Define target-state architecture covering data ingestion, medallion architecture (Bronze/Silver/Gold), data serving, and ML workloads on Databricks.
• Lead stakeholder discovery, translate business requirements into solution designs, reference architectures, and delivery roadmaps.
• Design marketing data solutions for customer identity resolution, segmentation, campaign measurement, and compliant data sharing.
• Architect collections data solutions for delinquency monitoring, contact data, agency/vendor integrations, and regulatory compliance.
• Define batch and streaming patterns, AWS networking, IAM, multi-account architecture, high availability, and disaster recovery.
• Establish security and governance using Unity Catalog, PII/NPI protection, encryption, audit logging, and data lineage.
• Plan migration from legacy Hadoop, Hive, EMR, and data warehouse platforms.
• Develop Databricks cost and capacity models, FinOps controls, and platform investment business cases.
• Lead architecture reviews, vendor evaluations, proofs of concept, and architecture standards.

Must-Have Skills

• 12+ years in data engineering or architecture, including 5+ years with Databricks.
• Strong expertise in Apache Spark, Delta Lake, SQL, medallion architecture, and dimensional data modeling.
• Strong AWS architecture knowledge, including networking, IAM, encryption, and multi-account design.
• Experience with Unity Catalog, data security, governance, and PII/NPI protection.
• Proven experience migrating legacy Hadoop/Hive and warehouse workloads.
• Knowledge of Kafka, Spark Structured Streaming, batch processing, disaster recovery, and high availability.
• Experience with Databricks cost optimization, FinOps, stakeholder management, and architecture documentation.

Good to Have

• Banking, lending, or regulated financial services experience.
• Marketing data architecture, Customer 360, consent management, or campaign analytics.
• Collections data, delinquency, agency, and dialer feed integrations.
• Databricks ML/MLOps, MLflow, feature stores, and Databricks/AWS certifications.