Databricks Solution Architect: Data & ML Platform Lead

Cloud Analytics Technologies, LLC; San Francisco (CA); In loco; USD 82.000 - 117.000; Tempo pieno

Descrizione del lavoro

Cloud Analytics Technologies, LLC in San Francisco, CA seeks a Databricks Solution Architect to define target-state architecture for data ingestion, medallion layers, and ML workloads on Databricks. You will lead stakeholder discovery, translate business needs into reference designs and roadmaps.

Required 7+ years in data engineering/architecture and 5+ years with Databricks, with strong Spark, Delta Lake, and AWS experience, plus governance and migration expertise.

Competenze

• 7+ years in data engineering or architecture, including 5+ years with Databricks.
• Strong expertise in Apache Spark, Delta Lake, SQL, medallion architecture, and dimensional data modeling.
• Strong AWS architecture knowledge, including networking, IAM, encryption, and multi-account design.
• Experience with Unity Catalog, data security, governance, and PII/NPI protection.
• Proven experience migrating legacy Hadoop/Hive and warehouse workloads.
• Knowledge of Kafka, Spark Structured Streaming, batch processing, disaster recovery, and high availability.
• Experience with Databricks cost optimization, FinOps, stakeholder management, and architecture documentation.

Mansioni

• Define target-state architecture covering data ingestion, medallion architecture, data serving, and ML workloads on Databricks.
• Lead stakeholder discovery, translate business requirements into solution designs, reference architectures, and delivery roadmaps.
• Design marketing data solutions for customer identity resolution, segmentation, campaign measurement, and compliant data sharing.
• Architect collections data solutions for delinquency monitoring, contact data, agency/vendor integrations, and regulatory compliance.
• Define batch and streaming patterns, AWS networking, IAM, multi-account architecture, high availability, and disaster recovery.
• Establish security and governance using Unity Catalog, PII/NPI protection, encryption, audit logging, and data lineage.
• Plan migration from legacy Hadoop, Hive, EMR, and data warehouse platforms.
• Develop Databricks cost and capacity models, FinOps controls, and platform investment business cases.
• Lead architecture reviews, vendor evaluations, proofs of concept, and architecture standards.

Conoscenze

Databricks

Apache Spark

Delta Lake

SQL

AWS

Data modeling

Medallion architecture

Security governance

Strumenti

Unity Catalog

Kafka

EMR/Hadoop migration

Descrizione del lavoro

Cloud Analytics Technologies, LLC in San Francisco, CA seeks a Databricks Solution Architect to define target-state architecture for data ingestion, medallion layers, and ML workloads on Databricks. You will lead stakeholder discovery, translate business needs into reference designs and roadmaps.

Required 7+ years in data engineering/architecture and 5+ years with Databricks, with strong Spark, Delta Lake, and AWS experience, plus governance and migration expertise.