Senior Databricks Architect for Data, AI & Streaming Platform

Valiant Technologies LLC; San Francisco (CA); In loco; USD 83.000 - 118.000; Tempo pieno

Descrizione del lavoro

Valiant Technologies LLC in San Francisco seeks a Databricks Solution Architect to own end-to-end data and ML architecture on Databricks. You will lead stakeholder workshops, convert business requirements into reference architectures, and drive delivery roadmaps for secure, compliant analytics platforms.

The role emphasizes medallion data architecture, Unity Catalog governance, and cost-aware FinOps. Collaboration with security, data teams, and vendors is essential to delivering scalable

Competenze

• 7+ years in data engineering/architecture with 5+ years on Databricks.
• Expertise in Spark, Delta Lake, SQL and medallion modeling.
• Strong AWS architecture knowledge and multi-account design.
• Experience with Unity Catalog, data security, and governance.
• Proven migration of Hadoop/Hive and warehouse workloads.
• Familiar with Kafka, Structured Streaming, and disaster recovery.

Mansioni

• Define target-state architecture across ingestion, medallion layers, serving and ML on Databricks.
• Lead stakeholder discovery; translate business requirements into designs and roadmaps.
• Design marketing data solutions: identity resolution, segmentation and campaign analytics.
• Architect collections data for delinquency, integration with agencies, and compliance.
• Define batch/streaming patterns, multi-account AWS networking and DR plans.
• Establish security/governance with Unity Catalog, encryption, logging and data lineage.
• Plan migration from Hadoop/Hive/EMR to modern platforms; develop cost models.
• Lead reviews, vendor evaluations, PoCs, and architecture standards.

Conoscenze

7+ years data engineering/architecture

5+ years with Databricks

Apache Spark

Delta Lake

SQL

Medallion architecture

AWS, multi-account design

Unity Catalog and data governance

Data security and PII/NPI protection

Kafka and Spark Structured Streaming

FinOps / cost optimization

Strumenti

Databricks

Unity Catalog

AWS (Networking, IAM, Encryption)

Descrizione del lavoro

Valiant Technologies LLC in San Francisco seeks a Databricks Solution Architect to own end-to-end data and ML architecture on Databricks. You will lead stakeholder workshops, convert business requirements into reference architectures, and drive delivery roadmaps for secure, compliant analytics platforms.

The role emphasizes medallion data architecture, Unity Catalog governance, and cost-aware FinOps. Collaboration with security, data teams, and vendors is essential to delivering scalable