Solution Architect, Data Platform (Databricks)

EXL; San Francisco (CA); Hybrid; USD 180,000 - 240,000; Full time

Job summary

EXL is seeking a Solution Architect for Data Platform on Databricks in San Francisco. The role owns end-to-end design across ingestion, medallion layering, serving, and ML, aligning with risk, fraud, finance, marketing, and collections needs.

You will drive architectural decisions, roadmap creation, and cross-team collaboration with engineering and vendors. The candidate will guide migration from legacy Hadoop/Hive/EMR to a scalable data platform, shaping security, governance, and cost models

Qualifications

• 12+ years in data engineering or architecture with 5+ years on Databricks lakehouse.
• Strong Spark, Delta Lake, SQL and data modelling experience.
• Experience with AWS networking, IAM, encryption and multi-account design.
• Secure design for PII/NPI with lineage and audit controls.
• Migration planning from Hadoop/Hive/EMR to modern platforms.
• Ability to decide batch vs. streaming and hybrid architectures.

Responsibilities

• Define enterprise data architecture on Databricks including ingestion, medallion layering, serving and ML.
• Run discovery sessions and translate requirements into solution designs and roadmaps.
• Design identities, segmentation and compliant data sharing for marketing platforms.
• Plan near-real-time delinquency data flows and vendor file exchanges.
• Make cross-cutting decisions on tech stack, DR, and cost models.
• Lead security architecture with Unity Catalog, encryption and lineage.
• Plan migration from legacy systems to the new platform.
• Build business case and FinOps guardrails for platform investments.
• Chair architecture review boards and maintain standards and references.
• Manage Databricks and cloud provider vendor relationships and evaluations.

Skills

Databricks expertise

12+ years experience

Stakeholder management

Vendor management

Communication

Cost modeling

Architectural decision making

Tools

Databricks

AWS

Kafka

Structured Streaming

SQL

Spark

Unity Catalog

FinOps

Job description

Role: Solution Architect, Data Platform (Databricks)

Location: San Francisco, CA (Hybrid 3 days in client office)

Experience: 12+ years

Duration: Full time

About the Role

We are building our modern data platform on Databricks and need a Solution Architect to own the end-to-end design. You will turn business needs from Risk, Fraud, Finance, Marketing, Collections and Product into scalable, secure data architectures that integrate cleanly with existing systems. You will own architectural decisions across teams and work closely with the Principal Engineer, who turns the architecture into detailed designs and working pipelines.

Key Responsibilities

• Target-state architecture: Define the enterprise data architecture on Databricks: ingestion, medallion layering, serving and ML, and how they connect to source systems, the legacy warehouse, loan servicing platforms and vendor feeds.
• Discovery and roadmaps: Run discovery with business and technology stakeholders and turn requirements into solution designs, reference architectures and delivery roadmaps.
• Marketing architecture: Design for identity resolution, segmentation, campaign and attribution measurement, and compliant data sharing with marketing platforms and vendors.
• Collections architecture: Design for near-real-time delinquency and contact data, agency and vendor file exchange, and compliant handling of contact preferences and regulatory restrictions.
• Cross-cutting decisions: Decide on batch vs. streaming, build vs. buy, workspace topology, network and multi-account design, and disaster recovery.
• Security architecture: Design the security and compliance architecture with InfoSec: Unity Catalog access model, PII/NPI protection, encryption, audit logging and lineage.
• Migration planning: Plan and sequence the migration of legacy Hadoop/Hive/EMR and warehouse workloads, including cutover strategy and risk mitigation.
• Cost and business case: Build the cost and capacity model (DBU forecasting, FinOps guardrails) and the business case for platform investments.
• Governance: Chair the architecture review board and maintain standards, reference patterns and decision records.
• Vendors: Manage the relationship with the Databricks account team and cloud provider, and lead vendor evaluations and proofs of concept.

Must-Have Skills (candidate must meet all)

• Experience: 12+ years in data engineering or architecture, with 5+ years on Databricks lakehouse platform.
• Data platform fundamentals: Strong Spark, Delta Lake and SQL, plus data modeling (medallion, dimensional).
• AWS architecture: Networking, IAM, encryption and multi-account design on AWS.
• Secure design: Platform design for PII/NPI, including Unity Catalog access models and lineage.
• Migration: Migration planning from legacy Hadoop/Hive/warehouse systems.
• Streaming and batch design: Ability to choose between batch and streaming (Kafka, Structured Streaming) and design for both.
• Resilience: Disaster recovery and high-availability design.
• Cost modeling: FinOps and Databricks cost and capacity modeling, including a business case.
• Communication: Stakeholder management, vendor management and clear architecture documentation.

Good-to-Have Skills (preferred, not required)

• Banking, lending or other regulated financial services experience
• Collections data architecture (delinquency, agency and dialer feeds)
• ML/MLOps architecture on Databricks (feature store, MLflow)
• Databricks and AWS Solution Architect certifications