Senior Data Architect: AI-Ready Big Data & Databricks

West Monroe (Campus); Los Angeles (CA); Hybrid; USD 205,000 - 234,000; Full time

Job summary

West Monroe in Los Angeles is seeking a Senior Architect to join our Technology and Experience Practice. You will design data pipelines and data architectures, with emphasis on Databricks, Spark, and AI readiness, collaborating with data science and engineering teams to productionize AI workloads.

The role demands 10+ years of professional experience and 8+ years in consulting, deep Databricks expertise, and leadership to coach consultants.

Qualifications

• 10+ years of professional experience, with 3+ years in Databricks.
• 8+ years of prior consulting experience.
• Deep Databricks architecture, performance optimization, Delta Lake, Unity Catalog, and security.
• Experience deploying AI/ML or Generative AI in enterprise environments.
• Hands-on with LLM ecosystems and frameworks such as OpenAI, Azure OpenAI, LangChain, Semantic Kernel, MLflow, or Databricks Mosaic AI.
• Knowledge of Retrieval-Augmented Generation (RAG), vector databases, embeddings, prompt engineering, AI orchestration.
• Familiarity with AI governance, model lifecycle management, responsible AI principles, and enterprise security considerations.
• Experience integrating AI into data platforms, analytics solutions, or customer-facing applications.
• Knowledge of standard methodologies, concepts, and best practices in Big Data.
• Travel up to 30-50% and hybrid in-office 1-2 days.

Responsibilities

• Design data flow and processing pipelines for analysis using Spark on Scala, Storm, Flume, Sqoop
• Design data structures for ingestion and reporting, specific to use case and technology
• Provide data management expertise to evaluate requirements and develop data architecture and platform selection
• Collaborate with data science and engineering teams to productionize AI workloads within cloud ecosystems
• Establish best practices for AI data readiness, feature engineering, model monitoring, observability, and governance
• Lead technical discussions around AI-enabled modernization strategies including semantic layers, vector databases, AI agents, and knowledge retrieval systems
• Evaluate emerging AI technologies and recommend fit-for-purpose tooling aligned to client objectives, scalability, security, and compliance requirements
• Guide and coach consultants, share best practices, perform code reviews
• Coordinate activities with data source application owners to ensure integration with analytics platform and source systems
• Ensure that data integrity issues are addressed
• Liaise and attend meetings necessary to perform duties and aid business and technical development

Skills

Databricks expertise

Big Data

AI/ML integration

LLM ecosystems

Data governance

Cloud data engineering

Data architecture

Leadership coaching

Tools

Spark/Scala

Storm

Flume

Sqoop

Delta Lake

Unity Catalog

MLflow

Databricks Mosaic AI

Vector databases

Job description

West Monroe in Los Angeles is seeking a Senior Architect to join our Technology and Experience Practice. You will design data pipelines and data architectures, with emphasis on Databricks, Spark, and AI readiness, collaborating with data science and engineering teams to productionize AI workloads.

The role demands 10+ years of professional experience and 8+ years in consulting, deep Databricks expertise, and leadership to coach consultants.