Senior Databricks & AI Architect/Lead - Only W2 - (JK)

Central Business Solutions, Inc; Dallas, New York (TX, NY); On-site; USD 150,000 - 190,000; Full time

Job summary

Central Business Solutions, Inc. is seeking a Senior Databricks & AI Architect/Lead to drive the evolution of a modern data platform and AI roadmap. You will design scalable Spark-based architectures and govern end-to-end data flows.

Initial onsite in Dallas, with potential to hybrid later, leading data engineering teams and shaping AI workflows using Agentic AI, LangChain, and RAG patterns for complex data domains.

Qualifications

• 8+ years in Data Engineering/Architecture with 3+ years in Databricks leadership.
• Expert Spark (PySpark/Scala), Delta Lake, and Databricks SQL.
• Proven AI solution deployment; Agentic AI and LangChain/LlamaIndex.
• Strong data modeling; reusable ETL/ELT templates.

Responsibilities

• Architectural Leadership: design and govern end-to-end data architectures on the Databricks platform.
• AI Innovation: lead design of AI workflows, including multi-agent systems and LLM orchestration.
• Data Engineering Excellence: standardise templates and robust data modelling for structured/unstructured data.
• Governance & Security: ensure designs comply with regulations using Unity Catalog.

Skills

Spark framework

PySpark

Scala

Databricks SQL

Delta Lake

Agentic AI

LangChain

LLM orchestration

Data modeling

ETL/ELT templates

Tools

Unity Catalog

Job description

Senior Databricks & AI Architect/Lead - Only W2 - (JK)

• Dallas, CA, TX, NJ, NY

Title: Senior Databricks & AI Architect/Lead

Location: Dallas, TX/NJ/NY/ California (Initial onsite and can converted to Hybrid then after)

Duration: Long Term contract role

We are seeking a visionary Senior Databricks & AI Architect to lead the evolution of our modern data platform and AI roadmap. You will be responsible for designing scalable Spark-based architectures.

Key Responsibilities

• Architectural Leadership: Design and govern end-to-end data architectures on the Databricks Data Intelligence Platform.
• AI Innovation: Lead the design of emerging AI workflows, specifically focusing on Agentic AI (multi-agent systems), LLM orchestration (LangChain/LlamaIndex), and RAG patterns tailored for insurance policy and regulatory documents.
• Data Engineering Excellence: Standardise data engineering templates (Medallion architecture) and implement robust data modelling (Star/Snowflake, Vault 2.0) to handle structured and unstructured insurance data.
• Governance & Security: Ensure all designs comply with industry regulations using Databricks Unity Catalog.

Required Skills & Experience

• Experience: 8+ years in Data Engineering/Architecture, with at least 3+ years of hands-on leadership within the Databricks ecosystem.
• Core Tech: Expert-level knowledge of the Spark framework (PySpark/Scala), Delta Lake, and Databricks SQL.
• AI/ML Expertise: Proven experience building and deploying AI solutions. Deep understanding of the Agentic AI landscape-moving beyond simple chatbots to autonomous, goal-oriented agents.
• Data Management: Strong background in data modelling and the creation of reusable ETL/ELT templates and frameworks.