Lead GenAI Data Architect (Databricks)

Karsun Solutions, LLC; Ohio; On-site; USD 160,000 - 175,000; Full time

Job summary

Karsun Solutions, LLC is seeking a Lead Data Architect to design, build, and operate enterprise data platforms powering GenAI and AI/ML use cases. This hands-on role spans data platform architecture, end-to-end data engineering, ML/LLM pipelines, and production-ready Databricks-centric solutions across cloud environments.

You will mentor teams, implement observability, CI/CD, and governance while optimizing cost and performance.

Qualifications

• BA/BS in CS, Computer Engineering, IT, or related field.
• 8+ years data engineering/platform architecture; 3+ years in lead/architect roles.
• Active AWS ML – Specialty or equivalent AWS cert.
• Hands-on Databricks experience: workspaces, Delta Lake, performance tuning.
• Deep Spark + PySpark with Databricks Runtime.
• Experience building ML/LLM pipelines and deploying models.
• Vector embeddings, semantic search, and RAG architectures.
• Python with ML libraries (PyTorch, TensorFlow, HuggingFace) and MLflow.
• CI/CD and infra-as-code (Terraform, GitOps, Jenkins/GitHub Actions).
• Data governance, IAM, lineage; security-focused.
• Design for scalability, reliability, and cost efficiency.

Responsibilities

• Architect and implement enterprise data platforms (batch + streaming) optimized for ML, LLMs, and GenAI workloads.
• Lead design and hands-on implementation of Databricks workspaces, Unity Catalog, Delta Lake patterns, cluster policies, and performance tuning.
• Build end-to-end data pipelines (ingest, transform, feature engineering, serving) using PySpark, Databricks Jobs, Spark SQL, Delta Lake, and orchestration tools.
• Design and operationalize model training, fine tuning (LLM), evaluation, deployment, and monitoring pipelines (MLOps/RAG/CAG) integrating Databricks MLflow, CI/CD, and infra-as-code.
• Implement vectorless and vectorization/embedding pipelines, vector store integrations, and retrieval layers for RAG (FAISS, Pinecone, Weaviate, Milvus).
• Define data schemas, governance, lineage, access controls, and data product APIs; implement Unity Catalog or equivalent for centralized governance.
• Drive cost/performance optimization for storage, compute (spot/preemptible),and query patterns.
• Collaborate with engineers, data scientists, product owners, and security to translate business needs into production GenAI solutions.
• Mentor and lead engineering teams; conduct architecture reviews, code reviews, and run technical deep dives.
• Implement observability for data and ML pipelines (metrics, logging, data quality tests, alerting).
• Create reproducible experiment tracking, model registry, and rollout strategies (canary, shadow testing, rollback).
• Stay current on GenAI/LLM architectures and evaluate/introduce new tooling and frameworks.

Skills

Databricks

PySpark

Python

ML/LLM pipelines

Vector embeddings

Semantic search

RAG architectures

SQL

CI/CD

Terraform

GitOps

Jenkins/GitHub Actions/GitLab CI

Security IAM

Data governance

Cost optimization

Education

Bachelor's degree in CS/CE/IT or related field

Tools

Delta Lake

Unity Catalog

Photon

Databricks SQL

Pinecone

neo4j

Weaviate

Milvus

LangChain

LlamaIndex

Airflow

Job description

Karsun Solutions, LLC is seeking a Lead Data Architect to design, build, and operate enterprise data platforms powering GenAI and AI/ML use cases. This hands-on role spans data platform architecture, end-to-end data engineering, ML/LLM pipelines, and production-ready Databricks-centric solutions across cloud environments.

You will mentor teams, implement observability, CI/CD, and governance while optimizing cost and performance.