Lead GenAI Data Platform Architect

Karsun Solutions, LLC; New York (NY); On-site; USD 160,000 - 175,000; Full time

Job summary

Karsun Solutions, LLC is seeking a Lead Data Architect to design, build, and operate enterprise data platforms powering GenAI and AI/ML use cases. This is a highly technical, hands-on role focused on data platform architecture, end-to-end data engineering, ML/LLM pipelines, and scalable Databricks-centric solutions across cloud environments.

The candidate will mentor engineering teams, lead architecture reviews, implement robust MLOps pipelines, and drive cost/performance optimization while

Qualifications

• 8+ years hands-on experience in data engineering and platform architecture, with 3+ years in lead/architect roles.
• Active AWS Machine Learning - Specialty certification or equivalent AWS cert.
• Proven Databricks design experience (workspaces, Delta Lake, performance tuning, productionizing Spark jobs).
• Strong Python expertise and ML libraries (PyTorch, TensorFlow) and MLflow.

Responsibilities

• Architect and build enterprise data platforms (batch + streaming) optimized for ML/LLMs and GenAI workloads.
• Lead Databricks workspace design, Unity Catalog, Delta Lake patterns, and performance tuning.
• Develop end-to-end data pipelines (ingest, transform, feature engineering, serving) using PySpark, Databricks Jobs, Spark SQL, Delta Lake.
• Design and operationalize model training, fine-tuning, deployment, and monitoring pipelines (MLOps/RAG/CAG).
• Implement vectorless and vectorization/embedding pipelines and retrieval layers for RAG (FAISS, Pinecone, Weaviate, Milvus).
• Define data schemas, governance, lineage, access controls, and data product APIs; implement Unity Catalog for governance.
• Drive cost/performance optimization for storage, compute, and queries; ensure security and IAM compliance.
• Collaborate with engineers, data scientists, product owners, and security to deliver production GenAI solutions.
• Mentor teams; conduct architecture reviews, code reviews, and deep-dive sessions.
• Establish observability for data and ML pipelines (metrics, logging, data quality tests, alerts).
• Create reproducible experiment tracking, model registry, and rollout strategies (canary, shadow testing, rollback).
• Stay current on GenAI/LLM architectures and evaluate new tooling.

Skills

Databricks

Python

Spark/PySpark

ML/LLM pipelines

SQL

AWS

CI/CD / IaC

Security / IAM

Data governance

Vector embeddings

Education

BA/BS in CS/CE/IT

Tools

Databricks Runtime

Delta Lake

Unity Catalog

Terraform

Kubernetes

Job description

Karsun Solutions, LLC is seeking a Lead Data Architect to design, build, and operate enterprise data platforms powering GenAI and AI/ML use cases. This is a highly technical, hands-on role focused on data platform architecture, end-to-end data engineering, ML/LLM pipelines, and scalable Databricks-centric solutions across cloud environments.

The candidate will mentor engineering teams, lead architecture reviews, implement robust MLOps pipelines, and drive cost/performance optimization while