Lead Data Architect

Karsun Solutions, LLC; Maryland; Hybrid; USD 160,000 - 190,000; Full time

Job summary

Karsun Solutions, LLC is looking for a Lead Data Architect in Maryland to design and operate enterprise data platforms that power GenAI and AI/ML use cases. This role requires hands-on expertise in data architecture, data engineering, and ML pipeline design using Databricks.

The ideal candidate will have 8+ years of experience and must hold an AWS Certified Machine Learning certification. The salary range for this position is between $160,000 and $190,000, reflecting various experience levels.

Qualifications

• 8+ years hands‑on experience in data engineering/platform architecture; 3+ years in an architect or lead role.
• Proven experience designing Databricks workspaces, Delta Lake, performance tuning.
• Strong experience building ML/LLM pipelines and operationalizing models.

Responsibilities

• Architect and implement enterprise data platforms optimized for ML, LLMs, and GenAI workloads.
• Lead design and hands‑on implementation of Databricks workspaces.
• Mentor and lead engineering teams; conduct architecture reviews and code reviews.

Skills

Data engineering/platform architecture

AWS Certified Machine Learning – Specialty

Databricks

Python

Spark + PySpark

ML/LLM pipelines

SQL

Education

BA or BS degree in CS or related field

Tools

Databricks Runtime

Terraform

CI/CD tools

Job description

Summary

The Lead Data Architect will design, build, and operate enterprise data platforms that power GenAI and AI/ML use cases. This highly technical, hands‑on role is responsible for data platform architecture, end‑to‑end data engineering, ML/LLM pipeline design, production model onboarding, and delivery of scalable Databricks‑centric solutions across cloud environments.

What You’ll Be Doing

• Architect and implement enterprise data platforms (batch + streaming) optimized for ML, LLMs, and GenAI workloads.
• Lead design and hands‑on implementation of Databricks workspaces, Unity Catalog, Delta Lake design patterns, cluster policies, and performance tuning.
• Build and own end‑to‑end data pipelines (ingest, transform, feature engineering, serving) using PySpark, Databricks Jobs, Spark SQL, Delta Lake, and orchestration tools.
• Design and operationalize model training, fine‑tuning (LLM), evaluation, deployment, and monitoring pipelines (MLOps/RAG/CAG) integrating Databricks MLflow, CI/CD, and infra‑as‑code.
• Implement vectorless and vectorization/embedding pipelines, vector store integrations, and retrieval layers for RAG (FAISS, Pinecone, Weaviate, Milvus).
• Define data schemas, governance, lineage, access controls, and data product APIs; implement Unity Catalog or equivalent for centralized governance.
• Drive cost/performance optimization for storage, compute (spot/preemptible), and query patterns.
• Collaborate with engineers, data scientists, product owners, and security to translate business needs into production GenAI solutions.
• Mentor and lead engineering teams; conduct architecture reviews, code reviews, and run technical deep dives.
• Implement observability for data and ML pipelines (metrics, logging, data quality tests, alerting).
• Create reproducible experiment tracking, model registry, and rollout strategies (canary, shadow testing, rollback).
• Stay current on GenAI/LLM architectures and evaluate/introduce new tooling and frameworks.

Required Qualifications

• BA or BS degree in CS, Computer Engineering, Information Technology or a related field.
• 8+ years hands‑on experience in data engineering/platform architecture; 3+ years in an architect or lead role.
• Candidate must hold an active AWS Certified Machine Learning – Specialty certification.
• Proven, hands‑on Databricks experience (designing workspaces, Delta Lake, performance tuning, productionizing Spark jobs).
• Deep Spark + PySpark expertise and experience with Databricks Runtime.
• Strong experience building ML/LLM pipelines and operationalizing models (training, fine‑tuning, serving).
• Practical experience with vector embeddings, semantic search, and RAG architectures.
• Solid Python expertise and common ML libraries (PyTorch, TensorFlow, Hugging Face transformers) and MLflow.
• Cloud platform experience (AWS strongly preferred).
• Experience with containerization and orchestration while leveraging open source libraries for unstructured and structured data processing, serving/inference.
• Strong SQL skills; experience with distributed query/warehouse systems and parquet/AVRO/Delta formats.
• CI/CD and infra‑as‑code experience (Terraform, GitOps, Jenkins/GitHub Actions/GitLab CI).
• Data governance, security, and IAM experience; experience implementing row/column level access controls and data lineage.
• Demonstrated ability to design for scalability, reliability, and cost efficiency.

Preferred Qualifications

• Prior experience with Databricks Unity Catalog, Photon, and Databricks SQL.
• Experience integrating Databricks with vector databases (Pinecone, Neo4j) and retrieval frameworks (LangChain, LlamaIndex).
• Familiarity with AWS Bedrock or other managed LLM services.
• Experience with realtime streaming (Kafka, Kinesis) and stream processing on Databricks Structured Streaming.
• Certifications: Databricks Certified Professional.
• Experience with data quality and profiling tools (Great Expectations, Soda).
• Experience with large‑scale ETL frameworks and tools (Airflow, Prefect).

Commitment to Non‑Discrimination

All qualified applicants will receive consideration for employment without regard to disability, status as a protected veteran or any other status protected by applicable federal, state, local, or international law.

Salary Range

The proposed salary range for this role is $160,000 to $190,000 USD. This range reflects experience levels and other factors considered by Karsun.

Clearance Information

This position requires eligibility to obtain a security clearance. The Defense Industrial Security Clearance Office (DISCO) handles and adjudicates the clearance process. More information about Security Clearances can be found on the U.S. Department of State website: https://www.state.gov/mds/clearances/c10978.htm

Location

To be considered for this role, you must reside in one of the following states: CA, CO, DC, FL, GA, IL, MD, NJ, NY, NC, OH, OK, PA, SC, TX, VA, WV.

Applicants must be authorized to work in the U.S. We may consider candidates currently in H‑1B status who are eligible for transfer.