Databricks Architect (Lead Data Platform Architect)

iLink Digital; New York (NY); On-site; USD 180,000 - 240,000; Full time

Job summary

iLink Digital is seeking a Databricks Architect (Lead Data Platform Architect) in New York, NY to design, build, and scale enterprise-grade Lakehouse data platforms. You will lead architecture strategy, platform standardization, and modernization initiatives leveraging Databricks across cloud ecosystems.

The ideal candidate brings deep expertise in Spark, Delta Lake, and cloud-native architecture, with proven leadership in large-scale data transformations.

Qualifications

• 12+ years in data engineering, architecture, or platform design.
• Strong leadership with proven enterprise data platform delivery.
• Deep expertise in Spark, Delta Lake, and cloud-native architectures.

Responsibilities

• Define end-to-end Databricks Lakehouse architecture.
• Design scalable batch and real-time data processing systems.
• Lead deployment across Azure/AWS/GCP Databricks environments.
• Establish standards for workspace design, cluster strategy, and data storage.
• Drive governance, security, and data privacy compliance.
• Mentor data engineers, scientists, and TPMs on best practices.

Skills

Leadership

Architecture design

Cloud architecture

Databricks

Spark/PySpark optimization

Data lakehouse

Education

Bachelor's degree in CS/Engineering or equivalent

Tools

Databricks

Delta Lake

Azure/AWS/GCP Databricks

Unity Catalog

Job description

Databricks Architect (Lead Data Platform Architect)

New York, United States | Posted on 06/25/2026

We are looking for a highly skilled Databricks Architect to design, build, and scale enterprise-grade Lakehouse data platforms. This role will drive architecture strategy, platform standardization, and enterprise data modernization initiatives, leveraging Databricks and cloud ecosystems.

The ideal candidate brings deep expertise in Spark, Delta Lake, and cloud-native architecture, along with strong leadership in driving large-scale data transformations.

Key Responsibilities

• Define and implement end-to-end Databricks Lakehouse architecture.
  
• Design scalable systems for:
  • Batch & real-time data processing
    
  • Structured & unstructured workloads
    
• Establish medallion architecture (Bronze, Silver, Gold layers) as a standard.
  
• Lead deployment and optimization of:
  • Azure Databricks / AWS Databricks / GCP Databricks
    
• Define standards for:
  • Workspace design & cluster strategy
    
  • Job orchestration
    
  • Data storage (Delta Lake)
    
• Drive adoption of:
  • Unity Catalog
    
  • MLflow
    
  • Databricks SQL & Photon
    
• Batch (ADF, Airflow)
  
• Data modeling (star schema, data vault, dimensional models)
  
• Guide engineering teams on best practices in Spark/PySpark optimization.
  

Performance & Cost Optimization

• Query performance
  
• Implement cost governance strategies (auto-scaling, job clusters, spot instances).
  

Data Governance & Security

• Fine-grained access control (RBAC/ABAC)
  
• Ensure compliance with data privacy and regulatory standards.
  
• Integrate Databricks with:
  • Data sources (ERP, CRM, APIs, IoT)
    
  • BI tools (Power BI, Tableau)
    
  • ML pipelines and AI platforms
    
• Collaborate with cloud architects for:
  • Networking, security, and storage strategies.
    
• Provide architectural guidance to data engineers, scientists, and TPMs.
  
• Conduct design reviews and enforce architecture governance.
  
• Mentor teams on emerging patterns:
  • Data Mesh
    
  • DataOps / MLOps
    
  • GenAI workloads on Databricks
    

Skills & Qualifications

Mandatory Skills

• 12+ years of experience in data engineering, architecture, or platform design.