Senior Databricks Architect

Arkhya Tech Inc.; New York (NY); On-site; USD 180,000 - 230,000; Full time

Job summary

Arkhya Tech Inc. in New York seeks a Databricks Lakehouse Architect with hands-on data engineering experience designing and implementing production-grade Databricks solutions on AWS.

The candidate will lead enterprise data platform modernization and migration from legacy ETL to Databricks. You will architect scalable Lakehouse solutions (Bronze–Silver–Gold) with Delta Lake, Unity Catalog, and Workflows; design batch, streaming, and real-time pipelines using PySpark, Spark SQL, Auto Loader, and

Qualifications

• Hands-on experience with Databricks Lakehouse platforms in production environments.
• Proficiency in PySpark, Spark SQL, Delta Lake, and Unity Catalog.
• Experience migrating from EMR/Redshift/NiFi to Databricks on AWS.
• Knowledge of data governance, security, and lineage concepts.

Responsibilities

• Lead enterprise data platform modernization and Databricks migration strategy.
• Architect Lakehouse solutions using Bronze/Silver/Gold layers with Delta Lake.
• Design batch, streaming, CDC, and real-time pipelines with PySpark and Auto Loader.
• Define standards for governance, security, metadata, and observability.
• Drive CI/CD, DevOps, and IaC practices; mentor engineers and architects.
• Collaborate with business, analytics, and data science teams on the target state.

Skills

Databricks Lakehouse

Delta Lake

DLT

Unity Catalog

Databricks Workflows

Auto Loader

PySpark

Spark SQL

Python

AWS

CDC & Data Migration

Data Governance

CI/CD

Lakehouse Architecture

Streaming

Tools

Airflow

GitHub

Jenkins

Job description

We are seeking a Databricks Lakehouse Architect with strong, hands‑on data engineering experience designing and implementing production‑grade Databricks solutions on AWS.

Key Responsibilities

• Lead enterprise data platform modernization and migration from AWS EMR, NiFi, Redshift, and legacy ETL to Databricks.
• Architect scalable Lakehouse / Medallion (Bronze, Silver, Gold) solutions using Databricks, Delta Lake, Unity Catalog, and Workflows.
• Design batch, streaming, CDC, and real‑time data pipelines using PySpark, Spark SQL, Auto Loader, DLT, and Structured Streaming.
• Define enterprise standards for data architecture, governance, security, metadata, lineage, data quality, and observability.
• Lead AWS data architecture leveraging S3, Glue, IAM, Redshift, EMR, Lambda, and Airflow.
• Drive CI/CD, DevOps, Infrastructure as Code, performance tuning, scalability, and cost optimization.
• Provide architecture leadership, conduct design reviews/workshops, define migration roadmaps, and mentor senior engineers/architects.
• Partner with business, engineering, analytics, and data science teams on target‑state architecture and AI/ML use cases.

Mandatory Skills

• Databricks Lakehouse, Delta Lake, DLT, Unity Catalog, Databricks Workflows
• Auto Loader, PySpark, Spark SQL, Python
• AWS: S3, IAM, Glue, Redshift, EMR, Lambda
• CDC & Data Migration
• Data Governance, Security, Quality & Lineage
• CI/CD: GitHub, Jenkins
• Strong experience with Lakehouse architecture, streaming, metadata‑driven frameworks, and enterprise data platforms