Databricks Architect - Tarrytown, NY [Hybrid]

OMG Technology; New York (NY); Hybrid; USD 180,000 - 240,000; Full time

Job summary

OMG Technology is seeking a Databricks Architect to lead enterprise-scale Lakehouse architecture and AWS-based implementation. You will drive migration of traditional workloads to Databricks, design scalable data solutions, and establish governance standards across the data platform.

The role requires deep expertise in Databricks, Delta Lake, Unity Catalog, and PySpark, with strong leadership in customer engagements and architecture reviews. Hybrid work arrangement offered in the NY area.

Qualifications

• 15+ years of overall Data & Analytics experience.
• 4+ years of hands-on Databricks experience as specified in the JD.
• Proficiency in PySpark, Spark SQL, Python, and Structured Streaming.
• AWS experience with S3, IAM, Glue, Redshift, EMR, and Lambda.
• Experience with Airflow, CDC, and enterprise data migration.
• Knowledge of data governance, security, and observability.
• Strong customer-facing, solution architecture, and technical leadership skills.

Responsibilities

• Lead enterprise-scale Databricks Lakehouse architecture and implementation on AWS.
• Drive migration of EMR, NiFi, Redshift, and legacy ETL workloads to Databricks.
• Evaluate newer Databricks capabilities, including Serverless, Lakehouse Federation, Genie, Databricks Apps, Lakebase, and AI/ML.
• Architect scalable batch, streaming, CDC, and real-time data solutions.
• Design reusable frameworks using Delta Lake, PySpark, Auto Loader, DLT, and Databricks Workflows.
• Define governance, security, lineage, and data-quality standards using Unity Catalog.
• Establish best practices for performance, scalability, and cost optimization.
• Lead customer workshops, architecture reviews, and solution assessments.
• Define target-state architecture, migration roadmaps, and reusable patterns.

Skills

Databricks

Delta Lake

Unity Catalog

DLT

Auto Loader

PySpark

Spark SQL

Python

Structured Streaming

AWS

S3

Airflow

Data governance

Security & Observability

Tools

Airflow

Job description

Job Details:

Position: Databricks Architect

Location: Tarrytown, NY

Work Arrangement: Hybrid — 3 days onsite per week

Contract Duration: 1 year

Role Summary

We are looking for a strong Databricks Architect to lead enterprise-scale Lakehouse architecture and implementation on AWS. The role involves designing scalable data solutions, modernizing existing data platforms, establishing governance standards, and leading architecture discussions with customers.

Key Responsibilities

• Lead enterprise-scale Databricks Lakehouse architecture and implementation on AWS.
• Drive migration of EMR, NiFi, Redshift, and legacy ETL workloads to Databricks.
• Evaluate newer Databricks capabilities, including Serverless, Lakehouse Federation, Genie, Databricks Apps, Lakebase, and AI/ML.
• Architect scalable batch, streaming, CDC, and real-time data solutions.
• Design reusable frameworks using Delta Lake, PySpark, Auto Loader, DLT, and Databricks Workflows.
• Define governance, security, lineage, and data-quality standards using Unity Catalog.
• Establish best practices for performance, scalability, and cost optimization.
• Lead customer workshops, architecture reviews, and solution assessments.
• Define target-state architecture, migration roadmaps, and reusable patterns.

Required Skills & Qualifications

• 15+ years of overall Data & Analytics experience.
• 4+ years of hands-on Databricks experience as specified in the JD.
• Strong experience with Databricks, Delta Lake, Unity Catalog, DLT, and Auto Loader.
• Proficiency in PySpark, Spark SQL, Python, and Structured Streaming.
• AWS experience with S3, IAM, Glue, Redshift, EMR, and Lambda.
• Experience with Airflow, CDC, and enterprise data migration.
• Knowledge of data governance, security, and observability.
• Strong customer-facing, solution architecture, and technical leadership skills.

Preferred Qualifications

• Healthcare or pharmaceutical domain experience is preferred but not mandatory.
• Experience with enterprise-scale data platform modernization and migration is beneficial.