Databricks Architect - Tarrytown, NY [Hybrid]

OMG Technology; Village of Tarrytown (NY); Hybrid; USD 160,000 - 210,000; Full time

Job summary

OMG Technology in New York is seeking a Databricks Architect to lead enterprise-scale Lakehouse architecture and implementation on AWS. The role focuses on designing scalable data solutions, modernizing existing platforms, establishing governance standards, and guiding customer discussions.

The candidate will drive migration of legacy ETL workloads to Databricks, define target-state architectures, and develop reusable patterns.

Qualifications

• 15+ years of data & analytics experience.
• 4+ years hands-on Databricks experience.
• Strong experience with Databricks, Delta Lake, Unity Catalog, DLT, and Auto Loader.
• Proficiency in PySpark, Spark SQL, Python, and Structured Streaming.
• AWS experience with S3, IAM, Glue, Redshift, EMR, and Lambda.
• Experience with Airflow, CDC, and enterprise data migration.
• Knowledge of data governance, security, and observability.
• Strong customer-facing, solution architecture, and technical leadership skills.

Responsibilities

• Lead enterprise-scale Databricks Lakehouse architecture and implementation on AWS.
• Drive migration of EMR, NiFi, Redshift, and legacy ETL workloads to Databricks.
• Evaluate newer Databricks capabilities, including Serverless, Lakehouse Federation, Genie, Databricks Apps, Lakebase, and AI/ML.
• Architect scalable batch, streaming, CDC, and real-time data solutions.
• Design reusable frameworks using Delta Lake, PySpark, Auto Loader, DLT, and Databricks Workflows.
• Define governance, security, lineage, and data-quality standards using Unity Catalog.
• Establish best practices for performance, scalability, and cost optimization.
• Lead customer workshops, architecture reviews, and solution assessments.
• Define target-state architecture, migration roadmaps, and reusable patterns.

Skills

Databricks

PySpark

Spark SQL

Python

Structured Streaming

AWS

Data governance

Leadership

Solution architecture

Customer-facing

Tools

Delta Lake

Unity Catalog

DLT

Auto Loader

Airflow

Databricks

AWS S3

Redshift

EMR

Glue

Job description

Job Details:

Position: Databricks Architect

Location: Tarrytown, NY

Work Arrangement: Hybrid — 3 days onsite per week

Contract Duration: 1 year

Role Summary

We are looking for a strong Databricks Architect to lead enterprise-scale Lakehouse architecture and implementation on AWS. The role involves designing scalable data solutions, modernizing existing data platforms, establishing governance standards, and leading architecture discussions with customers.

Key Responsibilities

• Lead enterprise-scale Databricks Lakehouse architecture and implementation on AWS.
• Drive migration of EMR, NiFi, Redshift, and legacy ETL workloads to Databricks.
• Evaluate newer Databricks capabilities, including Serverless, Lakehouse Federation, Genie, Databricks Apps, Lakebase, and AI/ML.
• Architect scalable batch, streaming, CDC, and real-time data solutions.
• Design reusable frameworks using Delta Lake, PySpark, Auto Loader, DLT, and Databricks Workflows.
• Define governance, security, lineage, and data-quality standards using Unity Catalog.
• Establish best practices for performance, scalability, and cost optimization.
• Lead customer workshops, architecture reviews, and solution assessments.
• Define target-state architecture, migration roadmaps, and reusable patterns.

Required Skills & Qualifications

• 15+ years of overall Data & Analytics experience.
• 4+ years of hands-on Databricks experience as specified in the JD.
• Strong experience with Databricks, Delta Lake, Unity Catalog, DLT, and Auto Loader.
• Proficiency in PySpark, Spark SQL, Python, and Structured Streaming.
• AWS experience with S3, IAM, Glue, Redshift, EMR, and Lambda.
• Experience with Airflow, CDC, and enterprise data migration.
• Knowledge of data governance, security, and observability.
• Strong customer-facing, solution architecture, and technical leadership skills.

Preferred Qualifications

• Healthcare or pharmaceutical domain experience is preferred but not mandatory.
• Experience with enterprise-scale data platform modernization and migration is beneficial.