Data Engineer - Databricks / Azure Consultant

Yoh, A Day & Zimmermann Company; Jersey City (NJ); Hybrid; USD 96,000 - 117,000; Full time

Benefits offered by this job

Medical Benefits

Dental & Vision Benefits

HSA

Life Insurance

EAP

Job summary

Yoh, A Day & Zimmermann Company is seeking a Databricks and Azure Consultant to design, implement, and support an enterprise-grade Databricks data platform on Microsoft Azure. This hybrid role in Jersey City brings together data engineering with cloud platform, security, automation, and operational best practices.

Responsibilities include designing and supporting the Databricks platform, partnering with technology teams to build scalable Azure-based data solutions, and focusing on data

Qualifications

• Azure, Databricks, and Apache Spark expertise.
• CI/CD experience across cloud data platforms.
• Hands-on with Databricks Workspace, Unity Catalog, and Delta Lake.
• Experience with Spark UI and MLflow.
• Strong development with Python, PySpark, SQL, or Java.
• Experience building and optimizing data processing and ETL/ELT pipelines.
• Azure Data Lake Storage Gen2 and Azure Data Factory familiarity.
• Git and CI/CD pipelines experience.
• OpenShift and Kubernetes experience.
• Terraform and IaC proficiency.
• Databricks security, access controls, secrets, and identity integration.
• Data governance and Unity Catalog experience.
• Databricks workflows, job orchestration, and monitoring.
• Production support and performance tuning.
• Collaboration across enterprise teams and governance standards.
• Apache Airflow knowledge.
• Desirable: AWS S3/IAM experience.

Responsibilities

• Design, implement, and support the Databricks data platform.
• Partner with technology and data teams on scalable Databricks solutions on Azure.
• Focus on data engineering, integration, security, automation, and platform best practices.
• Collaborate across application development, infrastructure, cloud, security, architecture, and data teams.

Skills

Azure

Databricks

Apache Spark

CI/CD

Workspace

Unity Catalog

Delta Lake

Spark UI

MLflow

Python

PySpark

SQL

Java

ETL/ELT

Azure Data Lake

Azure Data Factory

Git

OpenShift

Kubernetes

Terraform

IaC

Security

Data governance

Databricks workflows

Airflow

AWS

Tools

Databricks Workspace

Unity Catalog

Delta Lake

Spark UI

MLflow

Python

PySpark

SQL

Java

OpenShift

Kubernetes

Terraform

Infra as Code

Job description

Yoh, A Day & Zimmermann Company is seeking a Databricks and Azure Consultant to help design, implement, and support an enterprise-grade Databricks data platform on Microsoft Azure. This hybrid role in Jersey City brings together data engineering with cloud platform, security, automation, and operational best practices.

What you will do

• Design, implement, and support the Databricks data platform, including ongoing development


• Partner with technology and data teams to build and improve scalable data solutions using Databricks on Microsoft Azure


• Concentrate on data engineering, integration, security, automation, and platform best practices


• Collaborate across application development, infrastructure, cloud, security, architecture, and data teams

What you bring

• Azure, Databricks, and Apache Spark


• CI/CD experience


• Hands-on experience with Databricks Workspace, Unity Catalog, and Delta Lake


• Hands-on experience with Spark UI and MLflow


• Strong development experience with Python, PySpark, SQL, and/or Java


• Experience developing and optimizing data processing and ETL/ELT pipelines


• Strong Microsoft Azure experience, especially Azure Databricks


• Strong Microsoft Azure experience with Azure Data Lake Storage Gen2 (ADLS Gen2) and Azure Data Factory


• Experience with Git and CI/CD pipelines


• Experience with OpenShift and Kubernetes


• Experience with Terraform and Infrastructure as Code (IaC)


• Experience implementing or supporting Databricks security, access controls, secrets management, and identity integration


• Experience implementing or supporting data governance and Unity Catalog


• Experience with Databricks workflows, job orchestration, and monitoring


• Experience with troubleshooting and performance optimization, including production support


• Ability to collaborate across technical teams and operate within enterprise architecture, security, change-management, and governance standards


• Knowledge of Apache Airflow


• Desirable: experience with AWS (including S3 and IAM)

Technologies involved

• Azure, Azure Databricks, Azure Data Lake Storage Gen2 (ADLS Gen2), Azure Data Factory


• Databricks, Databricks Workspace, Unity Catalog, Delta Lake, Databricks workflows


• Apache Spark, Spark UI, Python, PySpark, SQL, Java


• MLflow, ETL/ELT


• CI/CD


• Git


• OpenShift, Kubernetes


• Terraform, Infrastructure as Code (IaC)


• Databricks security, Databricks access controls


• Apache Airflow


• AWS, S3, IAM

Location and schedule

• Jersey City, NJ (hybrid)


• Ideally 5 days onsite, though 3-4 days onsite may be acceptable and will be discussed during the interview


• Candidates locals to NJ/NY area


• USC & GC ONLY

Compensation

• Estimated rate range: $70.00 to $85.00 per hour (USD)


• Pay ranges shown are estimations; actual pay depends on experience, technical expertise, and other qualifications in the job description

Benefits

• Medical, Prescription, Dental & Vision Benefits (for employees working 20+ hours per week)


• Health Savings Account (HSA) (for employees working 20+ hours per week)


• Life & Disability Insurance (for employees working 20+ hours per week)


• MetLife Voluntary Benefits


• Employee Assistance Program (EAP)


• 401K Retirement Savings Plan


• Direct Deposit & weekly epayroll


• Referral Bonus Programs


• Certification and training opportunities

Preferred experience

• Experience designing and delivering production-grade Databricks solutions in a large enterprise environment, especially where security, data governance, resiliency, automation, and operational support are key requirements

Benefit eligibility is subject to applicable laws and client requirements.