Azure Databricks Engineer Jersey City, NJ w2

Steneral Consulting; Jersey City (NJ); On-site; USD 130,000 - 190,000; Full time

Job summary

Steneral Consulting seeks a data platform engineer to support the Databricks environment on Microsoft Azure and collaborate with technology and data teams to design scalable data solutions. You will develop and optimize ETL/ELT pipelines with Databricks, Spark, Python, PySpark, SQL, and Java, manage Unity Catalog and Delta Lake, and utilize MLflow for experiments.

Implement DevOps/MLOps practices, including Git, CI/CD, Kubernetes, OpenShift, and Terraform, ensuring security and governance across

Qualifications

• Experience building data platforms on Azure Databricks.
• Strong programming skills in Python/SQL/Java.
• Proficiency with Spark and big data processing.
• Familiarity with Unity Catalog, Delta Lake, MLflow.

Responsibilities

• Support design, implementation, and development of Databricks data platform on Microsoft Azure.
• Collaborate with technology and data teams to build scalable data solutions.
• Develop and optimize data processing and ETL/ELT pipelines using Databricks, Spark, and Azure technologies.
• Ensure security, automation, and governance within the Databricks ecosystem.
• Work hands-on with Databricks Workspace, Unity Catalog, Delta Lake, Spark UI, and MLflow.
• Automate workflows and orchestrate jobs using Airflow; provide production support.

Skills

Python

SQL

Java

PySpark

Tools

Databricks

Azure

Spark

Unity Catalog

Delta Lake

MLflow

Airflow

Git

CI/CD

Kubernetes

OpenShift

Terraform

Job description

• Support design, implementation, and development of Databricks data platform on Microsoft Azure.
• Collaborate with technology and data teams to build and enhance scalable data solutions.
• Develop and optimize data processing and ETL/ELT pipelines using Databricks, Spark, and Azure technologies.
• Ensure security, automation, and adherence to platform best practices within the Databricks ecosystem.
• Work hands-on with Databricks Workspace, Unity Catalog, Delta Lake, Spark UI, and MLflow.
• Program and engineer data solutions using Python, PySpark, SQL, and/or Java.
• Utilize Azure Databricks, Azure Data Lake Storage Gen2, and Azure Data Factory for cloud data solutions.
• Optionally leverage AWS services (S3, IAM) if required.
• Troubleshoot and performance-tune Spark workloads and data processing pipelines.
• Implement DevOps and MLOps practices, including using Git, CI/CD pipelines, Kubernetes, OpenShift, and Terraform (IaC).
• Apply security and governance controls such as access management, secrets handling, and Unity Catalog.
• Automate workflows, orchestrate jobs, monitor systems, and provide production support.
• Collaborate with application development, infrastructure, cloud, security, architecture, and data teams.
• Operate within established enterprise architecture, security, and governance standards.
• Utilize Apache Airflow for workflow management.
• Deliver production-grade, resilient, and governed Databricks solutions suitable for large enterprise environments.