Senior Azure Databricks Engineer, Jersey City, NJ, US

Intellibee; Jersey City (NJ); Hybrid; USD 140,000 - 180,000; Full time

Benefits offered by this job

Long-Term Stability

Comprehensive Health Coverage

Future Planning

GC Assistance

Job summary

IntelliBee is seeking experienced Databricks / Azure consultants to design, implement, and evolve a Databricks data platform. You will collaborate with technology and data teams to build scalable data solutions on Databricks and Azure.

Candidates should be hands-on with Unity Catalog, Delta Lake, Spark, MLflow, Python/PySpark, SQL, and Azure Data Factory; experience with AWS S3/IAM is a plus. Strong DevOps skills and security governance experience are valued.

Qualifications

• Databricks and Azure data platform design and implementation experience.
• Strong development experience with Python/PySpark/SQL and/or Java.
• Azure Databricks, ADLS Gen2, and Data Factory expertise; AWS S3/IAM a plus.
• Spark-based workload development, troubleshooting, and optimization.
• DevOps/MLOps practices: Git, CI/CD, OpenShift, Kubernetes, IaC.

Responsibilities

• Design, implement, and evolve the Databricks data platform.
• Collaborate with technology and data teams across enterprise functions.
• Ensure security, governance, and production reliability of data solutions.
• Develop and optimize data pipelines and workflows.

Skills

Databricks Workspace

Unity Catalog

Delta Lake

Spark

MLflow

Python

PySpark

SQL

Java

Azure Databricks

ADLS Gen2

Data Factory

AWS S3

IAM

Spark UI

OpenShift

Kubernetes

Terraform / IaC

Git

CI/CD

Airflow

Data governance

Security

Tools

Git

CI/CD

OpenShift

Kubernetes

Terraform

Job description

Seeking experienced Databricks / Azure Consultants to support the design, implementation, and ongoing development of the Databricks data platform. The consultants will work with technology and data teams to build and enhance scalable data solutions using Databricks and Microsoft Azure, with a strong focus on data engineering, integration, security, automation, and platform best practices.

Ideal candidates will have strong hands-on experience across the Databricks ecosystem, including Databricks Workspace, Unity Catalog, Delta Lake, Spark, and MLflow, along with experience developing solutions in Microsoft Azure. Candidates should be comfortable working in a large enterprise environment and collaborating across application development, infrastructure, cloud, security, architecture, and data teams.

Key Skills and Experience

• Databricks Ecosystem: Hands-on experience with Databricks Workspace, Unity Catalog, Delta Lake, Spark UI, and MLflow.
• Programming & Data Engineering: Strong development experience with Python, PySpark, SQL, and/or Java, including the development and optimization of data processing and ETL/ELT pipelines.
• Cloud Platforms: Strong Microsoft Azure experience, particularly Azure Databricks, Azure Data Lake Storage Gen2 (ADLS Gen2), and Azure Data Factory. Experience with AWS, including S3 and IAM, is also desirable.
• Apache Spark: Experience developing, troubleshooting, and performance-tuning Spark-based workloads and data processing pipelines.
• DevOps & MLOps: Experience with Git, CI/CD pipelines, OpenShift, Kubernetes, and Terraform / Infrastructure as Code (IaC).
• Security & Governance: Experience implementing or supporting Databricks security, access controls, secrets management, identity integration, data governance, and Unity Catalog.
• Automation & Operations: Experience with Databricks workflows, job orchestration, monitoring, troubleshooting, performance optimization, and production support.
• Enterprise Delivery: Ability to collaborate across technical teams and operate within established enterprise architecture, security, change-management, and governance standards.
• Knowledge of Apache Airflow

Preferred Experience

• Experience designing and delivering production-grade Databricks solutions in a large enterprise environment, particularly where security, data governance

Benefits at IntelliBee

• Long-Term Stability: Join us on a multi-year opportunities with room to grow.
• Comprehensive Health Coverage: Access quality healthcare benefits to keep you and your family well.
• Future Planning: Enroll in our 401(k) program and invest in your financial security.
• GC Assistance: We support immediate Green Card processing, if required.