Azure Databricks Platform Engineer — Scale Data Pipelines

GQR; New York (NY); On-site; USD 120,000 - 150,000; Full time

Job summary

A leading tech firm is seeking a hands-on Azure Databricks Platform Engineer in New York City. The role involves designing and automating Azure Databricks infrastructure, managing clusters and enabling scalable data processing. Ideal candidates will have over 3 years of experience with Azure and Databricks, as well as skills in CI/CD and troubleshooting. The position offers opportunities for collaboration across engineering and data teams, focusing on building a reliable cloud data platform.

Qualifications

• 3+ years of experience in Azure and Databricks.
• Strong understanding of Azure cloud infrastructure.
• Experience with data processing using Apache Spark.

Responsibilities

• Design and automate Azure Databricks infrastructure.
• Manage clusters and workspaces for optimal performance.
• Enable scalable data processing in a collaborative environment.

Skills

Azure + Databricks

PySpark

Spark SQL

Structured Streaming

Infrastructure-as-Code (Bicep/ARM)

CI/CD with Azure DevOps

Troubleshooting across cloud, data, and networking

Job description

A leading tech firm is seeking a hands-on Azure Databricks Platform Engineer in New York City. The role involves designing and automating Azure Databricks infrastructure, managing clusters and enabling scalable data processing. Ideal candidates will have over 3 years of experience with Azure and Databricks, as well as skills in CI/CD and troubleshooting. The position offers opportunities for collaboration across engineering and data teams, focusing on building a reliable cloud data platform.