Azure Databricks Platform Engineer

Steneral Consulting; Jersey City (NJ); Presencial; USD 130.000 - 190.000; Jornada completa

Descripción de la vacante

Steneral Consulting seeks a data platform engineer to support the Databricks environment on Microsoft Azure and collaborate with technology and data teams to design scalable data solutions. You will develop and optimize ETL/ELT pipelines with Databricks, Spark, Python, PySpark, SQL, and Java, manage Unity Catalog and Delta Lake, and utilize MLflow for experiments.

Implement DevOps/MLOps practices, including Git, CI/CD, Kubernetes, OpenShift, and Terraform, ensuring security and governance across

Formación

• Experience building data platforms on Azure Databricks.
• Strong programming skills in Python/SQL/Java.
• Proficiency with Spark and big data processing.
• Familiarity with Unity Catalog, Delta Lake, MLflow.

Responsabilidades

• Support design, implementation, and development of Databricks data platform on Microsoft Azure.
• Collaborate with technology and data teams to build scalable data solutions.
• Develop and optimize data processing and ETL/ELT pipelines using Databricks, Spark, and Azure technologies.
• Ensure security, automation, and governance within the Databricks ecosystem.
• Work hands-on with Databricks Workspace, Unity Catalog, Delta Lake, Spark UI, and MLflow.
• Automate workflows and orchestrate jobs using Airflow; provide production support.

Conocimientos

Python

SQL

Java

PySpark

Herramientas

Databricks

Azure

Spark

Unity Catalog

Delta Lake

MLflow

Airflow

Git

CI/CD

Kubernetes

OpenShift

Terraform

Descripción del empleo

Steneral Consulting seeks a data platform engineer to support the Databricks environment on Microsoft Azure and collaborate with technology and data teams to design scalable data solutions. You will develop and optimize ETL/ELT pipelines with Databricks, Spark, Python, PySpark, SQL, and Java, manage Unity Catalog and Delta Lake, and utilize MLflow for experiments.

Implement DevOps/MLOps practices, including Git, CI/CD, Kubernetes, OpenShift, and Terraform, ensuring security and governance across