Databricks Data Engineer

Emergys; Pune District; Presencial; INR 1.400.000 - 2.100.000; Jornada completa

Descripción de la vacante

Emergys is seeking a Databricks Data Engineer to design, build, and optimize scalable data pipelines on the Lakehouse platform. You will implement end-to-end ETL/ELT across bronze, silver, and gold layers, tune Spark jobs, and integrate with cloud storage.

You will also enforce data governance, security policies, and auditing. Applicants should have 3+ years in Databricks data engineering, strong Python/SQL skills, and hands-on experience with Airflow, Unity Catalog, and CI/CD pipelines.

Formación

• Bachelors or Masters in CS/CE or related field.
• 3+ years in data engineering, Databricks focus.
• Strong SQL, Python, Spark/PySpark experience.

Responsabilidades

• Design, build, and maintain scalable data pipelines.
• Implement ETL/ELT across bronze, silver, gold layers.
• Develop notebooks, workflows, and job clusters.
• Ensure data quality, governance, and security.
• Collaborate with architects and analysts to refine requirements.
• Mentor junior engineers and perform code reviews.

Conocimientos

Databricks

Spark

PySpark

Python

SQL

Cloud platforms

Git

CI/CD

Terraform

Airflow

Unity Catalog

Delta Lake

Delta Live Tables

Databricks Workflows

Azure

AWS

Educación

Bachelor’s degree in computing

Master’s degree in computing

Herramientas

Databricks Lakehouse Platform

Apache Airflow

Terraform

CloudFormation

Databricks Asset Bundles

Descripción del empleo

• Design, build, and maintain scalable data pipelines and architectures using Databricks Lakehouse Platform (Delta Lake, Spark, Spark SQL, Auto Loader, and MLflow)
• Implement complex ETL/ELT solutions across bronze, silver, and gold layers.
• Build reusable frameworks for ingestion, transformation, and monitoring.
• Develop and optimize Databricks notebooks, workflows, and job clusters.
• Troubleshoot performance issues and fine-tune Spark jobs for efficiency.
• Integrate Databricks with cloud storage (ADLS/S3).
• Implement security, secrets, and networking configurations.
• Data Quality, Observability & Governance:
• Build data validation frameworks and error-handling mechanisms.
• Work with Unity Catalog for cataloging, governance, and access control.
• Ensure data compliance, security standards, and audit processes.
• DevOps & CI/CD:
• Maintain Git-based development workflows.
• Develop CI/CD pipelines for Databricks workflows using Git, and orchestration tools like Apache Airflow or Databricks Workflows.
• Collaborate with architects, analysts, and business stakeholders to refine requirements.
• Mentor junior engineers and conduct code reviews.
• Participate in solution design and architecture discussions.

Skills and Qualifications:

· Bachelor’s / Master’s degree in computer science, computer engineering, or relevant field.

· Overall 3+ years of experience, at least 2 years of experience in Databricks Data Engineering.

· Strong, hands-on experience with the Databricks workspace, Delta Lake, Delta Live Tables (DLT), and Unity Catalog.

· Proficiency in Apache Spark, PySpark, and distributed computing principles.

· Advanced proficiency in Python and SQL.

· Hands-on experience with cloud environments such as AWS, Azure.

· Experience with Git, CI/CD pipelines, and infrastructure-as-code tools (Terraform, CloudFormation, or Databricks Asset Bundles).

· Experience with ETL processes: Extracting, transforming, and loading data.

· Strong analytical and problem-solving skills.

· Good communication and collaboration skills.

· Experience with data security and governance principles.

· Understanding of data visualization techniques.