Databricks Data Engineer Aurangabad - GrowthArc

OpenTalent; Mumbai City; Hybrid; INR 1,800,000 - 2,800,000; Full time

Job summary

OpenTalent in Bangalore is seeking an experienced Data Engineer to join an AI-driven data platform built from scratch. You will design accelerators, own end-to-end pipeline delivery from enterprise sources, and shape governance and semantic layers.

The role emphasizes Databricks, Spark, dbt modelling and cloud practices with hybrid work options across India. 6+ years of experience and immediate availability preferred.

Qualifications

• Databricks non-negotiable requirement.
• Production-grade Spark, Delta Lake, Workflows, DLT, and Repos experience.
• Unity Catalog implementation and governance experience.
• Performance tuning including partitioning, Z-ordering and caching.
• Strong Python and SQL skills with testing experience.
• Advanced SQL with analytics and optimization at scale.
• Proficient PySpark for DataFrame APIs and streaming.
• DBT modelling and semantic layer knowledge.
• Medallion architecture and lake house design principles.
• Cloud experience with AWS/Azure/GCP IAM, storage, and networking.
• CI/CD using GitHub Actions or Azure DevOps.
• Data security awareness and best practices.

Responsibilities

• Design and build reusable data engineering accelerators and self-service frameworks on Databricks
• Build end-to-end pipelines ingesting from enterprise systems including Bullhorn, Beeline, and SAP Fieldglass
• Lead Unity Catalog adoption data discovery, lineage, access control, and sharing standards
• Build API/MCP integrations enabling programmatic access to the data platform
• Develop dbt models and semantic layer components for commercial and operational reporting
• Embed AI tooling into development workflows and data pipeline design

Skills

Databricks

Spark

Delta Lake

Workflows

DLT

Repos

Unity Catalog

Governance

Partitioning

Z-ordering

Caching

Cluster config

AQE

Python

SQL

Testing

OOP

pytest

Packaging

DBT

dbt Core

Modeling

Medallion arch

Lake House

Cloud

AWS

Azure

GCP

IAM

Secrets

Networking

CI/CD

GitHub Actions

Security

Job description

Total Experience: 6+yrs (min 3+yrs in Databricks)

Notice: Immediate only

Location: Bangalore (Hybrid)/Rest of India- Remote)

ABOUT THE ROLE

We are looking for experienced Data Engineers to join a high-impact engagement building an AI-driven data platform from scratch. You will design accelerators, define self-service frameworks for a growing user base, and own end-to-end pipeline delivery from enterprise source systems.

Requirements

will evolve you are expected to drive clarity, not wait for it.

WHAT YOU WILL DO

• Design and build reusable data engineering accelerators and self-service frameworks on Databricks
• Build end-to-end pipelines ingesting from enterprise systems including Bullhorn, Beeline, and SAP Fieldglass
• Lead Unity Catalog adoption data discovery, lineage, access control, and sharing standards
• Build API/MCP integrations enabling programmatic access to the data platform
• Develop dbt models and semantic layer components for commercial and operational reporting
• Embed AI tooling into development workflows and data pipeline design

TECHNICAL

REQUIREMENTS

• Databricks Non-Negotiable
• Production experience with Spark, Delta Lake, Workflows, DLT, and Repos
• Unity Catalog implementation and governance
• Performance tuning: partitioning, Z-ordering, caching, cluster configuration, AQE
• Python & SQL Mandatory
• Strong Python: OOP, testing (pytest), modular design, packaging
• Advanced SQL: window functions, CTEs, query optimisation at scale
• Py Spark: Data Frame API, UDFs, Structured Streaming
• Data Modelling & dbt
• dbt Core or Cloud: models, macros, testing, documentation, semantic layer
• Medallion architecture (bronze/silver/gold) and lake house design principles
• Cloud & Infrastructure
• Hands-on with AWS, Azure, or GCP IAM, storage, secrets management, networking
• CI/CD pipelines with Git-based workflows (Git Hub Actions, Azure Dev Ops, or equivalent)
• Data Security

R