Databricks data engineer - GrowthArc

OpenTalent; Vizianagaram; Hybrid; INR 2,800,000 - 4,000,000; Full time

Job summary

OpenTalent is seeking an experienced Data Engineer to build an AI-driven data platform from scratch in a hybrid setup (Bangalore) with remote options across India.

You will design reusable data engineering accelerators on Databricks, build end-to-end pipelines from enterprise systems, and drive Unity Catalog governance and data discovery for broad user access.

The role demands 6+ years of experience with Databricks, Python/SQL, and modern data architectures, with cloud exposure and dbt expertise.

Qualifications

• Databricks non-negotiable.
• Production experience with Spark, Delta Lake, Workflows, DLT and Repos.
• Unity Catalog implementation and governance.
• Performance tuning: partitioning, Z-ordering, caching, cluster configuration, AQE.
• Python and SQL mandatory.
• Strong Python: OOP, pytest, modular design, packaging.
• Advanced SQL: window functions, CTEs, query optimisation at scale.
• PySpark: DataFrame API, UDFs, Structured Streaming.
• Data Modelling & dbt.
• dbt Core or Cloud: models, macros, testing, documentation, semantic layer.
• Medallion architecture and lake house design.
• Cloud & Infrastructure with AWS/Azure/GCP IAM, storage, secrets management, networking.
• CI/CD pipelines with GitHub Actions, Azure DevOps, or equivalent.
• Data security and row/column-level security.

Responsibilities

• Design and build reusable data engineering accelerators and self-service frameworks on Databricks.
• Build end-to-end pipelines ingesting from enterprise systems including Bullhorn, Beeline, and SAP Fieldglass.
• Lead Unity Catalog adoption data discovery, lineage, access control, and sharing standards.
• Build API/MCP integrations enabling programmatic access to the data platform.
• Develop dbt models and semantic layer components for reporting.
• Embed AI tooling into development workflows and data pipeline design.

Skills

Databricks

Spark

Delta Lake

Workflows

DLT

Repos

Unity Catalog

Performance tuning

Python

SQL

PySpark

dbt

Medallion arch.

Data modelling

Cloud

IAM

CI/CD

Security

Row-level security

Tools

AWS

Azure

GCP

Job description

Total Experience: 6+yrs (min 3+yrs in Databricks)

Notice: Immediate only

Location: Bangalore (Hybrid)/Rest of India- Remote)

ABOUT THE ROLE

We are looking for experienced Data Engineers to join a high-impact engagement building an AI-driven data platform from scratch. You will design accelerators, define self-service frameworks for a growing user base, and own end-to-end pipeline delivery from enterprise source systems.

WHAT YOU WILL DO

• Design and build reusable data engineering accelerators and self-service frameworks on Databricks
• Build end-to-end pipelines ingesting from enterprise systems including Bullhorn, Beeline, and SAP Fieldglass
• Lead Unity Catalog adoption data discovery, lineage, access control, and sharing standards
• Build API/MCP integrations enabling programmatic access to the data platform
• Develop dbt models and semantic layer components for commercial and operational reporting
• Embed AI tooling into development workflows and data pipeline design

TECHNICAL

REQUIREMENTS

• Databricks Non-Negotiable
• Production experience with Spark, Delta Lake, Workflows, DLT, and Repos
• Unity Catalog implementation and governance
• Performance tuning: partitioning, Z-ordering, caching, cluster configuration, AQE
• Python & SQL Mandatory
• Strong Python: OOP, testing (pytest), modular design, packaging
• Advanced SQL: window functions, CTEs, query optimisation at scale
• Py Spark: Data Frame API, UDFs, Structured Streaming
• Data Modelling & dbt
• dbt Core or Cloud: models, macros, testing, documentation, semantic layer
• Medallion architecture (bronze/silver/gold) and lake house design principles
• Cloud & Infrastructure
• Hands-on with AWS, Azure, or GCP IAM, storage, secrets management, networking
• CI/CD pipelines with Git-based workflows (Git Hub Actions, Azure Dev Ops, or equivalent)
• Data Security
• Row/column-level security