Cloudera Data Engineer

McLaren Strategic Ventures; Bengaluru; On-site; INR 6,000,000 - 9,000,000; Full time

Job summary

Next Generation of Technology Consulting is seeking a Senior Data Engineer to design, build, and optimize large-scale Databricks pipelines in Bengaluru. You will implement Medallion lakehouse architectures and ensure data quality and governance across multi-cloud environments.

You\'ll work with PySpark, Delta Lake, Unity Catalog, and Spark Structured Streaming to deliver production-grade workflows, CI/CD runbooks, and observability dashboards. Experience with regulated data workloads is valued.

Qualifications

• 8+ years of hands-on data engineering experience.
• 5+ years building enterprise-scale Databricks solutions.
• Experience delivering Medallion/zonal lakehouse architectures in production.
• Experience with high-volume, regulated data workloads.

Responsibilities

• Design, build, and optimize end-to-end ETL/ELT pipelines in Databricks using Delta Lake and PySpark.
• Implement Medallion (zoned) lakehouse architecture with bronze, silver, gold and validation.
• Leverage Unity Catalog for data governance, access control, and lineage.
• Develop incremental processing, CDC, backfill strategies, and partitioning/optimization techniques.
• Build production-grade Databricks Workflows, Jobs, and orchestration for batch and streaming.
• Establish observability with Databricks Lakehouse Monitoring and dashboards for SLA.

Skills

PySpark

Spark SQL

Delta Lake

Delta Live Tables

Unity Catalog

Data governance

SQL

Python

Spark Structured Streaming

Cloud platforms

Tools

Databricks

Delta Lake tooling

Spark tooling

Job description

Next Generation of Technology Consulting

Our approach is built on delivering value by combining our powerful ecosystem of platforms with capital efficient execution.

We bring together deep domain expertise and our strength in technology to help the world’s leading businesses build their digital core, optimize operations, accelerate revenue growth and deliver tangible outcomes at speed and scale.

Job Description

Key Responsibilities

• Design, build, and optimize end-to-end ETL/ELT pipelines inDatabricksusingDelta Lake,Delta Live Tables (DLT),Auto Loader,PySpark, andSpark SQLfor high-volume, multi-format partner ingestion.
• ImplementMedallion (zoned) architecture– Raw (bronze), Standardized (silver) with advanced validation, quarantine/reject logic, schema enforcement, and Curated (gold) consumer-ready datasets optimized for downstream COB/PI analytics.
• LeverageUnity Catalogfor data governance, access control, lineage, and secure multi-tenant data management.
• Develop incremental processing, change data capture (CDC), backfill strategies, late-arriving data handling, and partitioning/optimization techniques (Z-Ordering, Liquid Clustering, Auto-Optimize) to eliminate performance bottlenecks.
• Build robust data quality frameworks usingDelta constraints, expectations, and monitoring to ensure clean, reliable data for downstream consumption.
• Create production-gradeDatabricks Workflows,Jobs, and orchestration for reliable batch and near-real-time processing usingSpark Structured Streaming.
• Perform data profiling, mapping, reconciliation, and performance tuning of large-scale Spark jobs on Databricks clusters.
• Collaborate with Senior Data Architect and Data Modeller to translate target-state lakehouse design into implementable, testable increments.
• Deliver shippable, production-ready increments in Agile sprints within the implementation window, including CI/CD integration, unit/integration testing, and operational runbooks.
• Establish comprehensive observability usingDatabricks Lakehouse Monitoring, SQL Alerts, and dashboards for pipeline health and SLA compliance.

Requirements

Required Qualifications & Experience

• 8+ years of hands-on data engineering experience
• 5+ years building enterprise-scale solutions onDatabricks(Unity Catalog, Delta Lake, Delta Live Tables)
• Proven track record deliveringMedallion/zonal lakehouse architecturesin production
• Strong experience with high-volume, regulated data workloads (claims, financial, or healthcare data highly preferred)

Technical Skills – Databricks Expertise (Core)

• Databricks Platform : Unity Catalog, Delta Lake, Delta Live Tables (DLT), Auto Loader, Workflows, Jobs, Repos, Lakehouse Monitoring
• Core Technologies : PySpark, Spark SQL, Spark Structured Streaming, Delta constraints & expectations
• Optimization & Performance : Liquid Clustering, Z-Ordering, Auto-Optimize, Dynamic Partition Overwrite, Photon engine
• Governance & Quality : Unity Catalog ACLs, data lineage, schema evolution, Great Expectations (or equivalent)
• Languages : Expert Python (PySpark), SQL
• Cloud : AWS/Azure/GCP(Databricks on any cloud)

Preferred Qualifications

• Exposure to partner ingestion patterns, multi-format data (EDI, flat files, APIs), and downstream analytical workloads
• Familiarity with CMS/HIPAA data handling and compliance in Databricks environments