Databricks Data Engineer (GCP BigQuery To Azure Databricks Migration

Tech Mahindra; Hyderabad, Pune District, Bengaluru; Hybrid; INR 2,600,000 - 4,200,000; Full time

Job summary

Tech Mahindra is seeking a Databricks Data Engineer to lead a GCP BigQuery to Azure Databricks migration across Bangalore/Hyderabad/Pune. The role demands 6-12 years of data engineering experience and strong expertise in Azure Databricks, PySpark, Delta Lake, and lakehouse architectures.

You will design scalable ETL/ELT pipelines, migrate BigQuery datasets to Delta Lake, implement Medallion Architecture, and collaborate with data platform teams to ensure smooth production deployment and

Qualifications

• 6-12 years of Data Engineering experience.
• Minimum 3+ years of hands-on Azure Databricks experience.
• Proven experience in at least one large-scale cloud migration program involving BigQuery.
• Strong understanding of modern cloud-native data architectures and Lakehouse concepts.

Responsibilities

• Design, develop, and optimize scalable ETL/ELT pipelines using Azure Databricks and PySpark.
• Migrate BigQuery datasets, tables, views, and schemas to Azure Databricks Delta Lake.
• Implement Medallion Architecture and Lakehouse best practices.
• Develop robust batch and incremental data processing frameworks.
• Build data validation, reconciliation, and quality assurance frameworks.
• Perform SQL optimization, Spark performance tuning, and workload optimization.
• Support historical data migration, testing, deployment, and production stabilization.
• Collaborate with architects, business stakeholders, and data platform teams during migration.
• Implement CI/CD processes and version control using Git-based repositories.
• Ensure data governance standards through Unity Catalog and enterprise security controls.

Skills

Azure Databricks

PySpark

Spark SQL

Delta Lake

Unity Catalog

Data Lakehouse Architecture

Medallion Architecture

Google BigQuery

GCP to Azure Migration

BigQuery Schema Conversion

Data Validation & Reconciliation

SQL Performance Tuning

Git

CI/CD Pipelines

Azure Data Factory

ADLS Gen2

Data Pipeline Development

ETL/ELT Frameworks

Tools

Airflow/Cloud Composer

Google Dataflow

DBT

Azure Key Vault

Terraform

Job description

Databricks Data Engineer (GCP BigQuery to Azure Databricks Migration

Location: Bangalore / Hyderabad / Pune
Experience: 6-12 Years

We are looking for a highly skilled Databricks Data Engineer to join a large-scale enterprise cloud modernization program focused on migrating data platforms from Google Cloud Platform (BigQuery/GDAP) to Azure Databricks Lakehouse.

The ideal candidate will have hands‑on experience in BigQuery migration projects, modern data engineering practices, Azure Databricks, PySpark, Delta Lake, and large-scale ETL modernization initiatives.

This role requires strong technical expertise in designing scalable data pipelines, optimizing lakehouse architectures, and ensuring successful end‑to‑end migration of enterprise data assets.

Key Responsibilities

• Design, develop, and optimize scalable ETL/ELT pipelines using Azure Databricks and PySpark.


• Migrate BigQuery datasets, tables, views, and schemas to Azure Databricks Delta Lake.


• Implement Medallion Architecture (Bronze, Silver, Gold) and Lakehouse best practices.


• Develop robust batch and incremental data processing frameworks.


• Build data validation, reconciliation, and quality assurance frameworks.


• Perform SQL optimization, Spark performance tuning, and workload optimization.


• Support historical data migration, testing, deployment, and production stabilization.


• Collaborate with architects, business stakeholders, and data platform teams during the migration journey.


• Implement CI/CD processes and version control using Git-based repositories.


• Ensure data governance standards through Unity Catalog and enterprise security controls.

Mandatory Skills

Core Databricks & Spark

• Azure Databricks


• PySpark


• Spark SQL


• Delta Lake


• Unity Catalog


• Data Lakehouse Architecture


• Medallion Architecture

Azure Data Engineering

• Azure Data Lake Storage (ADLS Gen2)


• Azure Data Factory (ADF) (Preferred)


• Data Pipeline Development


• ETL/ELT Frameworks

Migration Expertise (Must Have)

• Google BigQuery


• GCP to Azure Migration


• BigQuery Schema Conversion


• Data Migration Strategy


• Data Validation & Reconciliation Frameworks

Development & DevOps

• SQL Performance Tuning


• Git


• CI/CD Pipelines


• Production Deployment & Support

Preferred Skills

• Airflow / Cloud Composer


• Google Dataflow


• DBT (Data Build Tool)


• Azure Key Vault


• Terraform


• Data Governance & Metadata Management


• Agile/Scrum Delivery Model

Qualifications

• 6-12 years of Data Engineering experience.


• Minimum 3+ years of hands‑on Azure Databricks experience.


• Proven experience in at least one large-scale cloud migration program involving BigQuery.


• Strong understanding of modern cloud-native data architectures and Lakehouse concepts.