GCP Databricks and Data Engineer

Tata Consultancy Services; Hyderabad; On-site; INR 1,200,000 - 2,200,000; Full time

Job summary

Dun & Bradstreet Corp. is seeking a GCP Databricks Data Engineer in Hyderabad to design and optimize scalable data solutions. You will work with Databricks, Spark, PySpark, and BigQuery to build data pipelines and maintain data lakes and warehouse assets.

Ideal candidates will have strong GCP experience, SQL and Python skills, and a track record of delivering enterprise-scale data modernization initiatives in a collaborative, Agile enviornment.

Qualifications

• Experience designing scalable data pipelines on GCP.
• Strong knowledge of data modeling and SQL.
• Experience with data governance and security practices.

Responsibilities

• Design, develop, and maintain scalable batch and real-time data pipelines using Databricks and PySpark.
• Build and optimize ETL/ELT workflows for structured and semi-structured datasets.
• Develop data ingestion frameworks into GCP.
• Create and maintain data lakes, data warehouses, and curated datasets in BigQuery.
• Implement data transformation, cleansing, validation, and enrichment processes.
• Optimize Spark jobs, partitioning strategies, and query performance.
• Collaborate with data architects, analysts, and business stakeholders to deliver scalable solutions.
• Monitor pipeline performance, troubleshoot failures, and ensure data quality.
• Implement CI/CD and automation for data engineering workflows.
• Ensure adherence to governance, security, and compliance standards.

Skills

GCP

Databricks

Apache Spark

PySpark

BigQuery

Cloud Storage

Dataproc

Dataflow

SQL & Data Modeling

Python

Airflow/Cloud Composer

Git & CI/CD

Agile

Tools

Airflow (Cloud Composer)

Git

CI/CD pipelines

Job description

Client: Dun & Bradstreet Corp
Experience: 6-8 Years
Location: Hyderabad /

We are seeking a highly skilled GCP Databricks Data Engineer to design, develop, and optimize scalable data solutions on Google Cloud Platform. The ideal candidate will have strong expertise in Databricks, Apache Spark, PySpark, BigQuery, and cloud-native data engineering practices to support enterprise-scale analytics and data modernization initiatives.

Key Responsibilities

• Design, develop, and maintain scalable batch and real-time data pipelines using Databricks and PySpark.
• Build and optimize ETL/ELT workflows for structured and semi-structured datasets.
• Develop data ingestion frameworks from multiple source systems into GCP.
• Create and maintain data lakes, data warehouses, and curated datasets in BigQuery.
• Implement data transformation, cleansing, validation, and enrichment processes.
• Optimize Spark jobs, partitioning strategies, and query performance.
• Collaborate with data architects, analysts, and business stakeholders to deliver scalable solutions.
• Monitor pipeline performance, troubleshoot failures, and ensure data quality.
• Implement CI/CD and automation for data engineering workflows.
• Ensure adherence to governance, security, and compliance standards.

Required Skills

• Strong experience with Google Cloud Platform (GCP).
• Hands-on expertise in Databricks, Apache Spark, and PySpark.
• Strong knowledge of BigQuery, Cloud Storage (GCS), Dataproc, Dataflow, Pub/Sub.
• Advanced SQL and Data Modeling skills.
• Experience building large-scale ETL/ELT pipelines.
• Strong Python programming skills.
• Experience with workflow orchestration tools such as Airflow/Cloud Composer.
• Understanding of Data Warehousing concepts, Star Schema, Snowflake Schema.
• Experience with Git, CI/CD, and Agile methodologies.

Good to Have

• Delta Lake architecture and optimization.
• Streaming data processing.
• Experience with Looker/Tableau/Power BI.
• Exposure to ML/AI data pipelines.
• Knowledge of Terraform or Infrastructure as Code.