GCP Data Engineer - (Databricks)

Parkar Global Technologies; Pune District; Hybrid; INR 2,000,000 - 4,000,000; Full time

Job summary

Parkar Global Technologies is seeking a Senior Data Engineer to design, build, and optimize scalable data pipelines on Google Cloud Platform and Databricks, working with cross-functional teams to deliver robust data systems.

The role emphasizes ETL/ELT development, Spark/PySpark workloads, Delta Lake, and workflow orchestration with Cloud Composer, data quality, governance, and performance tuning in production environments.

Qualifications

• 6-8 years of experience in data engineering.
• Strong hands-on experience with GCP (BigQuery, Dataflow, Pub/Sub, Cloud Storage).
• Hands-on experience with Databricks, including Delta Lake, Databricks Workflows/Jobs, and notebook-based development.
• Experience with Spark /PySpark and SQL optimization.
• Hands-on with Cloud Composer (Airflow).
• Strong understanding of data modeling and pipeline design.
• Experience with batch + streaming data processing.
• Familiarity with CI/CD and Git-based workflows.
• Strong problem-solving and analytical skills.

Responsibilities

• Design, build, and optimize scalable batch and real-time data pipelines on GCP and Databricks.
• Develop and maintain ETL/ELT workflows for diverse data sources.
• Build robust data ingestion frameworks and support data migration initiatives.
• Develop and optimize Spark/PySpark workloads in Databricks, including notebooks, jobs, and Delta Lake tables.
• Manage workflow orchestration, scheduling, and pipeline dependencies (Cloud Composer, Databricks Workflows).
• Ensure efficient and reliable data processing across environments.
• Optimize query performance and data processing jobs, including Databricks cluster and job tuning.
• Implement data validation, quality, and reconciliation frameworks.
• Monitor pipelines to ensure availability, reliability, and performance.
• Collaborate with cross-functional teams to translate business requirements into solutions.
• Contribute to data architecture and design discussions.
• Follow best practices for version control, CI/CD, and deployments.
• Ensure compliance with security, governance, and data standards, including access control and cataloging in Databricks.
• Support production systems through troubleshooting and continuous improvement.

Skills

GCP BigQuery

Databricks

Spark/PySpark

SQL optimization

Airflow / Cloud Composer

Data modeling

Batch + streaming processing

CI/CD & Git workflows

Problem solving & analytics

Tools

Delta Lake

Databricks Workflows/Jobs

Notebook development

Cloud Composer

Job description

Role Overview

We are looking for a Senior Data Engineer with strong hands-on experience in building and optimizing scalable data pipelines on Google Cloud Platform (GCP) and Databricks. This role requires deep technical expertise in data engineering, pipeline development, and cloud-native solutions, with the ability to work closely with cross-functional teams to deliver high-quality data systems.

Key Responsibilities

• Design, build, and optimize scalable batch and real-time data pipelines on GCP and Databricks.
• Develop and maintain ETL/ELT workflows for diverse data sources.
• Build robust data ingestion frameworks and support data migration initiatives.
• Develop and optimize Spark/PySpark workloads in Databricks, including notebooks, jobs, and Delta Lake tables.
• Manage workflow orchestration, scheduling, and pipeline dependencies (Cloud Composer, Databricks Workflows).
• Ensure efficient and reliable data processing across environments.
• Optimize query performance and data processing jobs, including Databricks cluster and job tuning.
• Implement data validation, quality, and reconciliation frameworks.
• Monitor pipelines to ensure availability, reliability, and performance.
• Collaborate with cross-functional teams to translate business requirements into solutions.
• Contribute to data architecture and design discussions.
• Follow best practices for version control, CI/CD, and deployments.
• Ensure compliance with security, governance, and data standards, including access control and cataloging in Databricks.
• Support production systems through troubleshooting and continuous improvement.

Must-Have Skills

• 6-8 years of experience in data engineering.
• Strong hands-on experience with GCP (BigQuery, Dataflow, Pub/Sub, Cloud Storage).
• Hands-on experience with Databricks, including Delta Lake, Databricks Workflows/Jobs, and notebook-based development.
• Experience with Spark /PySpark and SQL optimization.
• Hands-on with Cloud Composer (Airflow).
• Strong understanding of data modeling and pipeline design.
• Experience with batch + streaming data processing.
• Familiarity with CI/CD and Git-based workflows.
• Strong problem-solving and analytical skills.