GCP Data EngineerIntermediate

InfoVision Inc.; Bengaluru; On-site; INR 2,000,000 - 4,500,000; Full time

Job summary

InfoVision Inc. is seeking a skilled GCP Data Engineer in Bengaluru to build and maintain scalable cloud-based data pipelines on Google Cloud Platform. The role covers real-time and batch processing using BigQuery, Dataflow, Pub/Sub, Cloud Storage, and Dataproc.

You will work with Python to develop ETL/ELT workflows, write complex SQL, and design lakehouse models with governance and security in mind, while leveraging CI/CD and Terraform for provisioning.

Qualifications

• Hands-on expertise in GCP services including BigQuery, Dataflow, Pub/Sub, Cloud Storage and Dataproc.
• Proficiency in Python for ETL/ELT pipelines and data ingestion/scripts.
• Strong SQL skills for complex queries and data modeling in BigQuery.
• Experience designing data lakehouse architectures, dimensional models, and governance for quality, lineage and security.
• Familiarity with CI/CD pipelines, Git, and infrastructure-as-code tools (Terraform) for GCP provisioning.
• Knowledge of data orchestration platforms (Airflow/Cloud Composer) and observability tools (Cloud Logging/Monitoring).

Responsibilities

• Design, develop, and optimize end-to-end data pipelines on GCP (BigQuery, Dataflow, Pub/Sub) for real-time and batch processing.
• Integrate multi-cloud data sources across GCP and AWS environments to support analytics and ML teams.
• Leverage Claude AI to automate cataloging, generate SQL from natural language, and streamline data operations.
• Develop and maintain Python scripts for ingestion and transformation; ensure modular, production-ready code.
• Write advanced SQL queries/stored procedures in BigQuery to support BI dashboards and reports.
• Collaborate with data architects, analysts, and ML engineers to translate requirements into robust models.
• Monitor data pipelines, troubleshoot failures, optimize performance, and ensure data quality.
• Participate in code reviews, promote best practices, and mentor junior engineers.

Skills

GCP

BigQuery

Dataflow

Pub/Sub

Cloud Storage

Dataproc

Python

SQL

Airflow

Terraform

Git

Cloud Composer

Claude AI

Lakehouse

Governance

Tools

Cloud Composer

Terraform

Git

Claude AI

Job description

• We are seeking a skilled GCP Data Engineer with 3 to 7 years of experience to join our growing data engineering team in Chennai/Hyderabad/Bangalore/Pune, focused on building and maintaining scalable cloud-based data pipelines and infrastructure on Google Cloud Platform.
• The ideal candidate will have hands-on expertise in GCP services such as BigQuery, Dataflow, Pub/Sub, Cloud Storage, and Dataproc
• Proficiency in leveraging Claude AI and other LLM-based tools for intelligent data transformation, automation of data workflows, and natural language query generation is highly desirable.
• Strong programming skills in Python are required for developing ETL/ELT pipelines, data ingestion scripts, and custom data transformation logic across batch and streaming workloads.
• Solid command of SQL is essential for writing complex queries, optimizing data models in BigQuery, and supporting analytics and reporting teams with structured data access layers.
• Candidates must have experience designing and implementing data lakehouse architectures, dimensional data models, and governance frameworks that ensure data quality, lineage, and security compliance.
• Experience working in agile development environments with CI/CD pipelines, version control using Git, and infrastructure-as-code tools such as Terraform for GCP resource provisioning is expected.
• Familiarity with data orchestration platforms such as Apache Airflow or Cloud Composer and monitoring tools like Cloud Logging and Cloud Monitoring will be considered a strong advantage.

Roles & Responsibilities

• Design, develop, and optimize end-to-end data pipelines on GCP using BigQuery, Dataflow, Cloud Composer, and Pub/Sub to support real-time and batch data processing requirements.
• Integrate multi-cloud data sources across GCP and AWS(nice to have) environments, ensuring seamless data movement, transformation, and availability for downstream analytics and machine learning teams.
• Utilize Claude AI and generative AI tools to automate data catalog creation, generate SQL queries from natural language inputs, and enhance the efficiency of data operations workflows.
• Write and maintain high-quality Python scripts for data ingestion, transformation, and pipeline orchestration, ensuring code is modular, well-documented, and production-ready.
• Develop and maintain complex SQL queries and stored procedures in BigQuery to support business intelligence dashboards, ad hoc analysis, and scheduled reporting pipelines.
• Collaborate with data architects, analysts, and ML engineers to translate business requirements into robust data models and ensure data accuracy, freshness, and consistency across the platform.
• Monitor and troubleshoot data pipeline failures, performance bottlenecks, and data quality issues using GCP-native observability tools and proactively implement preventive measures.
• Participate in code reviews, contribute to data engineering best practices, and mentor junior team members to uphold high engineering standards across the data platform.