Reflections Info Systems - Senior Data Engineer - Google Cloud Platform

Vista Info Systems; Ernakulam; On-site; INR 2,600,000 - 4,200,000; Full time

Job summary

Vista Info Systems seeks a Senior Data Engineer to design and implement scalable data pipelines on Google Cloud Platform. You will build batch and real-time ETL/ELT pipelines using Dataflow, BigQuery, and Cloud Composer, ingesting data from diverse sources, and optimizing datasets.

Strong SQL/Python skills and 6+ years in data engineering are required. You will collaborate with data architects, analysts, and engineers to ensure reliable, secure, and cost-efficient data platforms, with CI/CD and

Qualifications

• 6+ years of experience in Data Engineering.
• Hands-on experience with Google Cloud Platform (GCP).
• Expertise in BigQuery, Dataflow, and Cloud Composer/Airflow.
• Strong SQL and Python programming skills.
• Experience with Apache Beam / Spark.
• Experience with ETL/ELT pipeline development and data integration.
• Knowledge of dbt / Dataform.
• Strong understanding of data warehousing, data modelling, partitioning, and clustering.
• Experience with data quality, reconciliation, lineage, monitoring, and troubleshooting.
• Hands-on experience with GCP IAM and service accounts.
• Experience with Terraform and Infrastructure as Code (IaC).
• Understanding of CI/CD and DevOps practices.

Responsibilities

• Design and develop batch and real-time data pipelines on GCP.
• Build scalable ETL/ELT pipelines using Dataflow, BigQuery, and Cloud Composer.
• Ingest data from databases, APIs, files, SaaS applications, and streaming sources.
• Design and optimize BigQuery datasets, tables, partitioning, and clustering.
• Develop data transformations using SQL, Python, Apache Beam, Spark, dbt, or Dataform.
• Implement workflow orchestration and dependency management using Apache Airflow / Cloud Composer.
• Implement data quality checks, reconciliation, lineage, monitoring, and validation processes.
• Configure and manage GCP IAM, service accounts, and least-privilege access.
• Optimize BigQuery and Dataflow workloads for performance, scalability, and cloud cost efficiency.
• Build and manage infrastructure using Terraform.
• Participate in CI/CD implementation and deployment processes.
• Troubleshoot and resolve production data-pipeline failures and performance issues.
• Collaborate with Data Architects, Data Analysts, Data Scientists, and application engineering teams.
• Ensure data pipelines are reliable, secure, scalable, and maintainable.

Skills

Data Engineering

GCP Experience

BigQuery

Dataflow

Cloud Composer

SQL

Python

Apache Beam

Spark

ETL Pipelines

dbt / Dataform

Data Warehousing

Data Modeling

Data Quality

IAM & Service Accounts

Terraform

CI/CD

DevOps

Tools

Airflow

dbt

Dataform

Terraform

Job description

Key Responsibilities

• Design and develop batch and real-time data pipelines on GCP.
• Build scalable ETL/ELT pipelines using Dataflow, BigQuery, and Cloud Composer.
• Ingest data from databases, APIs, files, SaaS applications, and streaming sources.
• Design and optimize BigQuery datasets, tables, partitioning, and clustering.
• Develop data transformations using SQL, Python, Apache Beam, Spark, dbt, or Dataform.
• Implement workflow orchestration and dependency management using Apache Airflow / Cloud Composer.
• Implement data quality checks, reconciliation, lineage, monitoring, and validation processes.
• Configure and manage GCP IAM, service accounts, and least-privilege access.
• Optimize BigQuery and Dataflow workloads for performance, scalability, and cloud cost efficiency.
• Build and manage infrastructure using Terraform.
• Participate in CI/CD implementation and deployment processes.
• Troubleshoot and resolve production data-pipeline failures and performance issues.
• Collaborate with Data Architects, Data Analysts, Data Scientists, and application engineering teams.
• Ensure data pipelines are reliable, secure, scalable, and maintainable.

Required Skills

• 6+ years of experience in Data Engineering.
• Strong hands-on experience with Google Cloud Platform (GCP).
• Expertise in BigQuery, Dataflow, and Cloud Composer/Airflow.
• Strong SQL and Python programming skills.
• Experience with Apache Beam / Spark.
• Experience with ETL/ELT pipeline development and data integration.
• Knowledge of dbt / Dataform.
• Strong understanding of data warehousing, data modelling, partitioning, and clustering.
• Experience with data quality, reconciliation, lineage, monitoring, and troubleshooting.
• Hands-on experience with GCP IAM and service accounts.
• Experience with Terraform and Infrastructure as Code (IaC).
• Understanding of CI/CD and DevOps practices.

Preferred Candidate Profile

• Strong experience designing production-grade, scalable data platforms on GCP.
• Experience handling both batch and streaming data workloads.
• Strong understanding of cloud cost optimization and performance tuning.
• Excellent analytical and problem-solving skills.
• Ability to work effectively with technical and business stakeholders.
• Strong communication and collaboration skills.

(ref:hirist.tech)