Gcp Data Engineer (Lead)

Tata Consultancy Services; Chennai District, Bengaluru, Hyderabad; On-site; INR 900,000 - 1,400,000; Full time

Job summary

Tata Consultancy Services in Chennai, India seeks a data/platform engineer to implement data quality checks, schema validation, and lineage capture across data pipelines.

The role involves optimizing BigQuery workloads, supporting migrations and historical loads, and leading CI/CD practices with modern tooling. Experience with CDC tools and AI-assisted patterns is a plus.

Qualifications

• Proven experience with data quality, reconciliation, observability, and testing frameworks.
• Hands-on with BigQuery workloads, SQL/Python for data pipelines.
• Familiarity with CI/CD, Git, Terraform, and cloud-native data services.

Responsibilities

• Implement data quality checks, schema validation, and lineage capture.
• Optimize BigQuery workloads including partitioning, clustering, and views.
• Support migration, historical and incremental loads with validation.
• Lead code reviews, CI/CD integration, and promote engineering standards.
• Knowledge of CDC and replication tools (Datastream, Debezium, etc.).
• Use testing/observability tools like Great Expectations, Soda, dbt tests, etc.
• Promote DevOps practices (Git, Terraform, Jenkins/GitHub Actions).
• Explore AI-assisted engineering patterns using Vertex AI and GenAI when appropriate.

Skills

Data quality

Observability

CI/CD

Mentoring

Tools

BigQuery

Datastream

Debezium

Qlik Replicate

Informatica

Talend

Great Expectations

dbt

Terraform

Jenkins

GitHub Actions

Vertex AI

GenAI

Job description

Role & responsibilities

• Implement data quality checks, schema validation, reconciliation logic, anomaly detection, lineage hooks, metadata capture, and operational monitoring.
• Partner with platform architects to optimize BigQuery workloads, partitioning, clustering, materialized views, storage patterns, and query performance.
• Support migration engineering, historical loads, incremental cutover, parallel run validation, defect remediation, and production support transition.
• Lead code reviews, engineering standards, development accelerators, reusable templates, CI/CD integration, and team mentoring.
• Knowledge of CDC and replication tools such as Datastream, Database Migration Service, Striim, Qlik Replicate, Debezium, Informatica, Talend, or equivalent tools.
• Experience with data quality, reconciliation, observability, and testing tools such as Great Expectations, Soda, dbt tests, Deequ, Monte Carlo, Datafold, or custom SQL/Python frameworks.
• Understanding of CI/CD, Git, Terraform, Cloud Build, Jenkins/GitHub Actions, release management, and environment promotion practices.
• Awareness of AI-assisted engineering patterns using Vertex AI, GenAI coding assistants, metadata generation, test generation, anomaly detection, and documentation acceleration under human review.