Senior Data Engineer - BigQuery, Spark & CI/CD

Tata Consultancy Services Limited

Minnesota City (MN)

On-site

USD 110,000 - 135,000

Full time

4 days ago
Be an early applicant

A complete application in a minute

Tailored resume and cover letter, ready to send.

Get past ATS filters

Benefits offered by this job

Discretionary Annual Incentive
Comprehensive Medical Coverage
Parental Leaves
Commuter Benefits

Job summary

Tata Consultancy Services Limited seeks a Data Engineer to design, build, and operate scalable data pipelines for customer, transaction, and marketing datasets. You will implement data contracts, lineage, and automated quality controls while optimizing throughput and cost in a cloud-first environment.

The role emphasizes identity resolution, consent-aware processing, and audience activation, with strong emphasis on security, monitoring, and production-readiness in an Agile setting.

Qualifications

  • Advanced SQL and Python for ingestion, transformation, validation, reconciliation, and performance optimization.
  • Hands-on GCP experience with BigQuery, Dataflow / Apache Beam, Pub/Sub, Cloud Storage, and Cloud Composer / Airflow.
  • Strong Apache Spark and distributed data-processing experience, including partitioning, skew handling, incremental loads, and tuning.

Responsibilities

  • Design, build, and operate scalable batch and near-real-time pipelines for customer, transaction, product, digital interaction, campaign, and partner data.
  • Develop curated data products supporting identity resolution, audience segmentation, activation, analytics, attribution, and measurement.
  • Partner with architects, product managers, analysts, data scientists, privacy teams, and application engineers to define data contracts, SLAs, and acceptance criteria.
  • Implement controls for schema, uniqueness, referential integrity, completeness, freshness, validity, and source reconciliation.
  • Optimize BigQuery, Spark, and pipeline workloads for throughput, latency, reliability, and cost.
  • Build unit, integration, regression, and production‑validation tests and embed quality gates in CI/CD.
  • Establish monitoring, alerting, dashboards, runbooks, incident triage, and permanent corrective actions.
  • Protect sensitive identifiers using approved access, encryption, tokenization / hashing, retention, auditability, and least‑privilege design.
  • Perform code reviews, maintain technical and lineage documentation, and contribute to reusable engineering standards.
  • Support releases, backfills, migrations, performance tuning, and controlled recovery from pipeline failures.

Skills

SQL
Python

Education

Bachelor's or Master's in CS/DI

Tools

BigQuery
Dataflow/Airflow
Spark

Job description

Tata Consultancy Services Limited seeks a Data Engineer to design, build, and operate scalable data pipelines for customer, transaction, and marketing datasets. You will implement data contracts, lineage, and automated quality controls while optimizing throughput and cost in a cloud-first environment.

The role emphasizes identity resolution, consent-aware processing, and audience activation, with strong emphasis on security, monitoring, and production-readiness in an Agile setting.

Get your free, confidential resume review.

or drag and drop your file here.

Similar jobs

Similar jobs worth comparing

Senior Data Engineer: Build Scalable ETL Pipelines on GCP
Senior Data Engineer: Build Scalable ETL Pipelines on GCP

Tata Consultancy Services Limited • Sunnyvale (CA)

On-site
USD 110,000 - 150,000
Discretionary Annual Incentive
Comprehensive Medical Coverage: Health
Dental & Vision
+6
Senior Data Engineer: PySpark, Airflow & GCP
Senior Data Engineer: PySpark, Airflow & GCP

Tata Consultancy Services Limited • Sunnyvale (CA)

On-site
USD 110,000 - 150,000
Discretionary Annual Incentive
Comprehensive Medical Coverage
Family Support: Parental Leaves
+4
Data Pipeline & Cloud Infrastructure Engineer
Data Pipeline & Cloud Infrastructure Engineer

Tata Consultancy Services Limited • Cleveland (OH)

On-site
USD 100,000 - 120,000
Discretionary Annual Incentive
Comprehensive Medical Coverage: Health
Family Support: Parental Leaves
+4
Data Engineer - GCP, PySpark, Airflow & Kafka
Data Engineer - GCP, PySpark, Airflow & Kafka

Tata Consultancy Services Limited • Sunnyvale (CA)

On-site
USD 100,000 - 120,000
Discretionary Annual Incentive
Comprehensive Medical Coverage
Family Leave
+3
Senior Data Engineer: Cloud, dbt, Airflow, BigQuery
Senior Data Engineer: Cloud, dbt, Airflow, BigQuery

All Generation Tech • United States

Remote
USD 120,000 - 150,000
Annual bonus
Comprehensive allowances
Provident Fund
+3
Senior Data Engineer: BigQuery & AI-Driven Data Solutions
Senior Data Engineer: BigQuery & AI-Driven Data Solutions

Applied Systems • Indiana

Hybrid
USD 130,000 - 190,000
Senior Data Engineer: Databricks Lakehouse Leader
Senior Data Engineer: Databricks Lakehouse Leader

Tata Consultancy Services Limited • Irvine (CA)

On-site
USD 120,000 - 180,000
Senior GCP Data Engineer: BigQuery, Spark & Dataflow
Senior GCP Data Engineer: BigQuery, Spark & Dataflow

Envision Technology Solutions • Minnesota City (MN)

On-site
USD 120,000 - 160,000
Senior Data Engineer: Azure Spark & Real-Time Pipelines
Senior Data Engineer: Azure Spark & Real-Time Pipelines

Tata Consultancy Services Limited • Eden Prairie (MN)

On-site
USD 110,000 - 120,000
Medical Coverage
401K Plan
Paid Time Off
Teradata Data Engineer: SQL, Spark & Data Pipelines
Teradata Data Engineer: SQL, Spark & Data Pipelines

Tata Consultancy Services Limited • Charlotte (NC)

On-site
USD 100,000 - 105,000
Discretionary Annual Incentive
Comprehensive Medical Coverage
401K Plan
+2