Senior Data Engineer: PySpark, Airflow & GCP

Tata Consultancy Services; Sunnyvale (CA); On-site; USD 110,000 - 150,000; Full time

Benefits offered by this job

Discretionary Annual Incentive

Comprehensive Medical Coverage

Family Support: Parental Leaves

Aut & Home Insurance

Commuter Benefits & Training Reimburse

Vacation, Sick Leave & Holidays

401K Plan, Bonus, College Fund

Job summary

Tata Consultancy Services in Sunnyvale, CA is seeking a Data Engineer to design, build, and optimize scalable ETL/ELT pipelines using PySpark and GCP. You will automate ingestion from multiple sources, ensure high performance, and manage Airflow workflows for batch and streaming data.

You will work with Hive and Kafka, implement data quality and governance, support incidents, and contribute to continuous improvements of data platforms and analytics readiness for AI/ML and business operations.

Qualifications

• Must have hands-on experience with scalable data pipelines and cloud data platforms.
• Proficiency in PySpark, Python, Airflow, Hive and Kafka is required.
• Experience with GCP data services and large-scale data processing is essential.

Responsibilities

• Design, build, and maintain scalable ETL/ELT pipelines.
• Automate data ingestion from multiple data sources and ensure high performance.
• Develop and optimize PySpark-based data processing frameworks.
• Build and manage Airflow workflows and orchestration pipelines.
• Work with Hive and Kafka-based batch and streaming data solutions.
• Implement data quality, monitoring, governance, and operational controls.
• Support production incidents, performance tuning, and continuous improvements.

Tools

GCP

PySpark

Python

Airflow

Hive

Kafka

Job description

Tata Consultancy Services in Sunnyvale, CA is seeking a Data Engineer to design, build, and optimize scalable ETL/ELT pipelines using PySpark and GCP. You will automate ingestion from multiple sources, ensure high performance, and manage Airflow workflows for batch and streaming data.

You will work with Hive and Kafka, implement data quality and governance, support incidents, and contribute to continuous improvements of data platforms and analytics readiness for AI/ML and business operations.