Senior Data Engineer

Tata Consultancy Services; Sunnyvale (CA); On-site; USD 110,000 - 150,000; Full time

Benefits offered by this job

Discretionary Annual Incentive

Comprehensive Medical Coverage

Family Support: Parental Leaves

Aut & Home Insurance

Commuter Benefits & Training Reimburse

Vacation, Sick Leave & Holidays

401K Plan, Bonus, College Fund

Job summary

Tata Consultancy Services in Sunnyvale, CA is seeking a Data Engineer to design, build, and optimize scalable ETL/ELT pipelines using PySpark and GCP. You will automate ingestion from multiple sources, ensure high performance, and manage Airflow workflows for batch and streaming data.

You will work with Hive and Kafka, implement data quality and governance, support incidents, and contribute to continuous improvements of data platforms and analytics readiness for AI/ML and business operations.

Qualifications

• Must have hands-on experience with scalable data pipelines and cloud data platforms.
• Proficiency in PySpark, Python, Airflow, Hive and Kafka is required.
• Experience with GCP data services and large-scale data processing is essential.

Responsibilities

• Design, build, and maintain scalable ETL/ELT pipelines.
• Automate data ingestion from multiple data sources and ensure high performance.
• Develop and optimize PySpark-based data processing frameworks.
• Build and manage Airflow workflows and orchestration pipelines.
• Work with Hive and Kafka-based batch and streaming data solutions.
• Implement data quality, monitoring, governance, and operational controls.
• Support production incidents, performance tuning, and continuous improvements.

Tools

GCP

PySpark

Python

Airflow

Hive

Kafka

Job description

• Design, build, and maintain scalable ETL/ELT pipelines.
• Automate data ingestion from multiple data sources.
• Ensure reliable, high-performance, and scalable data movement.
• Develop and optimize PySpark-based data processing frameworks.
• Build and manage Airflow workflows and orchestration pipelines.
• Work with Hive and Kafka-based batch and streaming data solutions.
• Implement data quality, monitoring, governance, and operational controls.
• Support production incidents, performance tuning, and continuous improvement initiatives
• The Data Engineer will be responsible for designing, building, governing, and optimizing data platforms, pipelines, and integrations to ensure secure, reliable, and scalable data availability for analytics, reporting, AI/ML, and business operations.

Must Have Technical/Functional Skills

• Google Cloud Platform (GCP)
• PySpark
• Python
• Airflow
• Hive
• Kafka

Data Pipeline Development, Enhancement & Maintenance

• Design, build, and maintain scalable ETL/ELT pipelines.
• Automate data ingestion from multiple data sources.
• Ensure reliable, high-performance, and scalable data movement.
• Develop and optimize PySpark-based data processing frameworks.
• Build and manage Airflow workflows and orchestration pipelines.
• Work with Hive and Kafka-based batch and streaming data solutions.
• Implement data quality, monitoring, governance, and operational controls.
• Support production incidents, performance tuning, and continuous improvement initiatives

Roles & Responsibilities

• The Data Engineer will be responsible for designing, building, governing, and optimizing data platforms, pipelines, and integrations to ensure secure, reliable, and scalable data availability for analytics, reporting, AI/ML, and business operations.

Preferred Experience

• Strong Functional Knowledge in Retail (Search & Personalization) Added advantage.
• Strong experience with large-scale GCP Data Engineering solutions.
• Hands-on experience with Airflow DAG development and support
• Experience building and optimizing Spark/PySpark data pipelines.
• Exposure to Hive-based data processing and Kafka-driven ingestion frameworks.
• Experience supporting analytics, personalization, recommendation, or search platforms is highly desirable

TCS Employee Benefits Summary

• Discretionary Annual Incentive.
• Comprehensive Medical Coverage: Medical & Health, Dental & Vision, Disability Planning & Insurance, Pet Insurance Plans.
• Family Support: Maternal & Parental Leaves.
• Insurance Options: Aut& Home Insurance, Identity Theft Protection.
• Convenience & Professional Growth: Commuter Benefits & Certification & Training Reimbursement.
• Time Off: Vacation, Time Off, Sick Leave & Holidays.
• Legal & Financial Assistance: Legal Assistance, 401K Plan, Performance Bonus, College Fund, Student Loan Refinancing.

Salary Range -$110,000-$150,000 a year