Pyspark Data Engineer

Tata Consultancy Services; Hyderabad, Bengaluru; On-site; INR 1,200,000 - 2,400,000; Full time

Job summary

Tata Consultancy Services in Bangalore is seeking a PySpark Data Engineer to join our data engineering team for cutting-edge cloud and big data solutions. The role focuses on building scalable data pipelines and optimizing Spark jobs.

You will work with Python, PySpark, SQL, and AWS services such as EMR, S3, IAM, Lambda, SNS, SQS, and Redshift to deliver reliable data platforms, support data analytics, and ensure performance and governance in enterprise environments.

Qualifications

• 5–10 years of data engineering experience.
• Strong PySpark, Python, and SQL skills.
• Hadoop ecosystem experience with Hive/Impala.
• Experience with AWS cloud services and Redshift.
• Hands-on in scalable ETL development.

Responsibilities

• Design, develop, and optimize scalable data pipelines using PySpark.
• Work on enterprise-level Data Engineering and Analytics projects.
• Develop and maintain ETL processes for large-scale datasets.
• Analyze and troubleshoot Spark jobs and optimize performance.
• Write efficient SQL queries with complex transformations.
• Collaborate with cross-functional teams to deliver data solutions.
• Work with AWS services including EMR, S3, Lambda, SNS, SQS, IAM, and Redshift.
• Implement best practices for data quality and governance.

Skills

Python

PySpark

SQL

Hadoop

Spark

Hive

Impala

Unix/Linux

HDFS Commands

Big Data

Tools

AWS EMR

AWS S3

AWS IAM

AWS Lambda

AWS SNS

AWS SQS

Redshift

Spark UI

Job description

Walk-in Drive | PySpark Data Engineer | Bangalore | 8th August

Are you passionate about Big Data, PySpark, and AWS technologies?
Join us for an exciting Walk-in Drive and be a part of a dynamic Data Engineering team working on cutting-edge cloud and big data solutions.

Position: PySpark Data Engineer

Location: Bangalore
Walk-in Date: 8th August
Experience Required: 5 - 10 Years

Required Technical Skills

• Python Programming
• PySpark
• Big Data Technologies
• Hadoop Ecosystem
• Hive, Impala
• SQL
• AWS Services:
• EMR
• S3
• IAM
• Lambda
• SNS
• SQS
• Redshift

• Unix/Linux
• HDFS Commands

Key Responsibilities

• Design, develop, and optimize scalable data pipelines using PySpark and Big Data technologies.
• Work on enterprise-level Data Engineering and Data Analytics projects.
• Develop and maintain ETL processes for large-scale datasets.
• Analyze and troubleshoot Spark jobs using Spark UI and performance tuning techniques.
• Write efficient SQL queries involving joins, subqueries, CTEs, and complex data transformations.
• Collaborate with cross-functional teams to understand business requirements and deliver data solutions.
• Work with AWS cloud services including EMR, S3, Lambda, SNS, SQS, IAM, and Redshift.
• Implement best practices for data quality, governance, and performance optimization.

Must-Have Skills

Strong experience in:

• Hadoop
• Spark / PySpark
• Python
• Hive
• Impala
• SQLData Engineering Projects
• Unix/Linux Environment
• HDFS Commands

Good-to-Have Skills

Experience with:

• Spark UI Monitoring and Optimization
• Performance Tuning & Debugging
• Python Scripting
• Advanced SQL Concepts
• Joins
• Subqueries
• CTEs
• Database Technologies
• AWS Cloud Services (EMR, S3, IAM, Lambda, SNS, SQS, Redshift)