Pyspark Developer (Open For Pune /Kolkata also)

Tata Consultancy Services; Hyderabad, Chennai District, Bengaluru; On-site; INR 1,200,000 - 1,800,000; Full time

Job summary

Tata Consultancy Services in Hyderabad, India is seeking an experienced ETL/Data Engineer with strong PySpark and Spark SQL skills to design and deliver complex ETL pipelines in a banking domain context.

The role involves technical leadership, collaborating with clients, and ensuring high-quality, scalable data processing solutions using Spark, SQL, and related tools. On-site work in Hyderabad is expected.

Qualifications

• Proficiency in ETL design and technical problem solving.
• Ability to translate business requirements into technical specifications and execution plans.
• Experience with effort estimation, task assignment, and performance tuning.

Responsibilities

• ETL technical designing, guiding team for solving technical problems, providing technical solution to client.
• Understanding business requirements and converting to technical specification, execution planning.
• Effort estimation, task assignment to the team and tracking of activities, performance tuning.
• Hands-on development for complex modules, ensuring coding standards and quality of delivery.

Skills

PySpark and Spark SQL

Dataframes APIs

Spark Functions

Advanced Data transformations

SQL knowledge

UDFS

Error handling & logging

Unix & Scheduling tools

Banking domain knowledge

Tools

Teradata BTEQ

ETL Estimation

Job description

Competencies Required

Essentials

• PySpark and Spark SQL
• Working with Dataframes using different APIs
• Spark Functions
• Advanced Data transformations
• Good knowledge in SQL
• Applying UDFS
• Error handling, logging and monitoring
• Unix, Scheduling Tool
• Performance Tuning & Generic process development
• Banking Domain Knowledge

Desirable

• ETL Estimation
• Teradata BTEQ

Responsibility of / Expectations from the Role

1. ETL Technical Designing, Guiding Team for solving Technical Problems, Providing Technical Solution to Client

2. Understanding Business Requirement and converting to Technical specification, Execution Planning

3. Effort Estimation, Task assignment to the team and tracking of activities, Performance Tuning

4. Hands on development for complex modules, ensuring coding standards, quality of delivery