Pyspark

Tata Consultancy Services; Chennai District; On-site; INR 1,800,000 - 2,600,000; Full time

Job summary

Tata Consultancy Services in Chennai, Hyderabad and Kolkata seeks a data engineer with 6-8 years of IT experience to design, develop and deploy big data ETL jobs using PySpark and SparkSQL. You will build Python-based apps, write complex SQL, and participate in CI/CD processes while leveraging AWS services and modern data lake tooling.

The role requires strong problem solving, collaboration with architects, and responsibility for performance tuning, testing support, and documentation throughout

Qualifications

• Experience in design, development and deployment of big data applications and ETL jobs using PySpark APIs/sparkSQL.
• Experience in design, build and deployment of Python based applications.
• Experience in writing complex SQL queries/procedures using Relational databases like SQL Server or Oracle.
• Experience in version control system like Git and CI/CD pipeline is a must.
• Experience in Delta lake APIs is a plus
• Experience in Docker and Kubernetes is a plus
• Knowledge of AWS services like S3, Athena, Glue, Lambda, Redshift or Cloud platform is a plus
• 15 years of Full Time Education

Responsibilities

• Designing, implementing, and maintaining jobs written in PySpark for extracting raw data, applying transformation, and writing into file/tables on Apache Spark cluster.
• Writing complex queries as per the business rules and data extraction logic
• Query tuning and Performance optimization of various components as and when required.
• Work with development leads, System Architects and other teams (as needed) to manage dependencies, risks and issues.
• Contributing in all phases of the development lifecycle
• Support testers and defect resolution
• Update status/risks/issues/impediments in daily scrum calls to the customer and track it to closure.
• Writing testable, scalable and efficient code
• Working on the necessary documentations as needed

Skills

PySpark

SparkSQL

Python

SQL

Git

CI/CD

Delta Lake

Docker

Kubernetes

AWS

Education

15 years of full-time education

Tools

Git

CI/CD tools

Delta Lake

Docker

Kubernetes

SQL Server

Oracle

Job description

JOB DESCRIPTION

Experience Range: - 06 To 08 Years (Mandatory)

(Note: Candidates below 6 years of total IT experience shall not be considered)

JOB LOCATION: Chennai, Hyderabad, Kolkata

Technical Skills:

• Experience in design, development and deployment of big data applications and ETL jobs using PySpark APIs/sparkSQL.
• Experience in design, build and deployment of Python based applications.
• Experience in writing complex SQL queries/procedures using Relational databases like SQL Server or Oracle
• Experience in version control system like Git and CI/CD pipeline is a must.
• Experience in Delta lake APIs is a plus
• Experience in Docker and Kubernetes is a plus
• Knowledge of AWS services like S3, Athena, Glue, Lambda, Redshift or Cloud platform is a plus

Responsibilities:

• Designing, implementing, and maintaining jobs written in PySpark for extracting raw data, applying transformation, and writing into file/tables on Apache Spark cluster.
• Writing complex queries as per the business rules and data extraction logic
• Query tuning and Performance optimization of various components as and when required.
• Work with development leads, System Architects and other teams (as needed) to manage dependencies, risks and issues.
• Contributing in all phases of the development lifecycle
• Support testers and defect resolution
• Update status/risks/issues/impediments in daily scrum calls to the customer and track it to closure.
• Writing testable, scalable and efficient code
• Working on the necessary documentations as needed

15 years of Full Time Education