Software Associate

Tata Consultancy Services; Hyderabad; On-site; INR 1,800,000 - 3,200,000; Full time

Job summary

Tata Consultancy Services is seeking an experienced Scala/PySpark associate to design, develop, and maintain scalable data solutions on the AWS cloud. The role emphasizes building modern data pipelines, data warehousing, and cloud-native analytics to support business stakeholders and data teams.

Key focus areas include migrating workloads to AWS, optimizing Spark jobs, and ensuring data governance, quality, and security across platforms.

Qualifications

• BE/BTech/Master’s in CS or equivalent; AWS data engineering is preferred.
• Strong knowledge of Scala or PySpark is required for data pipelines.
• Experience with AWS data services and cloud-native analytics.

Responsibilities

• Design, develop, and maintain scalable data pipelines on AWS using Scala/PySpark.
• Build batch and real-time ingestion and processing frameworks.
• Develop data warehousing solutions and implement ETL/ELT processes.
• Collaborate with data architects, scientists, and engineers to translate requirements.
• Migrate on-prem/multi-cloud workloads to AWS and optimize Spark workloads.

Skills

Excellent communication

Team collaboration

Documentation and knowledge sharing

Education

Bachelor's/Master’s in computer science

Tools

AWS Glue

AWS S3

Step Functions

Glue

Lambda

Pub/Sub

Python

SQL

Event Bridge

ECS

EKS

Snowflake

Job description

We are seeking an experienced Scala / Pyspark associate to design, develop, and maintain scalable data solutions on the Amazon Cloud Platform (AWS). The ideal candidate will have strong expertise in building modern data pipelines, data warehousing, big data processing, and cloud-native analytics solutions. The role requires close collaboration with business stakeholders, data architects, data scientists, and application teams to deliver reliable and high performance data platforms.

Key Responsibilities

• Design, develop, and optimize scalable data pipelines using scala and Pyspark in AWS environment.
• Build and maintain batch and real-time data ingestion and processing frameworks.
• Develop enterprise-grade data warehousing solutions using Scala and Pyspark.
• Implement ETL/ELT processes for structured and unstructured data.
• Analyze existing Scala and Spark applications and identify migration requirements.
• Convert Scala-based ETL, batch, and streaming pipelines into PySpark frameworks.
• Optimize PySpark jobs for performance, scalability, and resource utilization.
• Support cloud modernization initiatives on AWS/Databricks/Snowflake platforms.
• Integrate data from multiple sources including databases, APIs, files, and streaming platforms.
• Ensure data quality, governance, security, and compliance across data platforms.
• Automate deployment and operational processes using CI/CD and Infrastructure as Code (IaC).
• Monitor data pipelines and troubleshoot production issues.
• Collaborate with Data Architects, Business Analysts, and Data Scientists to translate business requirements into technical solutions.
• Implement data models, metadata management, and data lineage best practices.
• Support migration of on-premises or multi-cloud data platforms to AWS.
• Lead the migration, modernization, and optimization of Scala and Apache Spark workloads on Cloud environment, including conversion of Scala-based Spark applications to PySpark, performance tuning, cluster optimization, dependency management, and ensuring scalable, cost-effective, and resilient data processing solutions.

Required Technical Skills

• Scala
• PySpark
• AWS Glue
• AWS S3
• Step Functions
• Glue
• Lambda
• Pub/Sub
• Python
• SQL (Advanced)
• Event Bridge
• ECS
• EKS
• Snowflake

Domain

• CMTS

Soft Skills

• Excellent communication
• Team collaboration
• Documentation and knowledge sharing

Education Requirements

• Bachelor's/master’s in computer science or equivalent (Preferred)

Certifications

• AWS Cloud Data Engineer (Preferred)
• Snow pro certifications