Software Associate

Tata Consultancy Services; Bengaluru; On-site; INR 1,800,000 - 2,400,000; Full time

Job summary

Tata Consultancy Services in Bengaluru is seeking a Senior Data Engineer to design, develop, and optimize scalable data pipelines using Scala and PySpark in an AWS environment. You will build batch and real-time ingestion and processing frameworks, and contribute to data warehousing initiatives.

The role requires strong SQL, cloud experience, and collaboration with data scientists and architects to translate business needs into robust, cost-efficient solutions, with a focus on data quality,

Qualifications

• Proficient in designing scalable data pipelines with Scala and PySpark.
• Experience with AWS data tooling and cloud-native architectures.
• Strong knowledge of data warehousing principles and best practices.

Responsibilities

• Design, develop, and optimize scalable data pipelines using scala and Pyspark in AWS environment.
• Build and maintain batch and real-time data ingestion and processing frameworks.
• Develop enterprise-grade data warehousing solutions using Scala and Pyspark.
• Analyze existing Scala and Spark applications and identify migration requirements.
• Convert Scala-based ETL, batch, and streaming pipelines into PySpark frameworks.
• Optimize PySpark jobs for performance, scalability, and resource utilization.
• Support cloud modernization initiatives on AWS/Databricks/Snowflake platforms.
• Implement ETL/ELT processes for structured and unstructured data.
• Integrate data from multiple sources including databases, APIs, files, and streaming platforms.
• Ensure data quality, governance, security, and compliance across data platforms.
• Automate deployment and operational processes using CI/CD and Infrastructure as Code (IaC).
• Monitor data pipelines and troubleshoot production issues.
• Collaborate with Data Architects, Business Analysts, and Data Scientists to translate business requirements into technical solutions.
• Implement data models, metadata management, and data lineage best practices.
• Support migration of on-premises or multi-cloud data platforms to AWS.
• Lead the migration, modernization, and optimization of Scala and Apache Spark workloads on Cloud environment, including conversion of Scala-based Spark applications to PySpark, performance tuning, cluster optimization, dependency management, and ensuring scalable, cost-effective, and resilient data processing solutions.

Skills

Scala

PySpark

Python

SQL (Advanced)

Team collaboration

Documentation and knowledge sharing

Education

Bachelor's/Master's in Computer Science or equivalent

Tools

AWS Glue

AWS S3

Step Functions

Glue

Lambda

Pub/Sub

Event Bridge

ECS

EKS

Snowflake

Job description

Key Responsibilities*

• Design, develop, and optimize scalable data pipelines using scala and Pyspark in AWS environment.
• Build and maintain batch and real-time data ingestion and processing frameworks.
• Develop enterprise-grade data warehousing solutions using Scala and Pyspark.
• Analyze existing Scala and Spark applications and identify migration requirements.
• Convert Scala-based ETL, batch, and streaming pipelines into PySpark frameworks.
• Optimize PySpark jobs for performance, scalability, and resource utilization.
• Support cloud modernization initiatives on AWS/Databricks/Snowflake platforms.
• Implement ETL/ELT processes for structured and unstructured data.
• Integrate data from multiple sources including databases, APIs, files, and streaming platforms.
• Ensure data quality, governance, security, and compliance across data platforms.
• Automate deployment and operational processes using CI/CD and Infrastructure as Code (IaC).
• Monitor data pipelines and troubleshoot production issues.
• Collaborate with Data Architects, Business Analysts, and Data Scientists to translate business requirements into technical solutions.
• Implement data models, metadata management, and data lineage best practices.
• Support migration of on-premises or multi-cloud data platforms to AWS.
• Lead the migration, modernization, and optimization of Scala and Apache Spark workloads on Cloud environment, including conversion of Scala-based Spark applications to PySpark, performance tuning, cluster optimization, dependency management, and ensuring scalable, cost-effective, and resilient data processing solutions.

Required Technical Skills

• Scala
• PySpark
• AWS Glue
• AWS S3
• Step Functions
• Glue
• Lambda
• Pub/Sub
• Python
• SQL (Advanced)
• Event Bridge
• ECS
• EKS
• Snowflake

Section IV - Job Qualifications & Skills

CMTS

Soft Skills

• Team collaboration
• Documentation and knowledge sharing

Education Requirements

Bachelor's/master's in computer science or equivalent (Preferred)