Spark / Scala Data Engineer

Tata Consultancy Services; Bengaluru; On-site; INR 4,000,000 - 6,500,000; Full time

Job summary

Tata Consultancy Services in Bengaluru, India, seeks a Spark / Scala Data Engineer with 6–10 years of experience to design, build, and optimize scalable data pipelines on AWS. You will collaborate with data architects, scientists, and product teams to deliver high‑performant data platforms and modern cloud workloads.

Responsibilities include migrating Scala/Spark apps to PySpark, implementing ETL/ELT processes, and leveraging Databricks and Snowflake for analytics, data warehousing, and

Qualifications

• Experience designing and maintaining scalable data pipelines on AWS.
• Strong expertise in Scala and PySpark for ETL/ELT workloads.
• Hands-on history with data warehousing, data governance, and cloud analytics.

Responsibilities

• Design, develop, and optimize scalable data pipelines in AWS using Scala and PySpark.
• Build batch and real-time ingestion and processing frameworks.
• Migrate Scala/Spark apps to PySpark and optimize performance and cost.
• Implement CI/CD and IaC for automated deployment and management.
• Collaborate with data architects, scientists, and stakeholders on data platforms.

Skills

Scala

PySpark

AWS

Databricks

Snowflake

Data pipelines

Data warehousing

CI/CD

IaC

Cloud migration

Tools

Databricks

Snowflake

Job description

Greetings from Tata Consultancy Services (TCS)!!!

Job Title : Spark / Scala Data Engineer

Experience : 6- 10 Years

Job Requirements ;

We are seeking an experienced Scala / Pyspark associate to design, develop, and maintain scalable data solutions on the Amazon Cloud Platform (AWS). The ideal candidate will have strong expertise in building modern data pipelines, data warehousing, big data processing, and cloud-native analytics solutions. The role requires close collaboration with business stakeholders, data architects, data scientists, and application teams to deliver reliable and high-performance data platforms.

Key Responsibilities

• Design, develop, and optimize scalable data pipelines using scala and Pyspark in AWS environment.
• Build and maintain batch and real-time data ingestion and processing frameworks.
• Develop enterprise-grade data warehousing solutions using Scala and Pyspark.
• Analyze existing Scala and Spark applications and identify migration requirements.
• Convert Scala-based ETL, batch, and streaming pipelines into PySpark frameworks.
• Optimize PySpark jobs for performance, scalability, and resource utilization.
• Support cloud modernization initiatives on AWS/Databricks/Snowflake platforms.
• Implement ETL/ELT processes for structured and unstructured data.
• Integrate data from multiple sources including databases, APIs, files, and streaming platforms.
• Ensure data quality, governance, security, and compliance across data platforms.
• Automate deployment and operational processes using CI/CD and Infrastructure as Code (IaC).
• Monitor data pipelines and troubleshoot production issues.
• Collaborate with Data Architects, Business Analysts, and Data Scientists to translate business requirements into technical solutions.
• Implement data models, metadata management, and data lineage best practices.
• Support migration of on-premises or multi-cloud data platforms to AWS.
• Lead the migration, modernization, and optimization of Scala and Apache Spark workloads on Cloud environment, including conversion of Scala-based Spark applications to PySpark, performance tuning, cluster optimization, dependency management, and ensuring scalable, cost-effective, and resilient data processing solutions.