Big Data Engineer (PySpark & Scala)

Tata Consultancy Services Limited; Singapore; On-site; SGD 90,000 - 150,000; Full time

Job summary

Tata Consultancy Services Limited in Singapore is seeking a highly skilled Big Data Engineer with strong PySpark and Scala to design, build, and optimize large-scale data processing systems across enterprise data ecosystems. You will develop scalable data pipelines, data lakes, and streaming solutions on distributed platforms, enabling real-time and batch analytics, data quality, and governance.

The role emphasizes collaboration with cross-functional teams and delivery of robust data solutions

Qualifications

• Strong PySpark + Scala coding
• Deep big data architecture knowledge
• Hands-on experience with streaming + cloud data engineering
• Knowledge of data compliance and regulatory reporting

Responsibilities

• Design and develop scalable data pipelines using PySpark and Scala
• Data Platform & Architecture
• Data Processing & Optimization
• Streaming & Real-Time Data
• Cloud & Modern Data Stack
• Data Quality & Governance
• Collaboration & Delivery

Skills

PySpark

Scala programming

SQL

Python

Unix

Tools

Kafka

Sqoop

Airflow

Docker

Kubernetes

Git

CI/CD

Job description

Role Summary

We are seeking a highly skilled Big Data Engineer with strong experience in PySpark and Scala to design, build, and optimize large-scale data processing systems. The candidate will work on distributed data platforms, enabling real-time and batch analytics solutions across enterprise data ecosystems.

The role involves building scalable data pipelines, data lakes, and streaming solutions leveraging modern big data technologies.

Key Responsibilities

• Data Engineering & Development
• Design and develop scalable data pipelines using PySpark and Scala
• Data Platform & Architecture
• Data Processing & Optimization
• Streaming & Real-Time Data
• Cloud & Modern Data Stack
• Data Quality & Governance
• Collaboration & Delivery
• Required Skills
• Core Technologies
• Strong hands-on experience in:
• PySpark & Apache Spark
• Scala programming
• Expertise in:
• Spark SQL, DataFrames, RDDs
• Distributed data processing frameworks
• Big Data Ecosystem
• Hadoop: HDFS, Hive, YARN
• Data ingestion tools: Kafka, Sqoop
• Data formats: Parquet, ORC, JSON, Avro
• Programming & Query
• Python / Scala
• Advanced SQL (joins, aggregations, optimization)
• Shell ing / Unix basics
• Streaming & Messaging
• Kafka (must-have)
• Event-driven architecture
• Real-time data processing frameworks
• DevOps & Tools
• Git, CI/CD pipelines
• Docker / Kubernetes (good to have)
• Scheduling tools (Airflow / Control-M)
• Cloud (One or more preferred)
• AWS / Azure / GCP
• Databricks / Snowflake exposure
• ibu
• itcan
• Domain Experience (Preferred)
• BFSI / Banking / Payments domain
• Experience working with high-volume financial datasets
• Knowledge of data compliance and regulatory reporting
• This role requires a combination of:
• Strong PySpark + Scala coding
• Deep big data architecture knowledge
• Hands-on experience with streaming + cloud data engineering