PySpark/Scala Developer

Tata Consultancy Services; Pune District, Chennai District; On-site; INR 2,500,000 - 4,500,000; Full time

Job summary

Tata Consultancy Services in Pune is seeking experienced PySpark/Scala Developers to join a global banking risk analytics team. You will design and build large-scale data pipelines and production-grade data solutions on Big Data platforms.

The role emphasizes PySpark, Scala, Hadoop ecosystem, and experience within Credit Risk or Regulatory Risk in Financial Services. You will collaborate with Risk Analytics, Data Science, and Business teams to deliver data solutions and ML pipelines.

Qualifications

• 5+ years of experience in Data Engineering, Big Data Development, or Software Development.
• Hands-on experience in PySpark, Scala, Python, Spark, Hive, Hadoop Ecosystem, MapReduce, Unix Shell Scripting, SQL.
• Experience building enterprise-grade data pipelines and ingestion frameworks.
• Experience with distributed computing environments such as Hadoop, Spark, Databricks, or AWS EMR.
• Strong understanding of Data Lake and Data Warehouse concepts.
• Experience in Banking/Financial Services, Credit Risk, or Regulatory Risk projects.
• Knowledge of relational databases like PostgreSQL, Redshift, or equivalents.

Responsibilities

• Design, develop, and optimize scalable ETL/ELT pipelines using PySpark and Scala.
• Build ingestion frameworks for structured, semi-structured, and unstructured data sources.
• Develop machine learning model pipelines with data scientists.
• Implement feature engineering, data transformation, and model scoring.
• Build reusable Spark-based frameworks for batch and near real-time processing.
• Optimize Big Data workloads on Spark, Hadoop, EMR, Databricks, or similar platforms.
• Ensure data quality, lineage, governance, and documentation standards.
• Collaborate with Risk Analytics, Data Science, and Business teams to deliver data solutions.
• Support deployment, version control, CI/CD practices, and production support activities.

Skills

PySpark

Scala

Python

Spark

Hive

Hadoop Ecosystem

MapReduce

Unix Shell Scripting

SQL

Education

Engineering

Statistics

Mathematics

Finance

Computer Science

Tools

Databricks

Hadoop

EMR

AWS

Job description

About the Role

We are looking for skilled PySpark/Scala Developers to join a global banking risk analytics team. The role involves designing and developing large-scale data pipelines, implementing machine learning models, and building robust data processing solutions on Big Data platforms.

The ideal candidate will have expertise in PySpark, Scala, Hadoop ecosystem technologies, and experience within Credit Risk, Regulatory Risk, Banking, or Financial Services domains.

Key Responsibilities

• Design, develop, and optimize scalable ETL/ELT pipelines using PySpark and Scala.
• Build ingestion frameworks for structured, semi-structured, and unstructured data sources.
• Develop and enhance machine learning model pipelines in partnership with data scientists.
• Implement feature engineering, data transformation, model scoring, and retraining processes.
• Build reusable Spark-based frameworks for batch and near real-time data processing.
• Optimize Big Data workloads running on Spark, Hadoop, EMR, Databricks, or similar platforms.
• Ensure data quality, lineage, governance, and documentation standards.
• Collaborate with Risk Analytics, Data Science, and Business teams to deliver data solutions.
• Support deployment, version control, CI/CD practices, and production support activities.

Mandatory Skills

• 5+ years of experience in Data Engineering, Big Data Development, or Software Development.
• Strong hands-on experience in:
  • PySpark
  • Scala
  • Python
  • Spark
  • Hive
  • Hadoop Ecosystem
  • MapReduce
  • Unix Shell Scripting
  • SQL
• Experience building enterprise-grade data pipelines and ingestion frameworks.
• Experience with distributed computing environments such as Hadoop, Spark, Databricks, or AWS EMR.
• Strong understanding of Data Lake and Data Warehouse concepts.
• Experience working with Banking, Financial Services, Credit Risk, or Regulatory Risk projects.
• Knowledge of relational databases and platforms such as PostgreSQL, Redshift, or equivalent.

Preferred Skills

• Spark Structured Streaming
• Kafka
• Delta Lake
• AWS Glue
• Machine Learning frameworks (Spark ML, Scikit-Learn, TensorFlow, PyTorch)
• Jenkins, Git, CI/CD pipelines
• Data Governance and Data Lineage tools

Domain Preference

Candidates with experience in:

• Credit Risk
• Regulatory Risk
• Risk Modeling
• Consumer Banking
• Wholesale Banking
• Financial Services

Education

Master's or Bachelor's Degree in:

• Engineering
• Statistics
• Mathematics
• Finance
• Computer Science