PySpark Developer

Tata Consultancy Services; Pune District; On-site; INR 1,500,000 - 2,600,000; Full time

Job summary

Tata Consultancy Services is seeking skilled PySpark Developers to join a global banking team in Pune, India. You will design and develop large-scale data pipelines, implement ML models, and build robust data processing solutions on Big Data platforms.

The ideal candidate has 5+ years of experience in Data Engineering or Software Development with hands-on PySpark, Scala, Hadoop ecosystem, and exposure to Banking or Financial Services domains.

Qualifications

• 5+ years of experience in Data Engineering, Big Data Development, or Software Development.
• Strong hands-on experience in PySpark and Scala.
• Proficiency in Python, Spark, Hive, and Hadoop ecosystem.
• Experience with MapReduce, Unix shell scripting, and SQL.
• Experience building enterprise-grade data pipelines and ingestion frameworks.
• Exposure to Banking/Financial Services, Credit Risk, or Regulatory Risk projects.

Responsibilities

• Design, develop, and optimize scalable ETL/ELT pipelines using PySpark and Scala.
• Build ingestion frameworks for structured, semi-structured, and unstructured data sources.
• Develop and enhance ML model pipelines with data scientists.
• Implement feature engineering, data transformation, and model scoring workflows.
• Build reusable Spark-based frameworks for batch and near real-time processing.
• Optimize Big Data workloads on Spark, Hadoop, EMR, Databricks or similar platforms.
• Ensure data quality, lineage, governance, and documentation.
• Collaborate with Risk Analytics, Data Science and Business teams to deliver data solutions.
• Support deployment, version control, CI/CD practices and production support.

Skills

Data Engineering

Big Data Development

Software Development

ETL/ELT Pipelines

ML Pipelines

Tools

PySpark

Scala

Python

Spark

Hive

Hadoop Ecosystem

MapReduce

Unix Shell Scripting

SQL

Job description

We are looking for skilled PySpark Developers to join a global banking team. The role involves designing and developing large-scale data pipelines, implementing machine learning models, and building robust data processing solutions on Big Data platforms.

The ideal candidate will have expertise in PySpark, Scala, Hadoop ecosystem technologies, and experience within Credit Risk, Regulatory Risk, Banking, or Financial Services domains.

Key Responsibilities

• Design, develop, and optimize scalable ETL/ELT pipelines using PySpark and Scala.


• Build ingestion frameworks for structured, semi-structured, and unstructured data sources.


• Develop and enhance machine learning model pipelines in partnership with data scientists.


• Implement feature engineering, data transformation, model scoring, and retraining processes.


• Build reusable Spark-based frameworks for batch and near real-time data processing.


• Optimize Big Data workloads running on Spark, Hadoop, EMR, Databricks, or similar platforms.


• Ensure data quality, lineage, governance, and documentation standards.


• Collaborate with Risk Analytics, Data Science, and Business teams to deliver data solutions.


• Support deployment, version control, CI/CD practices, and production support activities.

Mandatory Skills

• 5+ years of experience in Data Engineering, Big Data Development, or Software Development.


• Strong hands-on experience in:


• PySpark


• Scala


• Python


• Spark


• Hive


• Hadoop Ecosystem


• MapReduce


• Unix Shell Scripting


• SQL


• Experience building enterprise-grade data pipelines and ingestion frameworks.


• Experience with distributed computing environments such as Hadoop, Spark, Databricks, or AWS EMR.


• Strong understanding of Data Lake and Data Warehouse concepts.


• Experience working with Banking, Financial Services, Credit Risk, or Regulatory Risk projects.


• Knowledge of relational databases and platforms such as PostgreSQL, Redshift, or equivalent.

Preferred Skills

• Spark Structured Streaming


• Kafka


• AWS Glue


• Machine Learning frameworks (Spark ML, Scikit-Learn, TensorFlow, PyTorch)