Pyspark Engineer

Synechron; Chennai District; Hybrid; INR 900,000 - 1,300,000; Full time

Job summary

Synechron in Chennai, Tamil Nadu, India, is seeking a experienced Big Data professional to design and optimize large-scale data processing pipelines using PySpark on Cloudera. You will collaborate with cross-functional teams to translate business requirements into scalable data solutions and write complex SQL to extract insights from both structured and unstructured data.

The role emphasizes data quality checks, ETL troubleshooting, and ensuring robust data ingestion processes.

Qualifications

• 5-10 years of IT experience with expertise in Big Data technologies such as Hadoop ecosystem (HDFS, YARN, MapReduce).
• Proficiency in Informatica BDM Development tools for technical data mapping and extraction.
• Proficiency in Apache Oozie workflow management system for scheduling jobs across multiple nodes.
• Experience with IFRS9 application development is an added advantage.

Responsibilities

• Design, develop, test, deploy, and maintain large-scale data processing pipelines using PySpark on Cloudera.
• Collaborate with cross-functional teams to identify business requirements and design solutions that meet those needs.
• Develop complex SQL queries to extract insights from structured and unstructured data sources.
• Implement data quality checks and validation processes to ensure high-quality data ingestion.
• Troubleshoot issues related to data extraction, transformation, loading (ETL), and reporting.

Tools

Hadoop ecosystem (HDFS, YARN, MapReduce)

Informatica BDM Development

Apache Oozie

Job description

Roles and Responsibilities

• Design, develop, test, deploy, and maintain large-scale data processing pipelines using PySpark on Cloudera.
  
• Collaborate with cross-functional teams to identify business requirements and design solutions that meet those needs.
  
• Develop complex SQL queries to extract insights from structured and unstructured data sources.
  
• Implement data quality checks and validation processes to ensure high-quality data ingestion.
  
• Troubleshoot issues related to data extraction, transformation, loading (ETL), and reporting.
  

Desired Candidate Profile

• 5-10 years of experience in IT industry with expertise in Big Data technologies such as Hadoop ecosystem (HDFS, YARN, MapReduce).
  
• Strong understanding of Informatica BDM Development tools for technical data mapping and extraction.
  
• Proficiency in Apache Oozie workflow management system for scheduling jobs across multiple nodes.
  
• Experience working with IFRS9 application development is an added advantage.