Big Data Engineer -Pyspark, Spark, Hadoop, SQL

Pi Square Technologies; Hyderabad, Pune District, Bengaluru; On-site; INR 1,800,000 - 3,000,000; Full time

Job summary

Pi Square Technologies is hiring an experienced Big Data Engineer to design and develop large-scale data pipelines using PySpark, Spark, and Hadoop. The role involves collaborating with global delivery teams to build scalable analytics solutions and ensure data quality.

The candidate will work in a hybrid setup with 3 days in the office per week and is expected to have 6+ years of experience, strong SQL skills, and expertise in Spark architecture. Immediate joiner preferred.

Qualifications

• At least 6+ years of experience in designing and developing large scale, distributed data processing pipelines using PySpark, Hadoop and related technologies.
• Having expertise in Pyspark, Spark Core, Spark SQL, Batch processing and Spark Streaming.
• Experience with Hadoop, HDFS, Hive and other BigData technologies.
• Familiarity with Data warehousing and ETL concepts and techniques.
• UNIX shell scripting will be an added advantage in scheduling/running application jobs.
• At least 5 years of experience in Project development life cycle activities and development/maintenance projects.
• Work with business stakeholders and other SMEs to understand high level business requirements.
• Work with the Solution Designers and contribute to the development of project plans by participating in the scoping and estimating of proposed project.
• Apply technical background understanding, business knowledge, system knowledge in the elicitation of Systems Requirements for projects.
• Possess good knowledge on Spark architecture and transformations using Spark and PySpark.
• Work in an Agile environment and participation in scrum daily standups, sprint planning reviews and retrospectives.
• Understand project requirements and translate them into technical solutions which meets the project quality standards.
• Ability to work in team in diverse/multiple stakeholder environment and collaborate with upstream/downstream functional teams to identify, troubleshoot and resolve data issues.
• Strong problem solving and Good Analytical skills.
• Excellent verbal and written communication skills.
• Experience and desire to work in a Global delivery environment.
• Stay up to date with new technologies and industry trends in Development.

Responsibilities

• Design and develop large-scale distributed data pipelines using PySpark and Hadoop.
• Build analytics solutions and work with cross-functional teams in a global delivery environment.
• Collaborate with stakeholders to translate requirements into scalable data processes.
• Implement Spark Streaming and batch processing pipelines.
• Perform data quality checks and optimize performance.
• Participate in Agile ceremonies.

Skills

Pyspark

Spark

Hadoop

SQL

Spark Core

Spark SQL

Batch processing

Spark Streaming

Tools

Hive

Sqoop

UNIX shell scripting

Job description

Job Location: Hyderabad, Pune, Chennai, Bangalore, Trivandrum

Work mode : Hybrid- 3 days WFO weekly

Important :

Notice period : Immediate to 15-30 days ONLY.

Candidates serving Notice period will be preferred.

Please note - There should not be Career Gap more than 6 months(No Career Break)

Graduation score should be 60% & above

BGV (Background Verification)is mandatory.

We are hiring experienced Big Data Engineers with strong expertise in PySpark, Spark, Hadoop, and SQL to build scalable data pipelines and analytics solutions. The role involves working with distributed systems, large datasets, and cross-functional teams in a global delivery environment.

Must have skills - 2 skills which are non-negotiable -

• Pyspark, Spark, Hadoop, SQL

Desirable skills - 1 skill which is nice to have -

• Hive, Sqoop, UNIX shell scripting

• At least 6+ years of experience in designing and developing large scale, distributed data processing pipelines using PySpark, Hadoop and related technologies.
• Having expertise in Pyspark, Spark Core, Spark SQL, Batch processing and Spark Streaming.
• Experience with Hadoop, HDFS, Hive and other BigData technologies.
• Familiarity with Data warehousing and ETL concepts and techniques
• UNIX shell scripting will be an added advantage in scheduling/running application jobs.
• At least 5 years of experience in Project development life cycle activities and development/maintenance projects
• Work with business stakeholders and other SMEs to understand high level business requirements.
• Work with the Solution Designers and contribute to the development of project plans by participating in the scoping and estimating of proposed project.
• Apply technical background understanding, business knowledge, system knowledge in the elicitation of Systems Requirements for projects.
• Possess good knowledge on Spark architecture and transformations using Spark and PySpark.
• Work in an Agile environment and participation in scrum daily standups, sprint planning reviews and retrospectives.
• Understand project requirements and translate them into technical solutions which meets the project quality standards.
• Ability to work in team in diverse/multiple stakeholder environment and collaborate with upstream/downstream functional teams to identify, troubleshoot and resolve data issues.
• Strong problem solving and Good Analytical skills.
• Excellent verbal and written communication skills.
• Experience and desire to work in a Global delivery environment.
• Stay up to date with new technologies and industry trends in Development.