Senior Data Engineer

TRUSTKLUB Consulting; Navi Mumbai, Mumbai; On-site; INR 1,200,000 - 1,800,000; Full time

Job summary

TRUSTKLUB Consulting in Navi Mumbai, Maharashtra, is seeking an experienced Data Engineer to design and implement large-scale data pipelines using Apache Spark, PySpark, and SQL. You will collaborate with cross-functional teams to gather requirements and deliver robust ETL solutions.

The role focuses on building data models, performing EDA, and optimizing queries for efficient processing, while ensuring timely delivery and addressing data quality and performance issues.

Qualifications

• 4–7 years of experience in Data Engineering with expertise in Spark/PySpark/SQL.
• Bachelor's degree in Computer Science and Engineering (CSE) or equivalent from a reputed institution.
• Strong understanding of Big Data concepts including Hadoop ecosystem components such as HDFS/Hive/Pig.
• Proficient in developing complex SQL queries with knowledge of UDFs and views.

Responsibilities

• Design, develop, and maintain large-scale data pipelines using Spark, PySpark, and SQL.
• Collaborate with cross-functional teams to gather requirements and deliver high-quality ETL solutions.
• Develop data models, perform EDA, query optimization, and database design for efficient data processing.
• Ensure timely delivery of projects by managing project timelines, resources, and stakeholders effectively.
• Troubleshoot issues related to big data processing, data quality, and performance tuning.

Skills

Analytical mindset

Attention to detail

Excellent communication

Education

Bachelor's degree in Computer Science and Engineering (CSE) or equivalent

Tools

Apache Spark

PySpark

SQL

Hadoop

HDFS

Hive

Pig

Job description

Roles and Responsibilities

• Design, develop, and maintain large-scale data pipelines using Apache Spark, PySpark, and SQL.
• Collaborate with cross-functional teams to gather requirements and deliver high-quality ETL solutions.
• Develop complex data models, perform exploratory data analysis (EDA), query optimization, and database design for efficient data processing.
• Ensure timely delivery of projects by managing project timelines, resources, and stakeholders effectively.
• Troubleshoot issues related to big data processing, data quality, and performance tuning.

Desired Candidate Profile

• 4-7 years of experience in Data Engineering with expertise in Apache Spark/PySpark/SQL.
• Bachelor's degree in Computer Science and Engineering (CSE) or equivalent from a reputed institution.
• Strong understanding of Big Data concepts including Hadoop ecosystem components such as HDFS/Hive/Pig etc. .
• Proficiency in developing complex SQL queries with knowledge of user-defined functions (UDFs) and views.