Data Engineer

Advance Career Solutions; Thane, Mumbai; On-site; INR 1,800,000 - 2,800,000; Full time

Job summary

Advance Career Solutions in Thane is seeking a Data Engineer with 5-8 years of hands-on experience to design, develop, and maintain high-performance data pipelines using PySpark and related Big Data technologies.

The role involves working with large datasets across distributed environments, collaborating with analytics teams, and ensuring robust, scalable solutions in Hadoop ecosystems including Cloudera, Hive, and Spark SQL. Proficiency in SQL, Git, and API data integration is essential.

Qualifications

• 5–8 years of hands-on Data Engineering experience.
• Strong experience with PySpark and Hadoop ecosystem.
• Knowledge of data pipelines from ingestion to consumption in distributed environments.

Responsibilities

• Develop and maintain robust data pipelines using PySpark.
• Handle large-scale datasets from ingestion to consumption.
• Collaborate with business teams to define data needs and solutions.
• Design data workflows across data lakes in Hadoop ecosystems.
• Write optimized SQL queries using Hive or Spark SQL.
• Work with HDFS, YARN, HBase, Oozie, Hive, MapReduce, and Spark Streaming.
• Maintain code quality with Git and perform debugging and tuning.
• Create design docs for ETL pipelines to ensure reproducibility.
• Integrate data via APIs across systems.

Skills

Analytical skills

Problem solving

Communication skills

Tools

PySpark

Hadoop

Cloudera

SQL

Hive

Spark SQL

HDFS

YARN

HBase

Oozie

MapReduce

Spark Streaming

Git

Job description

Role: Data Engineer (PySpark, Hadoop, Cloudera and SQL )

Experience: 5 to 8 years

Location: Thane

We are seeking a skilled and passionate Data Engineer with 4-10 years of experience to join our data engineering team The ideal candidate will play a key role in designing developing and maintaining high performance data pipelines using PySpark and other Big Data technologies

Key Responsibilities

• 1 Develop and maintain robust data pipelines using PySpark ensuring they are scalable efficient and optimized for performance and availability
• 2 Work with large scale datasets managing the full data lifecycle from ingestion transformation to consumption in a distributed environment
• 3 Collaborate with business and analytics teams to identify data requirements understand use cases and deliver technical solutions that solve complex business problems
• 4 Design and implement data workflows across data lakes with hands on experience in Hadoop ecosystems Cloudera
• 5 Strong command of SQL with the ability to write optimized queries for large datasets in tools such as Hive or Spark SQL
• 6 Experience working with big data tools like HDFS YARN HBase Oozie Hive Map Reduce and building streaming solutions using Spark Streaming
• 7 Maintain code quality and version control using Git and perform thorough technical debugging and performance tuning of data jobs
• 8 Create and maintain technical design documentation for all ETL pipelines and solutions developed ensuring transparency and reproducibility
• 9 Knowledge of working with APIs for data integration and building services that interface between various systems

Preferred SkillsQualifications

• 4-7 years of hands‑on experience in Data Engineering with a strong emphasis on PySpark and the Hadoop ecosystem
• Solid understanding of distributed computing principles and data architecture best practices
• Experience in handling large scale datasets and designing fault tolerant highly available systems
• Strong analytical problem solving and communication skills