Big Data Developer

Datametica; Pune District; On-site; INR 1,500,000 - 2,500,000; Full time

Job summary

Datametica Pune is seeking an experienced Big Data Developer to design, build, and optimize data processing solutions. The role emphasizes Spark, PySpark, and the Hadoop ecosystem, with both batch and real-time pipelines in a hybrid work setup.

You will collaborate with data engineers and business stakeholders, write complex SQL, model data for performance, and ensure quality, security, and governance across large-scale systems.

Qualifications

• 4-6 years of experience in Big Data development.
• Strong hands-on experience with Apache Spark and Hadoop ecosystem.
• Proficiency in Python and/or Scala.
• Strong expertise in SQL.
• Experience with Hive, Impala, Sqoop.
• Familiarity with Shell Scripting.
• Good understanding of data modeling concepts.
• Experience in performance tuning and optimization of big data applications.

Responsibilities

• Design, develop, and maintain scalable data processing systems using Spark and Hadoop ecosystem tools.
• Develop and optimize batch and real-time data pipelines using Spark / PySpark.
• Work extensively with Hive, Impala, and Sqoop for data ingestion and querying.
• Write efficient and complex SQL queries for data extraction, transformation, and analysis.
• Perform data modeling and ensure optimal performance tuning of large-scale data systems.
• Collaborate with cross-functional teams including data engineers, analysts, and business stakeholders.
• Monitor and troubleshoot data pipelines to ensure data quality and reliability.
• Implement best practices for big data processing, security, and governance.

Skills

Big Data development

Python/Scala

SQL

Shell scripting

Tools

Apache Spark

Hadoop ecosystem

Hive

Impala

Sqoop

Job description

Experience- 4-6 Years

Work Mode- Hybrid

Location- Pune

Key Responsibilities

• Design, develop, and maintain scalable data processing systems using Spark and Hadoop ecosystem tools
• Develop and optimize batch and real-time data pipelines using Spark / PySpark
• Work extensively with Hive, Impala, and Sqoop for data ingestion and querying
• Write efficient and complex SQL queries for data extraction, transformation, and analysis
• Perform data modeling and ensure optimal performance tuning of large-scale data systems
• Collaborate with cross-functional teams including data engineers, analysts, and business stakeholders
• Monitor and troubleshoot data pipelines to ensure data quality and reliability
• Implement best practices for big data processing, security, and governance.

Required Skills & Qualifications

• 4-6 years of experience in Big Data development
• Strong hands‑on experience with Apache Spark and Hadoop ecosystem
• Proficiency in Python and/or Scala
• Strong expertise in SQL
• Experience with Hive, Impala, Sqoop
• Familiarity with Shell Scripting
• Good understanding of data modeling concepts
• Experience in performance tuning and optimization of big data applications.