Bigdata team member - L3

Ness Digital Engineering; Hyderabad; Hybrid; INR 2,800,000 - 5,000,000; Full time

Job summary

Ness is seeking a hands-on Big Data Engineer to design and manage large-scale data platforms. You will work in Agile teams on financial, analytical, and innovation-driven data products, driving performance optimization and platform operations.

The role requires strong Spark, Python/Scala development skills, and a solid Hadoop ecosystem background. You will collaborate with architects, analysts, and stakeholders while supporting production environments and DR planning.

Qualifications

• 6–10 years of experience in big data development and operations.
• Strong programming expertise in Python and/or Scala.
• Hands-on with Apache Spark, Hadoop, Cloudera, Hive, and HDFS.
• Proficiency with SQL, NoSQL, Oracle, and scheduling tools (e.g., Autosys).
• Experience in performance tuning, troubleshooting, and distributed systems.
• Familiarity with Git, Jenkins, Ansible, and agile delivery models.

Responsibilities

• Design and develop big data solutions using Apache Spark, Hadoop, and cloud-native tools.
• Write efficient batch and stream processing pipelines in Python and/or Scala.
• Maintain and optimize data workflows and performance across large data volumes.
• Manage source control, versioning, and deployments using Git, Jenkins, and other DevOps tools.
• Conduct performance tuning, system monitoring, and issue resolution.
• Collaborate with architects, analysts, and business stakeholders to align technical delivery with use cases.
• Work with Cloudera Hadoop and manage activities like node configuration, file system maintenance, and backups.
• Follow best practices across software development, code quality, and agile ceremonies.
• Contribute to code reviews, technical documentation, and mentoring junior developers.
• Support and enhance production environments, including disaster recovery planning.

Skills

Big data development

Python/Scala

Spark/Hadoop

SQL/NoSQL

Git/Jenkins/Ansible

Performance tuning

Tools

Autosys

Git

Jenkins

Ansible

Job description

Life at Ness

At Ness, people come first. Here, you'll be part of a vibrant team that values curiosity, innovation, and growth. We work with industry-leading clients on projects that truly make an impact while supporting every team member in carving out their unique career path. With resources for learning, certifications, and hands‑on experiences, Ness offers you more than just a job—it’s a place where your ideas, ambition, and well‑being matter.

The Role

We are looking for a hands‑on Big Data Engineer with deep experience in developing and managing large‑scale data platforms. The ideal candidate will bring strong Spark, Python/Scala development skills, and solid understanding of the Hadoop ecosystem. You’ll work in Agile teams on financial, analytical, and innovation-driven data products—driving performance optimization, new development, and platform operations.

Key Responsibilities

• Design and develop big data solutions using Apache Spark, Hadoop, and cloud-native tools
• Write efficient batch and stream processing pipelines in Python and/or Scala
• Maintain and optimize data workflows and performance across large data volumes
• Manage source control, versioning, and deployments using Git, Jenkins, and other DevOps tools
• Conduct performance tuning, system monitoring, and issue resolution
• Collaborate with architects, analysts, and business stakeholders to align technical delivery with use cases
• Work with Cloudera Hadoop and manage activities like node configuration, file system maintenance, and backups
• Follow best practices across software development, code quality, and agile ceremonies
• Contribute to code reviews, technical documentation, and mentoring junior developers
• Support and enhance production environments, including disaster recovery planning

Qualifications and Skills

Core Skills

• 6–10 years of experience in big data development and operations
• Strong programming expertise in Python and/or Scala
• Hands‑on with Apache Spark, Hadoop, Cloudera, Hive, and HDFS
• Proficiency with SQL, NoSQL, Oracle, and scheduling tools (e.g., Autosys)
• Experience in performance tuning, troubleshooting, and distributed systems
• Familiarity with Git, Jenkins, Ansible, and agile delivery models

Nice to Have

• Java development background
• Financial domain experience
• Knowledge of cloud data solutions (AWS, Azure, GCP)
• Familiarity with DevOps and CI/CD practices
• Cloudera Hadoop Admin certification (for hybrid roles involving infra)
• Hands‑on development in fast‑paced environments
• Led small teams or initiatives within distributed systems projects
• Supported large‑scale production platforms
• Strong communicator who can work across business and engineering teams

Why Join Us

At Ness, you will work with diverse, talented professionals who are dedicated to making an impact through technology. We encourage applicants of all backgrounds to apply—even if you don’t meet every requirement, we’d love to connect with you if this role excites you. We’re committed to creating an inclusive workplace that celebrates each team member’s unique talents. With flexible remote options, diverse projects, and access to development resources, joining Ness means building a career that’s meaningful and impactful.