Azure Data Engineer

Ness Digital Engineering; Bengaluru; Hybrid; INR 1,500,000 - 2,300,000; Full time

Job summary

Ness Digital Engineering in Bengaluru, India is seeking a hands-on Big Data Engineer to design, implement, and optimize large-scale data platforms. You will work with Spark, Hadoop, Cloudera, Hive, and HDFS to build reliable data pipelines.

You will code in Python and/or Scala, collaborate with architects and analysts in Agile teams, and contribute to production operations, monitoring, and documentation. Knowledge of cloud data solutions and DevOps practices is a plus.

Qualifications

• Experience in big data development and operations.
• Strong programming expertise in Python and/or Scala.
• Hands‑on with Apache Spark, Hadoop, Cloudera, Hive, and HDFS.
• Proficiency with SQL, NoSQL, Oracle, and scheduling tools (e.g., Autosys).
• Experience in performance tuning, troubleshooting, and distributed systems.
• Familiarity with Git, Jenkins, Ansible, and agile delivery models.

Responsibilities

• Design and develop big data solutions using Apache Spark, Hadoop, and cloud‑native tools.
• Write efficient batch and stream processing pipelines in Python and/or Scala.
• Maintain and optimize data workflows and performance across large data volumes.
• Manage source control, versioning, and deployments using Git, Jenkins, and other DevOps tools.
• Conduct performance tuning, system monitoring, and issue resolution.
• Collaborate with architects, analysts, and business stakeholders to align technical delivery with use cases.
• Work with Cloudera Hadoop and manage activities like node configuration, file system maintenance, and backups.
• Follow best practices across software development, code quality, and agile ceremonies.
• Contribute to code reviews, technical documentation, and mentoring junior developers.
• Support and enhance production environments, including disaster recovery planning.

Skills

Python

Scala

Apache Spark

Hadoop

Cloudera

Hive

HDFS

SQL

NoSQL

Oracle

Git

Jenkins

Ansible

Agile

Tools

Autosys

Git

Jenkins

Job description

Life at Ness

At Ness, people come first. Here, you'll be part of a vibrant team that values curiosity, innovation, and growth. We work with industry-leading clients on projects that truly make an impact while supporting every team member in carving out their unique career path. With resources for learning, certifications, and hands‑on experiences, Ness offers you more than just a jobits a place where your ideas, ambition, and well‑being matter.

The Role

We are looking for a hands‑on Big Data Engineer with deep experience in developing and managing large‑scale data platforms. The ideal candidate will bring strong Spark, Python/Scala development skills, and solid understanding of the Hadoop ecosystem. Youll work in Agile teams on financial, analytical, and innovation‑driven data productsdriving performance optimization, new development, and platform operations.

Key Responsibilities

• Design and develop big data solutions using Apache Spark, Hadoop, and cloud‑native tools
• Write efficient batch and stream processing pipelines in Python and/or Scala
• Maintain and optimize data workflows and performance across large data volumes
• Manage source control, versioning, and deployments using Git, Jenkins, and other DevOps tools
• Conduct performance tuning, system monitoring, and issue resolution
• Collaborate with architects, analysts, and business stakeholders to align technical delivery with use cases
• Work with Cloudera Hadoop and manage activities like node configuration, file system maintenance, and backups
• Follow best practices across software development, code quality, and agile ceremonies
• Contribute to code reviews, technical documentation, and mentoring junior developers
• Support and enhance production environments, including disaster recovery planning

Qualifications and Skills

Core Skills

• 610 years of experience in big data development and operations
• Strong programming expertise in Python and/or Scala
• Hands‑on with Apache Spark, Hadoop, Cloudera, Hive, and HDFS
• Proficiency with SQL, NoSQL, Oracle, and scheduling tools (e.g., Autosys)
• Experience in performance tuning, troubleshooting, and distributed systems
• Familiarity with Git, Jenkins, Ansible, and agile delivery models

Nice to Have

• Java development background
• Financial domain experience
• Knowledge of cloud data solutions (AWS, Azure, GCP)
• Familiarity with DevOps and CI/CD practices
• Cloudera Hadoop Admin certification (for hybrid roles involving infra)
• Hands‑on development in fast‑paced environments
• Led small teams or initiatives within distributed systems projects
• Supported large‑scale production platforms
• Strong communicator who can work across business and engineering teams

Why Join Us

At Ness, you will work with diverse, talented professionals who are dedicated to making an impact through technology. We encourage applicants of all backgrounds to applyeven if you dont meet every requirement, wed love to connect with you if this role excites you. Were committed to creating an inclusive workplace that celebrates each team members unique talents. With flexible remote options, diverse projects, and access to development resources, joining Ness means building a career that’s meaningful and impactful.

What to Expect Next

We believe great experiences start with transparency—and that includes our hiring process. Here’s what you can typically expect after you hit "Apply":

1. HR Interview – A conversation to get to know you better and align on expectations

2. Technical Interview – A chance to showcase your skills and experience

3. Client Interview – Ensuring mutual fit between you and the client team

4. Offer Stage – If everything aligns, we’ll be happy to make it official

While many of our roles involve just 2–3 steps, some client processes may include additional conversations. Regardless of the path, we aim to keep every step transparent and respectful of your time.

Ready to Start Your Journey?