Staff Big Data Engineer

QLYS_IN Qualys Security TechServices Private Ltd.; Pune District; On-site; INR 4,000,000 - 6,000,000; Full time

Job summary

Qualys, Inc. in Pune, India invites a Staff Big Data Engineer to define and drive the evolution of the data platform powering the Enterprise TruRisk Platform.

You will build and scale distributed data systems handling billions of events daily, providing technical leadership across engineering teams. You will lead architectural decisions, design event-driven and streaming architectures, and mentor engineers to raise the bar on performance, reliability, and cost efficiency.

Qualifications

• Minimum 12 years of software/data engineering experience.
• Minimum 6 years with Apache Spark-based data processing solutions.
• Minimum 6 years building large-scale data pipelines processing billions of events daily.
• Minimum 4 years administering Apache Kafka in production.
• Minimum 4 years designing event-driven or streaming architectures.
• Strong expertise in distributed data processing, data lakes, and large-scale platform design.
• Experience delivering highly available, scalable distributed systems in production.
• Bachelor's degree in Computer Science or related technical discipline.

Responsibilities

• Define and drive the technical vision, architecture, and long-term strategy for the data platform and data pipeline ecosystem.
• Design, build, and maintain scalable, high-availability data processing systems using Spark, Kafka, and distributed data technologies.
• Lead architecture decisions across multiple teams to meet scalability, reliability, performance, security, and cost targets.
• Design event-driven, streaming, and large-scale architectures to support real-time and batch workloads.
• Troubleshoot and optimize performance bottlenecks across data pipelines and platform infrastructure.
• Collaborate with Product Management, Professional Services, and Sales Engineering to evaluate solutions.
• Establish engineering standards, architectural guidelines, and operational best practices.
• Research and recommend emerging technologies to improve scalability and efficiency.
• Mentor engineers on distributed systems design and big data technologies.

Skills

Distributed systems

Big data processing

Data pipelines

Leadership

Architectural design

Education

Bachelor's degree in Computer Science / related

Tools

Apache Spark

Apache Kafka

Scala/Java/Python

Job description

Come work at a place where innovation and teamwork come together to support the most exciting missions in the world! Staff Big Data Engineer Job Description We are seeking a Staff Big Data Engineer to define, design, and drive the evolution of the data platform and pipeline architecture powering Qualys' Enterprise TruRisk Platform. This role will be responsible for building and scaling distributed data systems that process billions of events and transactions daily while providing technical leadership across multiple engineering teams.

Responsibilities

• Define and drive the technical vision, architecture, and long-term strategy for the data platform and data pipeline ecosystem.
• Design, build, and maintain highly scalable, high-availability data processing systems using Apache Spark, Apache Kafka, and distributed data technologies.
• Lead architecture and design decisions across multiple engineering teams, ensuring solutions meet requirements for scalability, reliability, performance, security, and cost efficiency.
• Design and implement event-driven, streaming, and large-scale data processing architectures to support real-time and batch workloads.
• Troubleshoot, optimize, and resolve complex performance bottlenecks across data pipelines, distributed systems, and platform infrastructure.
• Partner with Product Management, Professional Services, and Sales Engineering teams to evaluate technical solutions, architecture trade-offs, and platform capabilities.
• Establish engineering standards, architectural guidelines, documentation practices, and operational best practices for the data platform.
• Research, evaluate, and recommend emerging technologies that improve platform scalability, performance, and operational efficiency.
• Mentor and guide engineers on distributed systems design, performance optimization, and big data technologies.
• Support architecture reviews, technical governance initiatives, and engineering excellence programs.

Required Qualifications

• Minimum 12 years of experience in software engineering, data engineering, distributed systems engineering, or big data platform development.
• Minimum 6 years of hands-on experience designing, developing, and optimizing Apache Spark-based data processing solutions.
• Minimum 6 years of experience building and operating large-scale data pipelines processing billions of events or transactions per day.
• Minimum 4 years of experience administering and operating Apache Kafka in production environments.
• Minimum 4 years of experience designing and implementing event-driven or streaming data architectures.
• Strong expertise in distributed data processing systems, data lake architectures, and large-scale platform design.
• Experience delivering highly available, scalable, and fault-tolerant distributed systems in production environments.
• Demonstrated experience leading architecture initiatives and influencing technical direction across multiple engineering teams.
• Bachelor's degree in Computer Science, Computer Engineering, Information Technology, or a related technical discipline.

Preferred Qualifications

• Experience working with high-volume, multi-tenant enterprise platform environments.
• Proficiency in Scala, Java, or Python for data processing applications.
• Proven experience mentoring junior or mid-level engineering team members.

Join our talent community and receive the latest Qualys news, content, and be first in line for new job opportunities. Join our Talent Community! Qualys, Inc. (NASDAQ: QLYS) is a pioneer and leading provider of disruptive cloud-based security, compliance and IT solutions with more than 10,000 subscription customers worldwide, including a majority of the Forbes Global 100 and Fortune 100. Qualys helps organizations streamline and automate their security and compliance solutions onto a single platform for greater agility, better business outcomes, and substantial cost savings.