Staff Big Data Engineer

Qualys, Inc.; Pune District; On-site; INR 4,200,000 - 7,000,000; Full time

Job summary

Qualys, Inc. in Pune is seeking a Staff Big Data Engineer to define and drive the data platform and pipeline architecture powering the Enterprise TruRisk Platform. You will design, scale, and optimize distributed data systems that process billions of events and transactions daily.

You will lead architecture decisions, mentor engineers, and collaborate with product, services, and sales teams to establish standards and best practices for high availability and performance.

Qualifications

• Minimum 12 years of experience in software engineering, data engineering, or distributed systems engineering.
• Minimum 6 years of hands-on experience designing, developing, and troubleshooting Apache Spark-based data processing solutions.
• Minimum 6 years of experience building and supporting large-scale data pipelines processing billions of events or transactions per day.
• Minimum 4 years of experience administering and operating Apache Kafka in production environments.
• Minimum 4 years of experience designing event-driven or streaming data architectures.
• Experience delivering highly available and scalable distributed systems in production environments.
• Experience leading architecture and technical initiatives across multiple engineering teams.
• Experience mentoring engineers and providing technical guidance on large-scale platform development.

Responsibilities

• Define the technical vision and long-term strategy for the data platform and pipeline architecture.
• Design and maintain scalable, high-availability data processing systems supporting billions of daily events and transactions.
• Lead architecture decisions across multiple engineering teams, ensuring alignment with business objectives including scalability, performance, reliability, cost, and time-to-market.
• Identify, troubleshoot, and resolve data platform performance bottlenecks and scalability challenges.
• Design and implement event-driven and streaming data architectures using distributed processing technologies.
• Partner with Product Management, Professional Services, and Sales Engineering teams to evaluate technical solutions and trade-offs; establish engineering standards and best practices.
• Research, evaluate, and recommend technologies for large-scale data processing and analytics platforms.
• Mentor engineers on distributed systems design, performance optimization, and big data technologies.
• Support technical reviews, architecture governance, and engineering excellence initiatives.

Skills

Apache Spark

Apache Kafka

Hadoop ecosystem technologies

Data lake architectures

Event-driven architectures

Distributed data processing systems

Oracle Database

Cassandra

Redis

Performance tuning

Linux/Unix environments

Large-scale infrastructure

Education

Bachelor's degree in Computer Science, Software Engineering, Information Technology, or a related technical discipline

Tools

Elasticsearch

Apache Solr

Trino

Apache Airflow

Distributed caching technologies

Lambda architecture

Kappa architecture

Kappa++ architecture

Apache Flink

Job description

## Staff Big Data EngineerApplylocations: Punetime type: Full timeposted on: Posted Todayjob requisition id: R0004882Come work at a place where innovation and teamwork come together to support the most exciting missions in the world!* **Job Summary** Qualys is seeking a Staff Big Data Engineer to define and drive the technical vision for the data platform and pipeline architecture powering the Enterprise TruRisk Platform. This role focuses on designing, scaling, and optimizing distributed data systems that process billions of events and transactions daily. The position requires hands-on technical leadership, architecture ownership, and direct involvement in solving large-scale performance and reliability challenges. **Responsibilities** + Define the technical vision and long-term strategy for the data platform and pipeline architecture. + Design and maintain scalable, high-availability data processing systems supporting billions of daily events and transactions. + Lead architecture and design decisions across multiple engineering teams, ensuring alignment with business objectives including scalability, performance, reliability, cost, and time-to-market. + Identify, troubleshoot, and resolve data platform performance bottlenecks and scalability challenges. + Design and implement event-driven and streaming data architectures using distributed processing technologies. + Partner with Product Management, Professional Services, and Sales Engineering teams to evaluate technical solutions and trade-offs. Establish engineering standards, architectural guidelines, and platform best practices. + Research, evaluate, and recommend technologies for large-scale data processing and analytics platforms. + Mentor engineers on distributed systems design, performance optimization, and big data technologies. + Support technical reviews, architecture governance, and engineering excellence initiatives. + # **Preferred Qualifications** + Experience with Elasticsearch or Apache Solr. + Experience with Trino. + Experience with Apache Airflow. + Experience with distributed caching technologies. + Experience implementing Lambda, Kappa, or Kappa++ architectures. + Experience with Apache Flink and real-time stream processing. Experience with rule-engine platforms. + Experience deploying and supporting machine learning models in production. + Experience administering enterprise Big Data platforms and services. **Technical Skills** Apache Spark Apache Kafka Hadoop ecosystem technologies Data lake architectures Event-driven architectures Distributed data processing systems Oracle Database Cassandra Redis Performance tuning and benchmarking of large-scale systems Linux/Unix environments Large-scale infrastructure troubleshooting and optimization **Professionnal Experience** + Minimum 12 years of experience in software engineering, data engineering, or distributed systems engineering. + Minimum 6 years of hands-on experience designing, developing, and troubleshooting Apache Spark-based data processing solutions. + Minimum 6 years of experience building and supporting large-scale data pipelines processing billions of events or transactions per day. + Minimum 4 years of experience administering and operating Apache Kafka in production environments. + Minimum 4 years of experience designing event-driven or streaming data architectures. + Experience delivering highly available and scalable distributed systems in production environments. + Experience leading architecture and technical initiatives across multiple engineering teams. + Experience mentoring engineers and providing technical guidance on large-scale platform development. **Education** Bachelor's degree in Computer Science, Software Engineering, Information Technology, or a related technical discipline. **Preferred Qualifications** + Experience with Elasticsearch or Apache Solr. Experience with Trino. + Experience with Apache Airflow. + Experience with distributed caching technologies. + Experience implementing Lambda, Kappa, or Kappa++ architectures. + Experience with Apache Flink and real-time stream processing. Experience with rule-engine platforms. + Experience deploying and supporting machine learning models in production. + Experience administering enterprise Big Data platforms and services.