Senior Big Data Engineer

V2 Solutions; Pune District; On-site; INR 3,500,000 - 6,000,000; Full time

Job summary

Qualys is seeking a Sr. Big Data Engineer to deliver roadmap features for the Enterprise TruRisk Platform, enabling customers to measure and eliminate cyber risks. You will prototype, design, and support a scalable SaaS data platform using big data and micro-services, processing billions of transactions daily.

You will lead data pipelines, provide architectural guidance, mentor teams, and collaborate across product and services to ensure robust, secure solutions.

Qualifications

• Bachelor's degree in computer science or equivalent with 5+ years of total experience.
• Proficiency in Java/Scala, Spark, Kafka, Hadoop, and NoSQL databases.
• Experience building scalable data platforms and pipelines in enterprise settings.
• Hands-on with RDBMS/NoSQL, data lakes, and big data ecosystem tools.
• Strong understanding of performance, scalability, and architecture.

Responsibilities

• Lead data platform design and roadmaps for a scalable SaaS product.
• Provide technical leadership on data pipelines and evaluation.
• Collaborate with product, services, and sales on solution trade-offs.
• Architect and optimize end-to-end data processing and storage.
• Mentor teams, drive best practices, and ensure system reliability.

Skills

Java/Scala

Apache Spark

Kafka

Hadoop

RDBMS/NoSQL

Data lakes

Presto

Airflow

Scala

Performance tuning

Education

Bachelor's degree in Computer Science or equivalent

Tools

Spark

Cassandra

Oracle

Redis

Flink

Job description

Come work at a place where innovation and teamwork come together to support the most exciting missions in the world! Job Description We are seeking a talented Sr. Big Data Engineer to deliver roadmap features of Enterprise TruRisk Platform which would help customers to Measure, Communicate and Eliminate Cyber Risks. Working with a team of engineers and architects, you will be responsible for prototyping, designing, developing and supporting a highly scalable, distributed SaaS based Security Risk Prioritization product. This is a fantastic opportunity to be an integral part of a team building Qualys next generation platform using Big Data & Micro-Services based technology to process over billions of transactions data per day, leverage open-source technologies, and work on challenging and business-impacting initiatives.

Key Responsibilities

• Be the thought leader in data platform and pipeline along with Risk Evaluation.
• Provide technical leadership to the engineering organization on data platform design, roll out and evolution.
• Liason to product teams, professional services and sales engineers on solution and trade-off reviews and represent engineering in such conversations.
• Drive technology explorations and roadmaps.
• Serve as a technical lead on our most demanding, cross-functional departments.
• Ensure the quality of architecture and design of systems.
• Functionally decompose complex problems into simple, straight-forward solutions.
• Fully and completely understand system interdependencies and limitations.
• Possess expert knowledge in performance, scalability, enterprise system architecture, and engineering best practices.
• Leverage knowledge of internal and industry prior art in design decisions.
• Effectively research and benchmark cloud technology against other competing systems in the industry.
• Able to document the details so it will be easy for developers to understand the requirements.
• Assisting developers with proper requirements and directions.
• Assist in the career development of others, actively mentoring individuals and the community on advanced technical issues and helping managers guide the career growth of their team members.
• Exert technical influence over multiple teams, increasing their productivity and effectiveness by sharing your deep knowledge and experience.
• Able to share knowledge and train others.

Requirements

• Bachelor's degree in computer science or equivalent 5+ years of total experience.
• Required Skills: Java/Scala, Apache Spark, Kafka, Hadoop, RDBMS (Oracle or equivalent), and NoSQL databases (Cassandra or equivalent).
• Experience building scalable, enterprise-grade big data solutions using either Java or Scala.
• 2+ years of relevant experience in design and architecture Big Data solutions using Spark
• 3+ years experience in working with engineering resources for innovation.
• 4+ years experience in understanding Big Data events flow pipeline.
• 3+ years experience in performance testing for large infrastructure.
• 3+ In depth experience in understanding various search solutions solr/elastic.
• 3+ years experience in Kafka
• In depth experience in Data lakes and related ecosystems.
• In depth experience of messing queue
• In depth experience in giving requirements to build a scalable architecture for Big data and Micro-services environments.
• In depth experience in understanding caching components or services
• Knowledge in Presto technology.
• Knowledge in Airflow.
• Hands-on experience in scripting and automation
• In depth understanding of RDBMS/NoSQL, Oracle , Cassandra , Kafka , Redis, Hadoop, lambda architecture, kappa , kappa ++ architectures with flink data streaming and rule engines
• Experience in working with ML models engineering and related deployment.
• Design and implement secure big data clusters to meet many compliances and regulatory requirements.
• Experience in leading the delivery of large-scale systems focused on managing the infrastructure layer of the technology stack.
• Strong experience in doing performance benchmarking testing for Big data technologies.
• Strong troubleshooting skills.
• Experience leading development life cycle process and best practices
• Experience in Big Data services administration would be added value.
• Experience with Agile Management (SCRUM, RUP, XP), OO Modeling, working on internet, UNIX, Middleware, and database related projects.
• Experience mentoring/training the engineering community on complex technical issue.