Lead Data & AI Engineer

Qualys; Pune District; On-site; INR 4,000,000 - 6,500,000; Full time

Job summary

Qualys is seeking an experienced Data Engineer/Architect to design and implement a complete Data Lake/Lakehouse from scratch, incorporating Bronze/Silver/Gold medallion architecture, governance, and scalable pipelines on a modern cloud platform.

You will integrate LLM APIs and build AI-powered applications atop the data platform, collaborating with CIO and enterprise teams to identify high-impact AI use cases and ensure secure, scalable delivery.

Qualifications

• 7–12+ years of hands-on data engineering experience with at least 2–3 years building enterprise AI solutions.
• Strong experience with Data Lakes/Lakehouse architectures, semantic layers, data pipelines, governance, and AI-ready data foundations.
• Deep expertise in Snowflake, Databricks, or equivalent platforms (Delta Lake, Apache Iceberg).
• Proficiency in Python, backend development, APIs, system integration, and workflow orchestration.
• Hands-on experience with GenAI or LLM-based applications in production contexts (RAG pipelines, LLM API integration, agentic workflows).
• Cloud experience on AWS, Azure, or GCP; scalable batch/streaming pipelines.

Responsibilities

• Architect and build enterprise data lake/lakehouse from scratch across ingestion, storage, transformation, serving, and governance layers on a modern cloud platform.
• Design and implement medallion architecture (Bronze/Silver/Gold) for multi-domain data.
• Build scalable batch and streaming data pipelines from enterprise apps and product sources.
• Define data quality frameworks, data contracts, and SLAs across ingestion and transformation layers.
• Implement data governance, lineage tracking, and metadata management for trustable assets.
• Define RBAC, row- and column-level security, and integrate identity across systems.
• Evaluate lake platform options and provide cost, governance, AI readiness, and ops recommendations.
• Build AI-powered apps on the data platform (NL interfaces, RAG search, automated reporting, workflows).
• Integrate LLM APIs into production data workflows and business apps.
• Assess AI use cases and enable data infrastructure to support them.
• Advise leadership on build-vs-buy for AI capabilities and tooling.
• Collaborate with business teams to deliver high-impact data/AI use cases.
• Develop reusable data frameworks, engineering standards, and mentor junior engineers.

Skills

Data lake architecture

Python

SQL

Cloud platforms

ETL pipelines

LLM integration

API integration

Data governance

Streaming data

RBAC / security

Education

BE/B.Tech/MCA

Tools

Snowflake

Databricks

Delta Lake

Apache Iceberg

Job description

Grow your career internally at Qualys, our best talent comes from within!

Design, build, and scale enterprise-grade Data & AI/GenAI solutions that leverage enterprise data platforms and business applications to automate processes, enhance decision-making and deliver measurable business outcomes. This role focuses on defining and implementing the foundational Data & AI strategy for the CIO organization by combining modern data platforms, AI technologies, enterprise systems, APIs, and analytics in a secure, scalable, and production-ready environment. The ideal candidate brings experience building Data Lake or Lakehouse architectures from scratch and implement AI solutions on top of it.

Responsibilities

• Architect and build enterprise data lake/lakehouse from scratch — including ingestion, storage, transformation, serving, and governance layers — on a modern cloud platform (Snowflake, Databricks, or equivalent).
• Design and implement a medallion (Bronze/Silver/Gold) architecture to serve structured, semi-structured, and unstructured data across multiple business domains.
• Build scalable, reliable batch and streaming data pipelines to ingest data from enterprise applications and product source systems.
• Define and enforce data quality frameworks, data contracts, and SLAs across all ingestion and transformation layers.
• Implement data governance, lineage tracking, and metadata management — ensuring every data asset is discoverable, trusted, and auditable.
• Design and manage access control across the lake — including role-based access control (RBAC), row-level and column-level security — harmonizing identity and entitlements across enterprise source systems.
• Own the Data Lake platform evaluation and produce a recommendation covering cost, governance, AI readiness, and operational overhead.
• Build AI-powered applications on top of the data platform — including natural language interfaces to enterprise data, RAG-based intelligent search, automated reporting, and workflow agents.
• Integrate LLM APIs (OpenAI, Anthropic Claude, AWS Bedrock, or equivalent) into production data workflows and business-facing applications.
• Assess and recommend AI use cases across business functions and build or enable the data infrastructure required to support them.
• Evaluate build-vs-buy decisions for AI capabilities and advise leadership on platform and tooling choices.
• Collaborate with business and enterprise application teams to identify and deliver high-impact data and AI use cases.
• Develop reusable data frameworks, engineering standards, and best practices; mentor junior data engineers.

Required Qualifications

• 7–12+ years of hands-on data engineering experience, with at least 2–3 years building enterprise AI solution.
• Strong experience on enterprise Data Architectures, building Data Lakes/Lakehouse, semantic layers, data pipelines, governance, and AI-ready data foundations.
• Deep expertise in at least one modern cloud data platform: Snowflake, Databricks, or equivalent (Delta Lake, Apache Iceberg).
• Strong expertise in Python, backend development, APIs, system integration, and workflow orchestration.
• Hands-on experience with GenAI or LLM-based applications in a production context — RAG pipelines, LLM API integration, agentic workflows, NL-to-SQL, or LLM evaluation frameworks.
• Hands-on cloud infrastructure experience on AWS, Azure, GCP
• Experience building batch and/or streaming pipelines at meaningful scale (multi-TB daily volumes or higher).
• Solid understanding of data modelling — dimensional modelling, medallion architecture, data vault — and the ability to choose the right pattern for the use case.
• Experience with data governance, access control (RBAC, row-level security, column masking), and lineage tracking in a multi-source enterprise environment.
• Strong SQL — query optimization, performance tuning, and large-scale analytical workloads.
• Ability to communicate technical strategy clearly to non-technical stakeholders and leadership; experience making the business case for data platform investments.

Preferred Qualifications

• Experience integrating enterprise applications as data sources into a Data Lake/Lakehouse.
• Experience integrating AI into enterprise applications and business workflows.
• Experience building reusable AI frameworks, accelerators, or enterprise AI platforms.
• Experience with vector databases, embeddings, semantic search, and advanced RAG architectures.
• Experience with real-time or near-real-time streaming technologies
• Familiarity with dbt for transformation layer engineering.
• Experience with data cataloguing and lineage tools.
• Prior experience in a product or SaaS company, cybersecurity, or high-growth technology environment.

Educational Qualification

BE/B.Tech/MCA, preferably in Computer Science, Information Technology, or a related field.