Technical Lead - Data Engineer (Data&AI)

Visa Hunt; India; On-site; INR 1,500,000 - 2,500,000; Full time

Job summary

Visa Hunt is seeking a Lead Data Engineer to guide architecture, pipelines, and CI/CD across Databricks, Snowflake, AWS, and Azure. You will lead a high-performing team delivering scalable, production-ready data solutions.

You will collaborate with data scientists on ML data readiness, implement governance, and stay ahead of MLOps and observability trends, driving improvements across the data stack. The role requires strong Python/SQL, Airflow, Airbyte, and DevOps experience to deliver reliable,

Qualifications

• 5+ years of experience in Data Engineering.
• Design, build, and manage large Data Lakes and Data Warehouses.
• Experience with CDC, batch and real-time data processing.
• Expertise across Databricks, Snowflake, AWS, and Azure.
• Strong Python programming and SQL skills.
• Experience with REST APIs and API frameworks.
• Hands-on with Airflow and Airbyte.
• Familiar with Git-based CI/CD, Docker, Kubernetes.
• Knowledge of MLOps concepts and model lifecycle.

Responsibilities

• Lead scalable ETL/ELT pipelines and data models for large retail data.
• Architect multi-platform data processing across Databricks, Snowflake, AWS, and Azure.
• Build data pipelines with Airflow and Airbyte for reliable data movement.
• Implement CDC, batch processing, and real-time streaming solutions.
• Develop REST APIs and internal APIs for downstream use.
• Leverage AI assistants to accelerate pipeline design and coding.
• Own CI/CD pipelines with Git workflows and automated testing.
• Collaborate with Data Scientists on ML data readiness.
• Mentor the team and uphold data governance and security.
• Stay updated on MLOps, observability, GenAI trends.

Skills

Python

SQL

Airflow

Airbyte

Databricks

Snowflake

AWS

Azure

CI/CD

Kubernetes

Docker

REST APIs

MLOps

Tools

Airflow

Airbyte

Databricks

Snowflake

AWS

Azure

Docker

Kubernetes

Git

Job description

Overview

We are looking for a Lead Data Engineer who combines hands-on multi-platform expertise with strong leadership in data architecture, pipelines, and CI/CD. This role requires a versatile engineer with deep technical skills across modern data platforms (such as Databricks, Snowflake, AWS, and Azure), an understanding of MLOps/DevOps practices, and the ability to guide a high-performing team in building scalable, production-ready data solutions. You will not be limited to a single platform but will leverage a diverse toolkit to solve complex data challenges.

Key Responsibilities

• Pipeline & Architecture: Lead hands-on development of scalable ETL/ELT pipelines, data models, and integration frameworks to process high-volume (billions of records) structured and unstructured retail data.
• Multi-Platform Engineering: Design, develop, and optimize data processing applications across multiple platforms, including Databricks (Spark/Delta Lake), Snowflake, AWS, or Azure.
• Data Integration & Orchestration: Build and manage robust data pipelines using Apache Airflow for orchestration and Airbyte for seamless data integration and movement.
• Data Processing: Architect and implement robust solutions for Change Data Capture (CDC), large-scale batch processing, and low-latency real-time/streaming data processing.
• API Management: Work extensively with external APIs for data ingestion, as well as design, create, and manage internal REST APIs to serve data to downstream applications and users.
• AI-Augmented Deliverables: Actively leverage AI assistants to conceptualize, design, and accelerate the development of data pipelines and everyday engineering tasks.
• DevOps & CI/CD: Own and evolve CI/CD pipelines (Git workflows, automated testing, release cycles, secrets management, documentation). Guide DevOps-oriented deployments utilizing Dockerized applications, Kubernetes orchestration, and monitoring/logging tools (Splunk, Datadog, Dynatrace).
• MLOps Alignment: Collaborate with Data Scientists on data readiness for ML projects and ensure alignment with ML lifecycle stages (data prep, feature engineering, model deployment).
• Governance & Leadership: Establish and enforce best practices in data governance, data quality, metadata, and security. Mentor team members through peer reviews, knowledge sharing, and technical leadership.
• Innovation: Stay ahead of industry trends in MLOps, observability, and GenAI, introducing relevant tools and practices.

Required Skills & Experience

• Experience: 5+ years of experience in Data Engineering.
• Data Lakes & Warehouses: Mandatory expertise in designing, building, and managing large-scale Data Warehouses and Data Lakes from the ground up.
• Data Processing Paradigms: Extensive, hands-on experience working with Change Data Capture (CDC) mechanisms, complex batch processing, and real-time/streaming data processing.
• Platform Expertise: Proven expertise in more than one major cloud data platform/ecosystem (e.g., Databricks, Snowflake, AWS Analytics, Azure Data Engineering).
• SQL Mastery: Advanced proficiency in writing, optimizing, and debugging complex SQL queries for large-scale data processing and analytics.
• Programming: Strong programming skills in Python (async, threading, decorators, advanced I/O).
• APIs: Strong proficiency in interacting with third-party APIs and hands-on experience creating and managing REST APIs (using frameworks like FastAPI, Flask, or similar).
• Tooling: Deep hands-on experience with workflow orchestration (Apache Airflow) and data integration platforms (Airbyte).
• AI-Assisted Engineering: Mandatory capability to use AI coding assistants and tools to design pipelines, write code, and enhance day-to-day productivity.
• Data Architecture: Experience with data modeling (e.g., Delta Lake or Snowflake architecture) and scalable ETL/ELT design.
• DevOps/CI/CD: Hands-on experience with Git-based CI/CD (GitLab preferred) and a working knowledge of Docker & Kubernetes for deployment and scaling.
• MLOps: Understanding of MLOps concepts including data preparation, model lifecycle, registries, and monitoring.
• Soft Skills: Strong problem-solving skills with the ability to design for scale and performance, coupled with excellent collaboration, communication, and leadership skills.

Good to Have

• Customer Data Platform (CDP): Experience working with, building, or implementing CDPs to unify customer data across systems.
• Experience in the retail domain or other large-scale data-heavy environments.
• Familiarity with streaming frameworks (Kafka, Spark Streaming, etc.).
• Agentic Pipeline Development: Experience or strong interest in building agentic pipelines using LLMs for dynamic data orchestration and automation.
• Knowledge of model observability tools and ML deployment pipelines.
• Exposure to GenAI concepts (vector embeddings, vector databases, RAG).