Lead Data Engineer - AWS/Azure Databricks

NetConnectGlobal; Bengaluru; Hybrid; INR 2,500,000 - 4,500,000; Full time

Benefits offered by this job

Hybrid work model

Job summary

NetConnectGlobal is seeking a hands-on Lead Data Engineer to architect scalable data platforms in a hybrid Bengaluru setting. You will guide a technical team, design data lakes/warehouses, and create robust ETL/ELT pipelines using Databricks, Snowflake, and multi-cloud services.

You will work with Python/SQL, Airflow, Airbyte, Docker, and Kubernetes, and contribute to CI/CD and MLOps initiatives. This is a 12-month contract with a strong emphasis on engineering excellence.

Qualifications

• 5+ years of experience in Data Engineering with strong technical leadership capabilities.
• Hands-on experience with Databricks, Snowflake, AWS and/or Azure.
• Experience building large-scale Data Lakes and Data Warehouses.
• Strong understanding of ETL/ELT, CDC, batch and real-time pipelines.
• Proficiency in SQL and Python.
• Experience with Apache Airflow and Airbyte.
• Experience developing REST API integrations and Git-based CI/CD (preferably GitLab).
• Containerization with Docker and Kubernetes.
• Knowledge of Data Architecture and Data Modeling.
• Familiarity with MLOps concepts and AI-assisted development.

Responsibilities

• Lead the design, development, and implementation of scalable data engineering solutions.
• Architect and build high-volume data lakes and data warehouses using cloud tech.
• Design and optimize ETL/ELT pipelines for batch and real-time workloads.
• Implement CDC pipelines and data integration solutions.
• Collaborate with teams to translate requirements into scalable data solutions.
• Provide technical leadership, code reviews, and mentorship to data engineers.
• Establish and maintain CI/CD pipelines and best practices.
• Support data infrastructure for ML workloads and related AI tooling.

Tools

Databricks

Snowflake

AWS

Azure

Python

SQL

Apache Airflow

Airbyte

Docker

Kubernetes

GitLab CI/CD

Job description

Role Overview

We are looking for a hands-on Lead Data Engineer with strong technical leadership and expertise in designing and building high-volume, production-grade data platforms. The ideal candidate will have extensive experience with modern cloud data technologies, data lakes, data warehouses, ETL/ELT pipelines, real-time data processing, and data architecture. The role requires someone who can combine strong engineering expertise with technical leadership to architect scalable data solutions and guide a technical team.

Lead Data Engineer

Experience : 5+ Years

Duration : 12 Months

Work Mode : Hybrid

Key Responsibilities

• Lead the design, development, and implementation of scalable data engineering solutions.


• Architect and build high-volume data lakes and data warehouses using modern cloud technologies.


• Design and optimize ETL/ELT pipelines supporting batch and real-time/streaming workloads.


• Implement Change Data Capture (CDC) pipelines and data integration solutions.


• Work with Databricks, Snowflake, AWS, and Azure to develop enterprise-grade data platforms.


• Develop robust data pipelines and integrations using Python and advanced SQL.


• Build and manage workflows using Apache Airflow and Airbyte.


• Design and implement integrations with REST APIs and third-party APIs.


• Establish and maintain Git-based CI/CD pipelines, preferably using GitLab.


• Containerize applications and data workloads using Docker and Kubernetes.


• Define and implement effective data architecture and data modeling strategies.


• Collaborate with engineering, analytics, data science, and business teams to translate requirements into scalable data solutions.


• Provide technical leadership, conduct code reviews, and mentor data engineers.


• Apply data engineering best practices for performance, scalability, security, reliability, and maintainability.


• Contribute to MLOps initiatives and support data infrastructure for machine learning workloads.


• Leverage AI coding assistants to improve engineering productivity, code quality, and development efficiency.

Must-Have Skills & Experience

• 5+ years of experience in Data Engineering with strong technical leadership capabilities.


• Strong expertise in Databricks, Snowflake, AWS, and/or Azure.


• Hands-on experience building large-scale Data Lakes and Data Warehouses.


• Strong understanding of ETL/ELT, CDC, batch processing, and real-time/streaming pipelines.


• Advanced proficiency in SQL and Python.


• Hands-on experience with Apache Airflow and Airbyte.


• Experience developing REST API and third-party API integrations.


• Strong understanding of Git-based CI/CD, preferably GitLab.


• Experience with Docker and Kubernetes.


• Strong knowledge of Data Architecture and Data Modeling.


• Understanding of MLOps concepts and practices.


• Experience using AI Coding Assistants to enhance software/data engineering productivity.


• Strong problem-solving, communication, and technical leadership skills.

Good To Have

• Experience working in retail or other large-scale data environments.


• Hands-on experience with Kafka and Spark Streaming.


• Knowledge of Customer Data Platforms (CDP).


• Experience with Agentic AI or LLM-based data pipelines.


• Exposure to Generative AI, RAG, and Vector Databases.


• Experience with observability and monitoring tools such as Splunk, Datadog, or Dynatrace.

Ideal Candidate

The ideal candidate is a hands-on Lead Data Engineer who can architect and build high-volume, production-grade data platforms while providing strong technical leadership. The candidate should be comfortable working across cloud, data engineering, DevOps, and emerging AI technologies and should be able to mentor engineers while driving technical excellence.

(ref:hirist.tech)