Lead Consultant(Product/Domain)

HCL Technologies Limited; Uttar Pradesh; Hybrid; INR 2,500,000 - 4,500,000; Full time

Job summary

HCLTech in India seeks a seasoned Data Engineer to architect end-to-end data products on the Medallion Architecture, built on Amazon S3 and Delta Lake. You will design scalable pipelines using PySpark, SQL, and Delta Live Tables for batch and real-time processing, ingest streams via Kinesis or MSK, and optimize storage and governance across AWS.

Strong testing, observability, and documentation are essential.

Qualifications

• 6+ years in Data Engineering with 4+ years on Databricks in AWS.
• Expert Python (PySpark) and advanced SQL skills.
• Deep Delta Lake and Delta Live Tables knowledge.
• Solid AWS experience (EC2, IAM, S3, Secrets Manager, VPC).
• CI/CD and IaC with Terraform; familiarity with GitHub Actions.

Responsibilities

• Architect end-to-end data products using Medallion Architecture on S3 and Delta Lake.
• Design and scale production pipelines with PySpark, SQL, and Delta Live Tables for batch and real-time data.
• Ingest diverse data streams by integrating Databricks with AWS messaging (Kinesis/MSK).
• Optimize lakehouse storage layouts with Delta features (Liquid Clustering, Z-Ordering).
• Ensure testing, monitoring, and observability; maintain thorough documentation.
• Implement data governance and security across AWS assets using Unity Catalog.
• Drive DevOps & CI/CD with IaC (Terraform) and automated deployments; mentor engineers.

Skills

Databricks

AWS

Python (PySpark)

SQL

Delta Lake

Delta Live Tables

Kinesis/MSK

Terraform

Apache Airflow

GitHub Actions

Unity Catalog

IAM

Tools

Databricks Workflows

Terraform

Apache Airflow

GitHub Actions

Job description

Key ResponsibilitiesDatabricks & AWS Data Pipeline EngineeringArchitect end-to-end data products using the Medallion Architecture (Bronze, Silver, Gold layers) natively built on Amazon S3 and Delta Lake.Design and scale production pipelines using PySpark, SQL, and Delta Live Tables (DLT) for both high-throughput batch and real-time streaming data.Ingest diverse data streams by integrating Databricks with AWS messaging systems like Amazon Kinesis or Managed Streaming for Apache Kafka (MSK).Optimize Lakehouse storage layouts by leveraging Delta features such as Liquid Clustering, Z-Ordering, and data compaction.Solid understanding of data testing methodologies, including unit, integration, and negative testing.Experience with model monitoring, alerting, and pipeline observability.Strong documentation and communication skills.AWS Infrastructure, Security & GovernanceImplement data governance and strict role-based access control across all AWS-hosted data assets using Unity Catalog.Manage infrastructure security by configuring security roles and personasOptimize cloud spend by managing Databricks clusters, selecting appropriate Amazon EC2 instance types (compute vs. memory-optimized), and tracking DBUs.Integrate security frameworks utilizing AWS Secrets Manager to securely handle pipeline credentials and API keys.DevOps & Engineering ExcellenceAzure DevOps via Git integration, automated unit testing, and custom CI/CD pipelines (e.g., GitHub Actions, Databricks DAB).Orchestrate data workflows seamlessly using Databricks Workflows or Apache AirflowDeploy Infrastructure as Code (IaC) using Terraform to provision and scale Databricks workspaces and AWS data resources.Mentor and guide mid-level engineers, establishing coding standards and running comprehensive peer code reviews.Required Skills & QualificationsTechnical ExpertiseExperience: 6+ years in Data Engineering, with 4+ years of deep focus on Databricks within an AWS environment.Languages: Expert-level Python (PySpark) and complex, analytical SQL.Storage & Format: Deep technical mastery of Delta Lake, Amazon S3, and advanced Spark optimization techniques.AWS Ecosystem: Strong hands-on proficiency with EC2, IAM, S3, Secrets Manager, and VPC networking fundamentals.[basic level]CI/CD & IaC: Proven experience with Terraform for data platform provisioning and automated deployment tools.Soft SkillsArchitectural Ownership:Ability to lead technical design discussions and choose cost-efficient cloud patterns.Technical Communication: Clear documentation skills to map complex data lineage and infrastructure dependencies.Preferred Certifications[optional]Databricks Certified Data Engineer Associate/Professional

Key Responsibilities

Key ResponsibilitiesDatabricks & AWS Data Pipeline EngineeringArchitect end-to-end data products using the Medallion Architecture (Bronze, Silver, Gold layers) natively built on Amazon S3 and Delta Lake.Design and scale production pipelines using PySpark, SQL, and Delta Live Tables (DLT) for both high-throughput batch and real-time streaming data.Ingest diverse data streams by integrating Databricks with AWS messaging systems like Amazon Kinesis or Managed Streaming for Apache Kafka (MSK).Optimize Lakehouse storage layouts by leveraging Delta features such as Liquid Clustering, Z-Ordering, and data compaction.Solid understanding of data testing methodologies, including unit, integration, and negative testing.Experience with model monitoring, alerting, and pipeline observability.Strong documentation and communication skills.AWS Infrastructure, Security & GovernanceImplement data governance and strict role-based access control across all AWS-hosted data assets using Unity Catalog.Manage infrastructure security by configuring security roles and personasOptimize cloud spend by managing Databricks clusters, selecting appropriate Amazon EC2 instance types (compute vs. memory-optimized), and tracking DBUs.Integrate security frameworks utilizing AWS Secrets Manager to securely handle pipeline credentials and API keys.DevOps & Engineering ExcellenceAzure DevOps via Git integration, automated unit testing, and custom CI/CD pipelines (e.g., GitHub Actions, Databricks DAB).Orchestrate data workflows seamlessly using Databricks Workflows or Apache AirflowDeploy Infrastructure as Code (IaC) using Terraform to provision and scale Databricks workspaces and AWS data resources.Mentor and guide mid-level engineers, establishing coding standards and running comprehensive peer code reviews.Required Skills & QualificationsTechnical ExpertiseExperience: 6+ years in Data Engineering, with 4+ years of deep focus on Databricks within an AWS environment.Languages: Expert-level Python (PySpark) and complex, analytical SQL.Storage & Format: Deep technical mastery of Delta Lake, Amazon S3, and advanced Spark optimization techniques.AWS Ecosystem: Strong hands-on proficiency with EC2, IAM, S3, Secrets Manager, and VPC networking fundamentals.[basic level]CI/CD & IaC: Proven experience with Terraform for data platform provisioning and automated deployment tools.Soft SkillsArchitectural Ownership:Ability to lead technical design discussions and choose cost-efficient cloud patterns.Technical Communication: Clear documentation skills to map complex data lineage and infrastructure dependencies.Preferred Certifications[optional]Databricks Certified Data Engineer Associate/Professional

Skill Requirements

Key ResponsibilitiesDatabricks & AWS Data Pipeline EngineeringArchitect end-to-end data products using the Medallion Architecture (Bronze, Silver, Gold layers) natively built on Amazon S3 and Delta Lake.Design and scale production pipelines using PySpark, SQL, and Delta Live Tables (DLT) for both high-throughput batch and real-time streaming data.Ingest diverse data streams by integrating Databricks with AWS messaging systems like Amazon Kinesis or Managed Streaming for Apache Kafka (MSK).Optimize Lakehouse storage layouts by leveraging Delta features such as Liquid Clustering, Z-Ordering, and data compaction.Solid understanding of data testing methodologies, including unit, integration, and negative testing.Experience with model monitoring, alerting, and pipeline observability.Strong documentation and communication skills.AWS Infrastructure, Security & GovernanceImplement data governance and strict role-based access control across all AWS-hosted data assets using Unity Catalog.Manage infrastructure security by configuring security roles and personasOptimize cloud spend by managing Databricks clusters, selecting appropriate Amazon EC2 instance types (compute vs. memory-optimized), and tracking DBUs.Integrate security frameworks utilizing AWS Secrets Manager to securely handle pipeline credentials and API keys.DevOps & Engineering ExcellenceAzure DevOps via Git integration, automated unit testing, and custom CI/CD pipelines (e.g., GitHub Actions, Databricks DAB).Orchestrate data workflows seamlessly using Databricks Workflows or Apache AirflowDeploy Infrastructure as Code (IaC) using Terraform to provision and scale Databricks workspaces and AWS data resources.Mentor and guide mid-level engineers, establishing coding standards and running comprehensive peer code reviews.Required Skills & QualificationsTechnical ExpertiseExperience: 6+ years in Data Engineering, with 4+ years of deep focus on Databricks within an AWS environment.Languages: Expert-level Python (PySpark) and complex, analytical SQL.Storage & Format: Deep technical mastery of Delta Lake, Amazon S3, and advanced Spark optimization techniques.AWS Ecosystem: Strong hands-on proficiency with EC2, IAM, S3, Secrets Manager, and VPC networking fundamentals.[basic level]CI/CD & IaC: Proven experience with Terraform for data platform provisioning and automated deployment tools.Soft SkillsArchitectural Ownership:Ability to lead technical design discussions and choose cost-efficient cloud patterns.Technical Communication: Clear documentation skills to map complex data lineage and infrastructure dependencies.Preferred Certifications[optional]Databricks Certified Data Engineer Associate/Professional

Other Requirements

Key ResponsibilitiesDatabricks & AWS Data Pipeline EngineeringArchitect end-to-end data products using the Medallion Architecture (Bronze, Silver, Gold layers) natively built on Amazon S3 and Delta Lake.Design and scale production pipelines using PySpark, SQL, and Delta Live Tables (DLT) for both high-throughput batch and real-time streaming data.Ingest diverse data streams by integrating Databricks with AWS messaging systems like Amazon Kinesis or Managed Streaming for Apache Kafka (MSK).Optimize Lakehouse storage layouts by leveraging Delta features such as Liquid Clustering, Z-Ordering, and data compaction.Solid understanding of data testing methodologies, including unit, integration, and negative testing.Experience with model monitoring, alerting, and pipeline observability.Strong documentation and communication skills.AWS Infrastructure, Security & GovernanceImplement data governance and strict role-based access control across all AWS-hosted data assets using Unity Catalog.Manage infrastructure security by configuring security roles and personasOptimize cloud spend by managing Databricks clusters, selecting appropriate Amazon EC2 instance types (compute vs. memory-optimized), and tracking DBUs.Integrate security frameworks utilizing AWS Secrets Manager to securely handle pipeline credentials and API keys.DevOps & Engineering ExcellenceAzure DevOps via Git integration, automated unit testing, and custom CI/CD pipelines (e.g., GitHub Actions, Databricks DAB).Orchestrate data workflows seamlessly using Databricks Workflows or Apache AirflowDeploy Infrastructure as Code (IaC) using Terraform to provision and scale Databricks workspaces and AWS data resources.Mentor and guide mid-level engineers, establishing coding standards and running comprehensive peer code reviews.Required Skills & QualificationsTechnical ExpertiseExperience: 6+ years in Data Engineering, with 4+ years of deep focus on Databricks within an AWS environment.Languages: Expert-level Python (PySpark) and complex, analytical SQL.Storage & Format: Deep technical mastery of Delta Lake, Amazon S3, and advanced Spark optimization techniques.AWS Ecosystem: Strong hands-on proficiency with EC2, IAM, S3, Secrets Manager, and VPC networking fundamentals.[basic level]CI/CD & IaC: Proven experience with Terraform for data platform provisioning and automated deployment tools.Soft SkillsArchitectural Ownership:Ability to lead technical design discussions and choose cost-efficient cloud patterns.Technical Communication: Clear documentation skills to map complex data lineage and infrastructure dependencies.Preferred Certifications[optional]Databricks Certified Data Engineer Associate/Professional

At HCLTech, you'll supercharge your potential. You'll find your career. And you'll find your spark. All at a place that knows that helping its customers stay on top starts by putting its people first.

HCLTech is a global technology company, home to more than 223,000 people across 60 countries, delivering industry-leading capabilities centered around digital, engineering, cloud and AI, powered by a broad portfolio of technology services and products. We work with clients across all major verticals, providing industry solutions for Financial Services, Manufacturing, Life Sciences and Healthcare, Technology and Services, Telecom and Media, Retail and CPG, and Public Services. Consolidated revenues as of 12 months ending June 2026totaled $14.8billion.