Technical Lead

Carelon Global Solutions; Gurugram District, Bengaluru; Hybrid; INR 3,000,000 - 5,500,000; Full time

Job summary

Carelon Global Solutions in Gurugram is seeking a senior Data Engineer/Technical Lead to drive scalable data engineering initiatives. You will architect and implement lakehouse solutions using Databricks, Python, and Spark, while guiding teams across batch and real-time processing projects.

The role requires strong Snowflake, Iceberg, and cloud platform expertise, plus leadership and stakeholder communication skills for large-scale data modernization programs. On-site in India.

Qualifications

• B.Tech/MCA with CS background or equivalent experience.
• 7 years total experience with minimum 3 years in Databricks.
• Hands-on Databricks design and implementation of scalable lakehouse solutions.
• strong Python and PySpark for building enterprise data pipelines.
• Snowflake data modeling, performance tuning, query optimization, and cost management.
• Experience with Apache Iceberg, Delta Lake, and open table formats.
• Strong knowledge of AWS/GCP cloud services and platform operations.
• Experience building batch and real-time data processing with Spark, Kafka, and cloud services.
• Proven ETL/ELT design, data integration, and cloud migration experience.
• Experience leading development teams and data modernization programs.
• Excellent troubleshooting and stakeholder communication skills.
• Expert Databricks knowledge: Lakehouse, Delta Lake, Unity Catalog, Workflows, optimization.

Responsibilities

• Lead development and implementation of scalable data engineering solutions.
• Drive end-to-end data pipeline development using Databricks and Python.
• Design batch and real-time data processing frameworks.
• Review code and establish engineering best practices.
• Guide teams on performance optimization and reliability improvements.
• Build and maintain ETL/ELT pipelines using Databricks, Spark, and Python.
• Develop scalable data lakehouse architectures.
• Implement data quality, governance, and observability frameworks.
• Optimize Snowflake workloads and data models.
• Design Iceberg-based data lakes for large-scale analytics.
• Work with AWS/GCP cloud services.
• Develop cloud-native data solutions and CI/CD awareness.
• Support platform modernization and cloud migration initiatives.
• Partner with architects, product owners, and business stakeholders.
• Translate business requirements into technical solutions.
• Provide effort estimation and delivery planning.

Skills

Databricks

Python

PySpark

Snowflake

Delta Lake

Iceberg

Open table formats

AWS

GCP

Spark

Kafka

ETL/ELT

Data governance

Data modeling

Unity Catalog

Workflows

SQL

Education

B.Tech degree/MCA with computer science background or equivalent experience.

Tools

CI/CD

Terraform

CloudFormation

Job description

QUALIFICATION

• B.Tech degree/MCA with computer science background or equivalent experience.

EXPERIENCE

• 7 years of total and minimum 3 years of relevant experience in Databricks.
• Proven hands-on experience with Databricks , including designing and implementing scalable data lakehouse solutions.

SKILLS

• Strong expertise in Python and PySpark for developing enterprise-grade data pipelines and distributed processing frameworks.
• Extensive experience with Snowflake, including data modeling, performance tuning, query optimization, and cost management.
• Hands-on experience with Apache Iceberg, Delta Lake, and modern open table formats for large-scale data lake implementations.
• Strong understanding of AWS/GCP cloud services, architecture design, deployment, and platform operations.
• Experience in building batch and real-time data processing pipelines using Spark, Kafka, and cloud-native services.
• Proven expertise in ETL/ELT design, data integration, and cloud migration initiatives.
• Experience leading development teams and delivering data modernization programs.
• Strong troubleshooting and analytical skills with the ability to resolve complex production issues.
• Excellent stakeholder management and communication skills, working across technical and business teams.
• Expert knowledge of Databricks, including Lakehouse Architecture, Delta Lake, Unity Catalog, Workflows, and Performance Optimization.
• Strong proficiency in Python, PySpark, Spark SQL, and distributed data processing.
• Extensive hands-on experience with Snowflake, including Snowpipe, Streams, Tasks, Data Sharing, and Query Optimization.
• Strong expertise in Apache Iceberg, open table formats, schema evolution, partitioning, and time-travel capabilities.
• Experience designing and implementing large-scale Data Engineering and ETL/ELT Pipelines.
• Hands-on experience with AWS/Azure/GCP cloud-native data services.
• Strong SQL and data modeling skills across OLTP and analytical environments.
• Experience with Real-Time Data Processing using Kafka, Event Hubs, Kinesis, or Pub/Sub.
• Knowledge of Data Governance, Metadata Management, Data Quality, and Security frameworks.

Good to Have

• Experience with Terraform, CloudFormation, or Infrastructure as Code (IaC).
• Knowledge of Docker, Kubernetes, and containerized data platforms.
• Experience with CI/CD tools such as GitHub Actions, Jenkins, Azure DevOps, or GitLab.
• Exposure to AI/ML, MLOps, and GenAI data platform integration.
• Knowledge of Data Mesh, Data Fabric, and modern enterprise data architectures.
• Databricks, Snowflake, and Cloud platform certifications.
• Expert knowledge of Databricks, including Lakehouse Architecture, Delta Lake, Unity Catalog, Workflows, and Performance Optimization.
• Strong proficiency in Python, PySpark, Spark SQL, and distributed data processing.
• Extensive hands-on experience with Snowflake, including Snowpipe, Streams, Tasks, Data Sharing, and Query Optimization.
• Strong expertise in Apache Iceberg, open table formats, schema evolution, partitioning, and time-travel capabilities.
• Experience designing and implementing large-scale Data Engineering and ETL/ELT Pipelines.
• Hands-on experience with AWS/Azure/GCP cloud-native data services.
• Strong SQL and data modeling skills across OLTP and analytical environments.
• Experience with Real-Time Data Processing using Kafka, Event Hubs, Kinesis, or Pub/Sub.
• Knowledge of Data Governance, Metadata Management, Data Quality, and Security frameworks.

Key Responsibilities

Technical Leadership

• Lead development and implementation of scalable data engineering solutions.
• Drive end-to-end data pipeline development using Databricks and Python.
• Design batch and real-time data processing frameworks.
• Review code and establish engineering best practices.
• Guide teams on performance optimization and reliability improvements.

Data Engineering & Analytics

• Build and maintain ETL/ELT pipelines using Databricks, Spark, and Python.
• Develop scalable data lakehouse architectures.
• Implement data quality, governance, and observability frameworks.
• Optimize Snowflake workloads and data models.
• Design Iceberg-based data lakes for large-scale analytics.

Cloud & Platform Engineering

• Work with AWS/GCP cloud services.
• Develop cloud-native data solutions using managed services.
• Awareness of CI/CD pipelines and Infrastructure as Code.
• Support platform modernization and cloud migration initiatives.

Stakeholder Management

• Partner with architects, product owners, and business stakeholders.
• Translate business requirements into technical solutions.
• Provide effort estimation and delivery planning.