Data Lake Engineer

Care Health insurance; Gurugram District; On-site; INR 1,200,000 - 2,000,000; Full time

Job summary

Care Health Insurance is seeking an AWS Data Engineer to design, develop, and maintain data pipelines on an enterprise AWS Data Lake platform. You will enable analytics and reporting by ingesting data from multiple systems using S3, Glue, EMR, Athena, Redshift and other services.

The role requires strong Python, PySpark, SQL skills and experience with Airflow for workflow orchestration. You will collaborate with analytics and business teams and ensure data governance and performance optimization.

Qualifications

• 2 to 5 years of experience in Data Engineering or Data Platform Development.
• Hands-on experience with AWS data services: S3, Glue, EMR, Athena, Redshift, Lambda, RDS, DMS, and IAM.
• Proficient in Python and PySpark.
• Experience with Apache Airflow for workflow orchestration.
• Strong SQL skills and data modelling knowledge.
• Experience building and supporting ETL / ELT pipelines.
• Understanding of data lake architecture, data warehousing, and data modelling.
• Working knowledge of PostgreSQL or similar relational databases.
• Experience handling production pipeline issues and troubleshooting.

Responsibilities

• Develop, maintain, and optimise data ingestion and ETL / ELT pipelines on AWS.
• Work with AWS services such as S3, Glue, EMR, Athena, Redshift, Lambda, RDS, and DMS.
• Build and manage data processing workflows using Python, PySpark, and Apache Airflow.
• Implement data lake architecture with staging, curated, and consumption layers.
• Write complex SQL queries and perform data transformation, validation, and modelling.
• Work with PostgreSQL and RDS for structured data storage and consumption.
• Support API-based data access using Hasura GraphQL or similar technologies.
• Monitor production data pipelines, troubleshoot failures, and perform root cause analysis.
• Optimise Spark jobs, SQL queries, and data pipelines for performance and cost efficiency.
• Follow AWS security practices, IAM permissions, and data governance standards.
• Collaborate with application, analytics, and business teams to understand data requirements.

Skills

Python

PySpark

SQL

Data engineering

Airflow

Data pipelines

AWS services

PostgreSQL

Tools

AWS services (S3, Glue, EMR, Athena, Redshift, Lambda, RDS, DMS, IAM)

Hasura GraphQL

DBT

Kafka

Job description

We are looking for a skilled AWS Data Engineer with 2 to 5 years of experience in designing, developing, and maintaining data pipelines and cloud-based data platforms. The candidate will work on an enterprise AWS Data Lake platform that collects data from multiple business systems, processes large datasets, and makes them available for analytics, reporting, and application development. The ideal candidate should have strong hands-on experience with AWS data services, Python, PySpark, SQL, and Apache Airflow, along with an understanding of data warehousing and API-based data serving.

Responsibilities:

• Develop, maintain, and optimise data ingestion and ETL / ELT pipelines on AWS.
• Work with AWS services such as S3, Glue, EMR, Athena, Redshift, Lambda, RDS, and DMS.
• Build and manage data processing workflows using Python, PySpark, and Apache Airflow.
• Implement data lake architecture with staging, curated, and consumption layers.
• Write complex SQL queries and perform data transformation, validation, and modelling.
• Work with PostgreSQL and RDS for structured data storage and consumption.
• Support API-based data access using Hasura GraphQL or similar technologies.
• Monitor production data pipelines, troubleshoot failures, and perform root cause analysis.
• Optimise Spark jobs, SQL queries, and data pipelines for performance and cost efficiency.
• Follow AWS security practices, IAM permissions, and data governance standards.
• Collaborate with application, analytics, and business teams to understand data requirements.

Requirements:

• 2 to 5 years of experience in Data Engineering, AWS Data Engineering, or Data Platform Development.
• Strong hands-on experience with AWS services: S3, Glue, EMR, Athena, Redshift, Lambda, RDS, DMS, and IAM.
• Good experience in Python and PySpark.
• Practical experience with Apache Airflow for workflow orchestration.
• Strong SQL skills and understanding of database concepts.
• Experience in building and supporting ETL / ELT pipelines.
• Understanding of data lake architecture, data warehousing, and data modelling.
• Working knowledge of PostgreSQL or similar relational databases.
• Experience handling production pipeline issues and troubleshooting.

Good to Have:

• Experience with Hasura GraphQL or any GraphQL-based API / data-serving platform.
• Knowledge of DBT (Data Build Tool).
• Understanding of Lakehouse architecture and table formats such as Apache Iceberg or Delta Lake.
• Experience with Kafka or other streaming data technologies.
• Knowledge of Amazon Redshift performance optimisation, distribution keys, and sort keys.
• Experience with AWS Aurora and API Gateway.
• BFSI, Banking, or Insurance domain experience.

Ideal Candidate Profile:

• Strong hands-on technical and problem-solving skills.
• Experience working on real-time or batch-based production data pipelines.
• Good understanding of data processing, transformation, and storage concepts.
• Ability to troubleshoot technical issues and work independently.
• Good communication and collaboration skills.
• Note: Candidates with strong practical experience in AWS Data Lake, PySpark, Airflow, Python, and SQL will be preferred. Real-world project experience is more important than theoretical knowledge or certifications.