Data Engineer

Cornerstone; Hyderabad; Sur place; INR 2 500 000 - 4 500 000; Plein temps

Résumé du poste

Cornerstone is hiring a Data Engineer to design, build, and maintain scalable cloud-based data pipelines and platforms across AWS, GCP, Kafka, and Databricks. The role focuses on reliable ETL/ELT workflows, real-time data ingestion, data transformation, and cost-aware optimization.

You will implement secure data solutions, monitor performance, and collaborate with cross-functional teams to deliver robust data capabilities for analytics and product use cases.

Qualifications

• 5+ years of experience in Data Engineering / Big Data Engineering.
• Strong hands-on experience with AWS, Databricks, and Kafka.
• Experience in multi-cloud environments (AWS/GCP).
• Knowledge of distributed systems, streaming, CDC, schema evolution, and data governance.
• Experience troubleshooting high-volume production data pipelines.
• Experience with cloud performance and cost optimization.

Responsabilités

• Design and develop ETL/ELT data pipelines with AWS Glue, Databricks, and Spark.
• Build batch and real-time pipelines using Apache Kafka.
• Develop CDC-based data ingestion and event-driven processing.
• Work with AWS services: Glue, S3, Lambda, IAM, KMS, Secrets Manager.
• Develop and optimize data solutions on GCP: BigQuery, Cloud Storage, Cloud Functions.
• Implement data processing with Databricks, PySpark, Delta Lake, Bronze/Silver/Gold, and DLT.
• Manage Elasticsearch/OpenSearch data ingestion, indexing, queries, and clusters.
• Implement data quality, monitoring, logging, and alerting.

Connaissances

AWS

Multi-cloud

Distributed systems

Data governance

Troubleshooting

Outils

Databricks

Kafka

Spark

BigQuery

Description du poste

Data Engineer - Cloud & Big Data

Role Summary

Responsible for designing, developing, and maintaining scalable cloud-based data pipelines and data platforms across AWS, GCP, Kafka, and Databricks. The role involves building reliable ETL/ELT pipelines, real-time data ingestion, data transformation, monitoring, and performance/cost optimization.

In this role you will...

• Design and develop ETL/ELT data pipelines using AWS Glue, Databricks, and Spark.
• Build and maintain batch and real-time data pipelines using Apache Kafka / Confluent Kafka.
• Develop CDC-based data ingestion and event-driven data processing.
• Work with AWS services including Glue, S3, Lambda, IAM, KMS, Secrets Manager, and related services.
• Develop and optimize data solutions on GCP, including BigQuery, Cloud Storage, Cloud Functions, and related services.
• Implement data processing using Databricks, PySpark, Delta Lake, Bronze/Silver/Gold architecture, and DLT.
• Manage and optimize Elasticsearch/OpenSearch data ingestion, indexing, queries, and cluster performance.
• Implement data quality, validation, monitoring, logging, and alerting across data pipelines.
• Troubleshoot production issues involving Kafka, schema compatibility, CDC, Spark, Glue, Databricks, and cloud infrastructure.
• Optimize pipeline performance and cloud infrastructure costs.
• Implement secure data solutions using IAM, encryption, network controls, secrets management, and least-privilege access.
• Collaborate with application, DevOps, QA, analytics, and product teams to deliver reliable data solutions.

You'vegot what it takes if you have...

• 5+ years of experience in Data Engineering / Big Data Engineering.
• Strong hands-on experience with AWS + Databricks + Kafka.
• Experience working in multi-cloud environments (AWS/GCP).
• Strong understanding of distributed systems, streaming, CDC, schema evolution, and data governance.
• Experience troubleshooting high-volume production data pipelines.
• Experience with cloud performance and cost optimization.

#LI-Onsite