Data Engineer

CyberU, Inc.; Hyderabad; On-site; INR 900,000 - 1,500,000; Full time

Job summary

Cornerstone is seeking a Data Engineer based in Hyderabad to design, build, and maintain scalable cloud-based data pipelines across AWS, GCP, Kafka, and Databricks. You will craft both batch and real-time data flows, implement CDC-based ingestion, and optimize data platforms for reliability and cost.

Key focus areas include Spark-based ETL/ELT, Bronze/Silver/Gold architecture, and secure data solutions with IAM and encryption.

Qualifications

• 5+ years of experience in Data Engineering / Big Data Engineering.
• Hands-on experience with AWS, Databricks and Kafka.
• Experience working in multi-cloud environments (AWS/GCP).
• Strong understanding of distributed systems, streaming, CDC, schema evolution, and data governance.
• Experience troubleshooting high-volume production pipelines.
• Experience with cloud performance and cost optimization.

Responsibilities

• Design and develop ETL/ELT data pipelines using AWS Glue, Databricks, and Spark.
• Build and maintain batch and real-time data pipelines using Apache Kafka / Confluent Kafka.
• Develop CDC-based data ingestion and event-driven data processing.
• Work with AWS services including Glue, S3, Lambda, IAM, KMS, Secrets Manager, and related services.
• Develop and optimize data solutions on GCP, including BigQuery, Cloud Storage, Cloud Functions, and related services.
• Implement data processing using Databricks, PySpark, Delta Lake, Bronze/Silver/Gold architecture, and DLT.
• Manage and optimize Elasticsearch/OpenSearch data ingestion, indexing, queries, and cluster performance.
• Implement data quality, validation, monitoring, logging, and alerting across data pipelines.
• Troubleshoot production issues involving Kafka, schema compatibility, CDC, Spark, Glue, Databricks, and cloud infrastructure.
• Optimize pipeline performance and cloud infrastructure costs.
• Implement secure data solutions using IAM, encryption, network controls, secrets management, and least-privilege access.
• Collaborate with application, DevOps, QA, analytics, and product teams to deliver reliable data solutions.

Skills

Big Data

Distributed Systems

Streaming

ETL/ELT

Tools

AWS

Databricks

Kafka

Job description

Job Description:

Were looking for a

Data Engineer

Office Based,Hyderabad Office

Data Engineer – Cloud & Big Data

Role Summary:
Responsible for designing, developing, and maintaining scalable cloud-based data pipelines and data platforms across AWS, GCP, Kafka, and Databricks. The role involves building reliable ETL/ELT pipelines, real-time data ingestion, data transformation, monitoring, and performance/cost optimization.

In this role you will…

• Design and develop ETL/ELT data pipelines using AWS Glue, Databricks, and Spark.
• Build and maintain batch and real-time data pipelines using Apache Kafka / Confluent Kafka.
• Develop CDC-based data ingestion and event-driven data processing.
• Work with AWS services including Glue, S3, Lambda, IAM, KMS, Secrets Manager, and related services.
• Develop and optimize data solutions on GCP, including BigQuery, Cloud Storage, Cloud Functions, and related services.
• Implement data processing using Databricks, PySpark, Delta Lake, Bronze/Silver/Gold architecture, and DLT.
• Manage and optimize Elasticsearch/OpenSearch data ingestion, indexing, queries, and cluster performance.
• Implement data quality, validation, monitoring, logging, and alerting across data pipelines.
• Troubleshoot production issues involving Kafka, schema compatibility, CDC, Spark, Glue, Databricks, and cloud infrastructure.
• Optimize pipeline performance and cloud infrastructure costs.
• Implement secure data solutions using IAM, encryption, network controls, secrets management, and least-privilege access.
• Collaborate with application, DevOps, QA, analytics, and product teams to deliver reliable data solutions.

Youvegot what it takes if you have…

• 5+ years of experience in Data Engineering / Big Data Engineering.
• Strong hands-on experience with AWS + Databricks + Kafka.
• Experience working in multi-cloud environments (AWS/GCP).
• Strong understanding of distributed systems, streaming, CDC, schema evolution, and data governance.
• Experience troubleshooting high-volume production data pipelines.
• Experience with cloud performance and cost optimization.

#LI-Onsite

Our Culture:

Spark Greatness. Shatter Boundaries. Share Success. Are you ready? Because here, right now – is where the future of work is happening. Where curious disruptors and change innovators like you are helping communities and customers enable everyone – anywhere – to learn, grow and advance. To be better tomorrow than they are today.

Who We Are:

At Cornerstone, we believe in AI that works in the service of people, amplifying their judgment to drive high-performing, future-ready organizations forward. Cornerstone Workforce AI, the intelligence platform for workforce readiness, brings together workforce and labor market data into a proprietary Cornerstone People Graph, translating signals into intelligence, targeting learning where it matters, developing critical skills, and surfacing hidden talent. Delivered as an open, enterprise platform across whatever application your people work in every day, Cornerstone Workforce AI is built for scale, security, and trust, with certified AI guardrails. As an industry leader, Cornerstone is helping approximately 7,000 organizations, 140M+ users, across 186 countries build continuous workforce readiness.

Check us out on LinkedIn, Comparably, Glassdoor, and Facebook!

Requirements: