Data Engineer

Cornerstone Research; Hyderabad; On-site; INR 1,500,000 - 2,800,000; Full time

Job summary

Cornerstone is hiring a Data Engineer – Cloud & Big Data to design, build, and maintain scalable data pipelines across AWS, GCP, Kafka, and Databricks. You will craft reliable ETL/ELT processes, real-time ingestion, data transformations, monitoring, and cost-optimized architectures.

Collaborate with application, DevOps, QA, analytics, and product teams to deliver robust data solutions. The role requires 5+ years in data engineering and deep cloud experience across multi-cloud environments.

Qualifications

• 5+ years of experience in Data Engineering / Big Data Engineering.
• Strong hands-on experience with AWS + Databricks + Kafka.
• Experience working in multi-cloud environments (AWS/GCP).
• Strong understanding of distributed systems, streaming, CDC, schema evolution, and data governance.
• Experience troubleshooting high-volume production data pipelines.
• Experience with cloud performance and cost optimization.

Responsibilities

• Design and develop ETL/ELT data pipelines using AWS Glue, Databricks, and Spark.

Skills

Distributed systems

Streaming data

Data governance

Performance optimization

Problem solving

Tools

AWS

Databricks

Kafka

GCP

Job description

Data Engineer

Data Engineer – Cloud & Big Data

Role Summary:

Responsible for designing, developing, and maintaining scalable cloud-based data pipelines and data platforms across AWS, GCP, Kafka, and Databricks. The role involves building reliable ETL/ELT pipelines, real-time data ingestion, data transformation, monitoring, and performance/cost optimization.

In this role you will…

• Design and develop ETL/ELT data pipelines using AWS Glue, Databricks, and Spark.
• Build and maintain batch and real-time data pipelines using Apache Kafka / Confluent Kafka.
• Develop CDC-based data ingestion and event-driven data processing.
• Work with AWS services including Glue, S3, Lambda, IAM, KMS, Secrets Manager, and related services.
• Develop and optimize data solutions on GCP, including BigQuery, Cloud Storage, Cloud Functions, and related services.
• Implement data processing using Databricks, PySpark, Delta Lake, Bronze/Silver/Gold architecture, and DLT.
• Manage and optimize Elasticsearch/OpenSearch data ingestion, indexing, queries, and cluster performance.
• Implement data quality, validation, monitoring, logging, and alerting across data pipelines.
• Troubleshoot production issues involving Kafka, schema compatibility, CDC, Spark, Glue, Databricks, and cloud infrastructure.
• Optimize pipeline performance and cloud infrastructure costs.
• Implement secure data solutions using IAM, encryption, network controls, secrets management, and least-privilege access.
• Collaborate with application, DevOps, QA, analytics, and product teams to deliver reliable data solutions.

You'vegot what it takes if you have…

• 5+ years of experience in Data Engineering / Big Data Engineering.
• Strong hands-on experience with AWS + Databricks + Kafka.
• Experience working in multi-cloud environments (AWS/GCP).
• Strong understanding of distributed systems, streaming, CDC, schema evolution, and data governance.
• Experience troubleshooting high-volume production data pipelines.
• Experience with cloud performance and cost optimization.

#LI-Onsite

Our Culture:

Spark Greatness. Shatter Boundaries. Share Success. Are you ready? Because here, right now – is where the future of work is happening. Where curious disruptors and change innovators like you are helping communities and customers enable everyone – anywhere – to learn, grow and advance. To be better tomorrow than they are today.

Who We Are:

At Cornerstone, we believe in AI that works in the service of people, amplifying their judgment to drive high-performing, future-ready organizations forward. Cornerstone Workforce AI™, the intelligence platform for workforce readiness, brings together workforce and labor market data into a proprietary Cornerstone People Graph™, translating signals into intelligence, targeting learning where it matters, developing critical skills, and surfacing hidden talent. Delivered as an open, enterprise platform across whatever application your people work in every day, Cornerstone Workforce AI is built for scale, security, and trust, with certified AI guardrails. As an industry leader, Cornerstone is helping approximately 7,000 organizations, 140M+ users, across 186 countries build continuous workforce readiness.

Check us out on LinkedIn, Comparably, Glassdoor, and Facebook!