Big Data Engineer

Cognine Technologies; Hyderabad; On-site; INR 1,800,000 - 3,200,000; Full time

Benefits offered by this job

Palantir Foundry exposure (preferred)

Job summary

Cognine Technologies in Hyderabad is seeking a skilled Big Data Engineer with strong PySpark and data engineering experience. The ideal candidate will design scalable ETL pipelines, work with streaming data, and deploy solutions on cloud platforms.

5–10 years of hands-on experience, knowledge of Kafka, Flink, Kubernetes, Hadoop and cloud services (Azure, AWS, GCP) are preferred. Palantir Foundry exposure is a plus.

Qualifications

• 5–10 years of Data Engineering experience.
• Strong Python and PySpark skills.
• Experience with cloud platforms (Azure/AWS/GCP).

Responsibilities

• Design, develop, and maintain ETL/data pipelines using Python and PySpark
• Work with real-time/streaming data systems
• Build and optimize data processing solutions on cloud platforms
• Collaborate with cross-functional teams to understand business requirements
• Develop and maintain CI/CD pipelines
• Ensure data quality, performance, and scalability

Skills

PySpark

Python

Data engineering

ETL pipelines

Streaming data

Cloud platforms

Data architecture

CI/CD

Tools

Kafka

Flink

Kubernetes

Hadoop

Spark

MongoDB

Azure

AWS

GCP

Palantir Foundry

Job description

We are looking for a skilled Big Data Engineer with strong experience in PySpark and data engineering. The ideal candidate will have hands‑on experience in building scalable data pipelines, working with streaming data, and cloud platforms. Experience with Palantir Foundry is highly preferred.

5 – 10 years of experience

Hyderabad

Full-Time

Key Responsibilities

• Design, develop, and maintain ETL/data pipelines using Python and PySpark
• Work with real‑time/streaming data systems
• Build and optimize data processing solutions on cloud platforms
• Collaborate with cross‑functional teams to understand business requirements
• Develop and maintain CI/CD pipelines
• Ensure data quality, performance, and scalability

Required Skills

• 5–10+ years of experience in Data Engineering
• Strong expertise in Python and PySpark
• Experience with Kafka, Flink, and Kubernetes
• Hands‑on with Big Data technologies (Hadoop, Spark, MongoDB)
• Experience with cloud platforms (Azure, AWS, or GCP)
• Good understanding of data architecture and pipeline design

Preferred Skills

• Experience with Palantir Foundry
• Knowledge of Java
• Experience in consulting or Utilities domain