Cloud Data Engineer

SWITS DIGITAL Private Limited; Mumbai; Hybrid; INR 1,200,000 - 1,800,000; Full time

Job summary

SWITS DIGITAL Private Limited seeks a Cloud Data Engineer to design, build, and optimize data pipelines using Python and Spark in a hybrid Bangalore environment. You will work with Hadoop/Hive/HDFS, Spark tuning, and distributed processing to deliver scalable data solutions.

Strong hands-on experience with Hadoop ecosystem and containerized Spark workloads is required, with familiarity in Cloudera and CI/CD practices to accelerate delivery.

Qualifications

• 5–7 years of hands-on experience in data engineering or big data platforms.

Responsibilities

• Develop, maintain, and optimize scalable data engineering solutions using Python and Apache Spark/PySpark.

Skills

Hadoop

Hive

HDFS

PySpark

Python

Distributed data processing

Spark optimization

Tools

Kubernetes

JFrog

Cloudera

Impala

CI/CD pipelines

Job description

Job Title: Cloud Data Engineer - Location - Bangalore - (Hybrid - 3Days a Week)

Looking for Immediate Joiners / Serving Notice Peroid.

Experience

• 5 7 years of relevant experience in Data Engineering, Big Data, or Cloud Data Engineering.
• Strong hands-on experience with Hadoop, Hive, HDFS, and PySpark is mandatory.

Key Responsibilities & Technical Skills

• Develop, maintain, and optimize scalable data engineering solutions using Python and Apache Spark/PySpark.
• Design and build reusable, scalable data ingestion and data transformation frameworks for large-scale data processing.
• Work extensively with the Hadoop ecosystem, including Hadoop, Hive, HDFS, and Impala, preferably in Cloudera-based environments.
• Demonstrate strong understanding of HDFS architecture and internals, including:
  • Data storage architecture
  • Partition management
  • Compaction strategies
  • Resource utilization
  • Cluster performance optimization
• Troubleshoot complex platform, storage, compute, and cluster-level issues, perform detailed root cause analysis, and implement performance improvements.
• Work with a variety of database technologies and object storage platforms, ensuring optimized connectivity, data access, and processing patterns.
• Apply strong knowledge of distributed data processing, Spark optimization, and data platform engineering best practices.
• Develop and support applications using Kubernetes, including:
  • Containerized Spark workloads
  • Pod management
  • Application scaling
  • Troubleshooting
  • Operational support
• Build, package, and deploy Spark workloads using JFrog-managed container images and automated CI/CD pipelines.
• Leverage AI Agents, Generative AI, and LLM-powered development tools to improve engineering productivity and accelerate software delivery.
• Collaborate with engineering and platform teams to develop reliable, scalable, and high-performance data solutions.

Mandatory Skills

• Hadoop
• Hive
• HDFS
• PySpark / Apache Spark
• Python
• Distributed data processing
• Spark optimization

Good to Have / Added Advantage

• Cloudera ecosystem
• Impala
• Kubernetes
• Containerized Spark workloads
• JFrog
• CI/CD pipelines
• Object storage platforms
• Scala development
• Generative AI / AI Agents / LLM-powered development tools

Preferred Profile

Candidates should have 5 7 years of hands-on experience in data engineering or big data platforms, with strong practical expertise in Hadoop, Hive, HDFS, and PySpark. Experience in Cloudera, Kubernetes, containerized Spark, CI/CD, and platform performance optimization will be an added advantage.