Data Platform Engineer

Yotta Infrastructure; Delhi; On-site; INR 600,000 - 1,000,000; Full time

Job summary

Yotta Infrastructure in Delhi is seeking a Junior Data Platform Engineer to support and manage Spark, Airflow, JupyterHub, Python, and Linux-based data platforms. You will handle monitoring, deployment, troubleshooting, and performance optimization under supervision.

You will also provide AWS L2 support covering core services, IAM, monitoring, and incident troubleshooting. The role involves close collaboration with senior teams to ensure reliable and secure data operations.

Qualifications

• Basic understanding of Apache Spark, Hive, Hadoop HDFS, and Airflow, including DAGs, scheduling, and task dependencies.
• Hands-on Python scripting, Linux command-line, system troubleshooting, container environments, Git, and basic ETL/distributed computing concepts.
• AWS L1 + Engineering skills with basic knowledge of core AWS services, monitoring, IAM, compute, storage, networking, and cloud operations.
• Ability to support deployment, operations, monitoring, and troubleshooting of data platforms and AWS environments while working closely with senior engineers.

Responsibilities

• Assist in the setup, monitoring, and maintenance of Apache Spark clusters, Apache Hive, Hadoop and Airflow environments.
• Support the development and scheduling of data pipelines using Airflow DAGs and Python scripts.
• Help manage and configure JupyterHub for multi-user access and integration with Spark.
• Monitor cluster health and performance under guidance and assist in troubleshooting Spark job failures.
• Write and maintain Python automation scripts for data workflows, ETL, and process automation.
• Participate in code reviews, documentation, and deployment activities.
• Learn and follow best practices for distributed data processing, CI/CD, and DevOps workflows.
• Collaborate with senior engineers and data scientists to implement improvements and new features.

Skills

Spark basics

Python scripting

Linux commands

Airflow DAGs

ETL concepts

Git

AWS basics

DevOps basics

CI/CD basics

JupyterHub basics

Education

Bachelor's degree

Tools

Docker

Kubernetes

JupyterHub

Hadoop

Hive

Spark cluster

Job description

Job Scope

We are looking for a Junior Data Platform Engineer to support and manage Apache Spark, Apache Airflow, JupyterHub, Python, and Linux-based data platforms.
The role includes platform monitoring, deployment, troubleshooting, performance optimization, and L2 operational support.

You will also provide AWS L2 support covering core AWS services, compute, storage, networking, IAM, monitoring, and incident troubleshooting.

The engineer will work closely with senior teams to ensure reliable, secure, and efficient operation of the data and cloud environment.

Total /Relevant Experience:

• 1+ years experience

Key Responsibilities:

• Assist in the setup, monitoring, and maintenance of Apache Spark clusters, Apache Hive, Hadoop and Airflow environments.
• Support the development and scheduling of data pipelines using Airflow DAGs and Python scripts.
• Help manage and configure JupyterHub for multi-user access and integration with Spark.
• Monitor cluster health and performance under guidance and assist in troubleshooting Spark job failures.
• Write and maintain Python automation scripts for data workflows, ETL, and process automation.
• Participate in code reviews, documentation, and deployment activities.
• Learn and follow best practices for distributed data processing, CI/CD, and DevOps workflows.
• Collaborate with senior engineers and data scientists to implement improvements and new features.

Must-have skill:

• Basic understanding of Apache Spark, Hive, Hadoop HDFS, and Airflow, including DAGs, scheduling, and task dependencies.
• Hands-on Python scripting, Linux command-line, system troubleshooting, container environments, Git, and basic ETL/distributed computing concepts.
• AWS L1 + Engineering skills with basic knowledge of core AWS services, monitoring, troubleshooting, IAM, compute, storage, networking, and cloud operations.
• Ability to support deployment, operations, monitoring, and troubleshooting of data platforms and AWS environments while working closely with senior engineers.

Good-to-Have Skills:

• Exposure to Jupyter / JupyterHub for collaborative notebook environments.
• Knowledge of Docker or Kubernetes.
• Knowledge of Hadoop and Apache Spark Cluster.
• Familiarity with SQL and working with structured/unstructured data.
• Experience with cloud platforms (AWS, GCP, or Azure) is a plus.
• Interest in big data (Hadoop), DevOps, and data pipeline automation.

Qualifications Criteria:

• Bachelors or any relevant Degree.