Data Platform Engineer

Umanist NA

Bengaluru

Hybrid

INR 1,200,000 - 1,800,000

Full time

14 hours ago
Be an early applicant

Get a reply from this employer

Your resume and cover letter, tailored to exactly what they’re hiring for.

Get past ATS filters

Job summary

Umanist NA in Bengaluru is seeking a Big Data and Operation Engineer to manage and improve enterprise data platforms, spanning data engineering, pipeline development, production support and data operations.

The ideal candidate will have strong hands-on experience with Python, PySpark, SQL and Big Data technologies, and understand production environments, workflow orchestration, data quality and application operations.

Qualifications

  • Strong programming skills in Python, PySpark and SQL.
  • Hands-on experience with BigData tech stacks with Hadoop, Hive, HDFS, Apache Spark and Spark SQL.
  • Good understanding of ETL processes, data transformation, data modelling and pipeline optimization.
  • Experience with batch processing and Spark Structured Streaming.
  • Working knowledge of MongoDB and relational databases.
  • Experience working with Parquet and Avro file formats.
  • Hands-on exposure to Apache Kafka and Kafka-based data processing.
  • Experience with Apache Airflow for workflow scheduling, monitoring and troubleshooting.
  • Knowledge of Power BI for dashboard development and report refresh management.
  • Understanding of CI/CD pipelines and deployment tools such as GitHub, Bitbucket or equivalent.
  • Working knowledge of Linux / RHEL operating systems.
  • Understanding of ITIL processes, incident management and ServiceNow/ITSM tools.
  • Familiarity with development tools such as VS Code, Jupyter Notebook or equivalent.

Responsibilities

  • Data Engineering & Pipeline Development: Build, maintain and enhance scalable data pipelines using Python, PySpark, SQL and Apache Spark.
  • Develop data processing solutions for both batch and streaming workloads.
  • Work with large datasets using Hadoop, Hive, HDFS and Spark-based technologies.
  • Handle data extraction, transformation, integration and processing across multiple sources.
  • Implement data models and transformation logic to support business and reporting requirements.
  • Identify performance bottlenecks and optimize existing data pipelines.
  • Investigate data inconsistencies, processing failures and technical defects.
  • Contribute to platform enhancements, automation and data architecture improvements.

Skills

Python
PySpark
SQL

Tools

Hadoop
Hive
HDFS
Apache Spark
Spark SQL
Parquet
Avro
Apache Kafka
Apache Airflow
Power BI
GitHub
Bitbucket
VS Code
Jupyter Notebook
Linux / RHEL
ITIL/ServiceNow

Job description

  • Work Mode:** Hybrid
  • Employment Type:** Full-Time
Big Data and Operation Engineer
  • Experience:** 4–10 Years (Relevant Experience)
  • Location:** Bangalore
  • Work Mode:** Hybrid
  • Employment Type:** Full-Time
Role Overview

We are looking for a skilled **Big Data and Operation Engineer** to manage and improve enterprise data platforms, with responsibilities spanning data engineering, pipeline development, production support and data operations.

The role involves building and maintaining data pipelines, managing large-scale data processing workflows, monitoring platform performance and resolving data-related issues. You will also work closely with business stakeholders and technical teams to understand requirements and deliver reliable data solutions.

The ideal candidate should have strong hands‑on experience with Python, PySpark, SQL and Big Data technologies, along with an understanding of production environments, workflow orchestration, data quality and application operations.

Key Responsibilities
1. Data Engineering & Pipeline Development
  • Build, maintain and enhance scalable data pipelines using Python, PySpark, SQL and Apache Spark.
  • Develop data processing solutions for both batch and streaming workloads.
  • Work with large datasets using Hadoop, Hive, HDFS and Spark-based technologies.
  • Handle data extraction, transformation, integration and processing across multiple sources.
  • Implement data models and transformation logic to support business and reporting requirements.
  • Identify performance bottlenecks and optimize existing data pipelines.
  • Investigate data inconsistencies, processing failures and technical defects.
  • Contribute to platform enhancements, automation and data architecture improvements.
2. Data Platform Monitoring & Operations
  • Monitor scheduled data workflows, applications, microservices and reporting activities.
  • Manage data pipeline execution, scheduling and workflow dependencies using Apache Airflow.
  • Investigate failed jobs, delayed data processing and application-related issues.
  • Perform controlled data reloads, job restarts and reprocessing activities whenever required.
  • Monitor data availability, freshness and overall platform performance.
  • Support production deployments, releases and ongoing platform maintenance.
  • Ensure smooth functioning of data processing systems and minimize operational disruptions.
3. Production Support & Incident Management
  • Manage production incidents, service requests and change activities following ITIL practices.
  • Investigate recurring technical issues and perform root-cause analysis.
  • Coordinate with application, infrastructure, development and database teams for issue resolution.
  • Work with ITSM platforms such as ServiceNow to track and manage operational tickets.
  • Maintain technical documents, troubleshooting guides and operational knowledge resources.
  • Provide timely updates to stakeholders regarding production issues and their resolution.
4. Data Quality & Testing
  • Perform data validation and quality checks across different processing stages.
  • Identify missing records, incorrect values, inconsistencies and data processing errors.
  • Conduct unit, integration, regression and functional testing for data solutions.
  • Validate data transformations, reporting outputs and downstream data consumption.
  • Develop mechanisms to improve data accuracy, reliability and monitoring.
  • Ensure that data pipelines meet expected quality and performance standards.
5. Reporting & Business Intelligence
  • Develop and maintain reports and dashboards using Power BI.
  • Monitor scheduled report refreshes and troubleshoot data connectivity or refresh failures.
  • Work with business users to understand reporting requirements and data needs.
  • Ensure accurate and consistent data availability for business reporting.
6. Stakeholder & Requirement Management
  • Collaborate with business users, data engineers, architects and application teams to understand technical requirements.
  • Convert business expectations into technical specifications and data engineering solutions.
  • Participate in requirement-gathering sessions and technical discussions.
  • Evaluate the impact of new requirements on existing data pipelines and platform architecture.
  • Identify opportunities to leverage available data for additional business use cases.
  • Coordinate with cross-functional teams to ensure smooth implementation of requirements.
  • Communicate technical challenges, dependencies and proposed solutions effectively.
Required Technical Skills
  • Strong programming skills in Python, PySpark and SQL.
  • Hands‑on experience BigData tech stacks with Hadoop, Hive, HDFS, Apache Spark and Spark SQL.
  • Good understanding of ETL processes, data transformation, data modelling and pipeline optimization.
  • Experience with batch processing and Spark Structured Streaming.
  • Working knowledge of MongoDB and relational databases.
  • Experience working with Parquet and Avro file formats.
  • Hands‑on exposure to Apache Kafka and Kafka‑based data processing.
  • Experience with Apache Airflow for workflow scheduling, monitoring and troubleshooting.
  • Knowledge of Power BI for dashboard development and report refresh management.
  • Understanding of CI/CD pipelines and deployment tools such as GitHub, Bitbucket or equivalent.
  • Working knowledge of Linux / RHEL operating systems.
  • Understanding of ITIL processes, incident management and ServiceNow/ITSM tools.
  • Familiarity with development tools such as VS Code, Jupyter Notebook or equivalent.
Get your free, confidential resume review.

or drag and drop your file here.

Similar jobs

Similar jobs worth comparing

Data Engineer (Platform Focus)
Data Engineer (Platform Focus)

Comviva • Gurugram District

On-site
INR 900,000 - 1,500,000
Big Data Engineer
Big Data Engineer

Global Infovision • Chennai, Bengaluru

Hybrid
INR 1,500,000 - 2,600,000
Data Platform Engineer
Data Platform Engineer

Mployee.me • Delhi

On-site
INR 420,000 - 650,000
Data Platform Engineer
Data Platform Engineer

Yotta Infrastructure • Delhi

On-site
INR 600,000 - 1,000,000
Senior Data Platform Engineer
Senior Data Platform Engineer

Relevance Lab • Bengaluru

On-site
INR 400,000 - 700,000
Senior Data Engineer
Senior Data Engineer

Luxoft India • India

On-site
INR 1,500,000 - 2,000,000
Lead Data Engineer (Lead Software Engineer)
Lead Data Engineer (Lead Software Engineer)

DigitalXNode • Ahmedabad District

On-site
INR 1,800,000 - 2,600,000
Data Engineer
Data Engineer

Model Business Services • SECTOR 78

On-site
INR 1,200,000 - 1,800,000
Data Engineer
Data Engineer

Idexcel • Bengaluru

On-site
INR 900,000 - 1,400,000
Data Engineer
Data Engineer

Primesoft Solutions • Mumbai, Hyderabad, Bengaluru

On-site
INR 1,800,000 - 2,400,000