Senior Software Engineer

TOMIA

Bengaluru

Hybrid

INR 1,800,000 - 3,600,000

Full time

14 days+

A complete application in a minute

Tailored resume and cover letter, ready to send.

Get past ATS filters

Job summary

MACH, a leader in global monetization solutions, is seeking a Senior Data Engineer to design, build and maintain end-to-end data pipelines in a Hadoop-based environment. This role focuses on scalable batch and real-time processing to enable analytics and ML applications.

You will own ingestion using NiFi or Kafka, develop transformations with Spark, optimize storage with HDFS formats, and tune Hive queries. Collaboration with DevOps and stakeholders is key in a hybrid Bengaluru-based setting.

Qualifications

  • Bachelor’s or Master’s degree in a relevant field (CS/IT)
  • 6-8 years of IT industry experience, Telecom background preferred but not mandatory
  • 3-7+ years of dedicated data engineering on Hadoop clusters, end-to-end production pipelines
  • Proficiency with Linux/Unix, shell scripting, and Git

Responsibilities

  • Design, build, and optimize end-to-end data pipelines (ETL/ELT) using Hadoop ecosystem components.
  • Ingest via Apache NiFi or Apache Kafka or Flume for streaming and batch data.
  • Develop data transformation workflows with Apache Spark (PySpark/Scala) and MapReduce.
  • Optimize storage in HDFS with Parquet/ORC and manage NoSQL like HBase or Cassandra.
  • Write and tune complex distributed queries in Apache Hive to meet SLAs.
  • Automate and monitor multi-stage pipelines with Airflow or Oozie.

Skills

Python
SQL
Scala/Java
Spark (PySpark/Scala)
Hive
HDFS
Kafka/Nifi
Linux
Git

Education

Bachelor’s or Master’s in CS/IT

Tools

Hadoop
Spark
Hive
Kafka
Nifi
Oozie
Airflow
HBase
Cassandra
Parquet/ORC

Job description

Job Description

Senior Data Engineer

Department

R&D

Position Type

Full-time

Job Location

Bangalore

Shift Timings

Regular

Work Mode

Hybrid

Industry Type

Telecom

Travel Required

NA

Why Us?

MACH is an innovator and leader in global monetization solutions. Their software optimizes and orchestrates critical inter-carrier business processes including data and financial clearing, network roaming, real-time campaign management, and interconnect solutions. MACH creates transformative communications solutions for today, tomorrow, and beyond, bridging wholesale and inter-carrier connectivity with roaming and clearing.

Collaboration and Innovation are at MACH’s core. Our success is built on diverse strengths across different teams, functions, and geographies summarised in our company values:

  • Open collaboration
  • Driving innovation
  • Customer commitment
  • Growth mindset
  • Deliver results

Please visit our website to know more about us - https://mach.com/

A Day in the Life of Senior Data Engineer:

We are seeking a highly skilled and analytical Senior Data Engineer with deep expertise in designing, building, and maintaining end-to-end distributed data pipelines within the Apache Hadoop ecosystem. In this role,

  • You will lead the architecture and implementation of scalable batch and real-time streaming pipelines.
  • You will be responsible for ingest, storage, large-scale processing, and serving layers, transforming massive volumes of unstructured, semi-structured, and structured data into high-performance datasets ready for downstream analytics and machine learning applications
So, how are we going to keep you busy?
  • Pipeline Architecture: Design, build, and optimize scalable end-to-end data pipelines (ETL/ELT) using Hadoop ecosystem components.
  • Data Ingestion: Implement high-throughput ingestion pipelines utilizing Apache Nifi or Apache Kafka or Flume to capture streaming event data and transactional batch files.
  • Distributed Processing: Develop robust data transformation workflows using Apache Spark (Core, SparkSQL, Structured Streaming) and MapReduce.
  • Data Storage & Modeling: Optimize data layout in HDFS utilizing columnar formats (Parquet, ORC) and manage NoSQL persistence layers using HBase or Cassandra
  • Query Performance Optimization: Author and tune highly complex distributed queries in Apache Hive to meet tight SLAs.
  • Orchestration & Workflow Management: Automate and monitor multi-stage pipeline DAGs using orchestration tools like Apache Airflow or Oozie.
  • Cluster Performance Tuning: Partner with DevOps/Infrastructure teams to debug issues related to data skew, resource allocation ( YARN ), cluster memory management, and Spark join strategies.
What will make you eligible for this role?
  • Relevant bachelor’s degree or master’s degree
  • 6-8 years of relevant experience in IT industry (Telecom preferred but not mandatory).
  • 3 to 7+ years of dedicated data engineering experience, with at least 3 years actively managing end-to-end production pipelines on Hadoop clusters
  • Proficiency working in Linux/Unix operating systems, shell scripting, and using code versioning tools like Git.
Experience, that will be an advantage:
  • Core Big Data Frameworks: Hands-on mastery of HDFS, MapReduce, YARN, Hive.
  • Distributed Compute: Advanced proficiency in Apache Spark using Python (PySpark) or Scala.
  • Streaming & Ingestion: Deep operational knowledge of Apache Nifi or Kafka (brokers, producers, consumers, and schema registry).
  • Programming Languages: Strong foundations in Python , SQL , and Scala or Java.
  • NoSQL Database Systems: Direct experience handling unstructured/semi-structured data architectures in HBase, MongoDB, or Cassandra.
  • Data Warehouse & Modelling: Strong grasp of dimensional data modelling, star/snowflake schemas, and data governance practices.
Limitation and Disclaimer:

The aforementioned job description is aimed to provide an overview of the overall type and level of work being performed for this role; it shouldn’t be interpreted as a complete list of all obligations, responsibilities, and qualifications needed for the job.

In no way does this job description imply or say that these are the exclusive responsibilities of the employee holding this position. Any additional job-related instructions or tasks that are requested by the manager must be completed by the employee.

Get your free, confidential resume review.

or drag and drop your file here.

Similar jobs

Similar jobs worth comparing

Senior Software Engineer
Senior Software Engineer

MACH Clearing Solutions India Pvt Ltd (IND) • Bengaluru

Hybrid
INR 2,500,000 - 4,000,000
Senior Software Engineer
Senior Software Engineer

Luminegrp • Bengaluru

Hybrid
INR 1,800,000 - 4,000,000
Big Data Engineer
Big Data Engineer

Infosys • Hyderabad, Pune District, Bengaluru

On-site
INR 1,200,000 - 2,000,000
Senior/Lead Big Data Engineer with Hadoop,Hive,Spark,SQL-Bangalore
Senior/Lead Big Data Engineer with Hadoop,Hive,Spark,SQL-Bangalore

Tredence • Bengaluru

Hybrid
INR 2,500,000 - 3,600,000
Senior Data Engineer
Senior Data Engineer

Luxoft India • India

On-site
INR 1,500,000 - 2,000,000
Big Data Developer
Big Data Developer

Viraaj HR Solutions • Maharashtra

On-site
INR 1,000,000 - 1,500,000
Collaborative engineering culture
Competitive compensation
Training in cloud and Big Data technologies
Big Data & Hadoop Developer
Big Data & Hadoop Developer

Tata Consultancy Services Limited • Mumbai

On-site
INR 1,200,000 - 2,000,000
Big Data Engineer
Big Data Engineer

Global Infovision • Chennai, Bengaluru

Hybrid
INR 1,500,000 - 2,600,000
Big Data Engineer_STG
Big Data Engineer_STG

Infosys • Hyderabad, Pune District, Bengaluru

Remote
INR 2,000,000 - 4,000,000
Senior Software Engineer – Data Platform Engineering
Senior Software Engineer – Data Platform Engineering

Myntra • Bengaluru

On-site
INR 2,400,000 - 4,200,000
Restaurant d'entreprise
Indemnités de stage/alternance