Data Engineer

Amplicomm; Bengaluru; On-site; INR 1,500,000 - 2,800,000; Full time

Job summary

Amplicomm is seeking an experienced Data Engineer to design, build, and optimize scalable data pipelines using Spark, PySpark, Kafka, Airflow, Hive, and Hadoop. You will own end-to-end data workflows, from ingestion and transformation to orchestration and delivery.

The role emphasizes streaming and batch processing, data quality, and performance tuning. Ideal for professionals with 4–6 years of hands-on Big Data experience in modern cloud ecosystems.

Qualifications

• 4-6 years of hands-on experience as a Data Engineer building and running production data pipelines.
• Strong hands-on experience with Apache Spark, PySpark, and Scala Spark.
• Strong knowledge of Spark SQL, DataFrames, joins, UDFs, and Spark performance tuning.
• Hands-on experience with Hadoop, Hive, and HDFS.
• Strong experience with Apache Kafka, including topics, partitions, consumer groups, offsets, and message delivery.
• Hands-on experience with Apache Airflow, including DAG development, scheduling, dependencies, retries, backfills, and monitoring.
• Strong programming skills in Python, SQL, and Scala.
• Experience with incremental imports, partitioning, bucketing, and large-scale data processing.
• Hands-on experience managing Spark jobs using spark-submit and monitoring production workloads.
• Experience with AWS, particularly S3, EMR, Glue, EC2, and IAM.
• Experience with Docker for packaging and deploying data workloads.
• Understanding of data modelling, schema evolution, and data formats such as Parquet, ORC, Avro, and JSON.
• Strong analytical, problem-solving, debugging, and data-quality mindset.

Responsibilities

• Design, develop, and maintain scalable Big Data solutions using Apache Spark, PySpark, Scala Spark, Hive, Hadoop, HDFS, Kafka, and Airflow.
• Build and maintain batch and streaming data pipelines using PySpark, Scala, Python, and SQL.
• Process large volumes of structured, semi-structured, and streaming data efficiently.
• Build complex data processing pipelines and integrate data through REST APIs.
• Implement incremental data processing, partitioning, and bucketing to improve Spark performance and cost efficiency.
• Submit, monitor, tune, and troubleshoot Spark jobs, including resource sizing, executors, memory, and failure recovery.
• Read, process, and write data using Hive and Spark.
• Build Kafka-based pipelines for real-time and near-real-time data processing.
• Design and manage Airflow DAGs, scheduling, dependencies, retries, backfills, SLAs, and alerts.
• Deploy and run data workloads using AWS and Docker.
• Troubleshoot production pipeline failures and drive issues to root cause and closure.
• Improve pipeline performance, reliability, scalability, and data quality.
• Work closely with Product, Analytics, and Engineering teams to deliver accurate and timely data.
• Document data pipelines, data models, processes, and technical solutions.

Skills

Apache Spark

PySpark

Scala Spark

Spark SQL

DataFrames

Hadoop

Hive

HDFS

Kafka

Airflow

Python

SQL

Scala

AWS

Docker

Parquet

ORC

Avro

JSON

Data modelling

Tools

spark-submit

Job description

We are looking for an experienced Data Engineer with strong hands-on experience in Big Data technologies to design, build, maintain, and optimize scalable data pipelines and processing systems.

The role involves building batch and streaming pipelines using PySpark, Scala Spark, Kafka, Airflow, Hive, and Hadoop, processing large volumes of structured, semi-structured, and streaming data, and ensuring pipeline performance, reliability, and data quality.

The candidate will own data pipelines end-to-end, from data ingestion and transformation to orchestration, monitoring, troubleshooting, and downstream data delivery.

Essential Skills & Experience:

• 4-6 years of hands-on experience as a Data Engineer building and running production data pipelines.
• Strong hands-on experience with Apache Spark, PySpark, and Scala Spark.
• Strong knowledge of Spark SQL, DataFrames, joins, UDFs, and Spark performance tuning.
• Hands-on experience with Hadoop, Hive, and HDFS.
• Strong experience with Apache Kafka, including topics, partitions, consumer groups, offsets, and message delivery.
• Hands-on experience with Apache Airflow, including DAG development, scheduling, dependencies, retries, backfills, and monitoring.
• Strong programming skills in Python, SQL, and Scala.
• Experience with incremental imports, partitioning, bucketing, and large-scale data processing.
• Hands-on experience managing Spark jobs using spark-submit and monitoring production workloads.
• Experience with AWS, particularly S3, EMR, Glue, EC2, and IAM.
• Experience with Docker for packaging and deploying data workloads.
• Understanding of data modelling, schema evolution, and data formats such as Parquet, ORC, Avro, and JSON.
• Strong analytical, problem-solving, debugging, and data-quality mindset.

Key Responsibilities:

• Design, develop, and maintain scalable Big Data solutions using Apache Spark, PySpark, Scala Spark, Hive, Hadoop, HDFS, Kafka, and Airflow.
• Build and maintain batch and streaming data pipelines using PySpark, Scala, Python, and SQL.
• Process large volumes of structured, semi-structured, and streaming data efficiently.
• Build complex data processing pipelines and integrate data through REST APIs.
• Implement incremental data processing, partitioning, and bucketing to improve Spark performance and cost efficiency.
• Submit, monitor, tune, and troubleshoot Spark jobs, including resource sizing, executors, memory, and failure recovery.
• Read, process, and write data using Hive and Spark.
• Build Kafka-based pipelines for real-time and near-real-time data processing.
• Design and manage Airflow DAGs, scheduling, dependencies, retries, backfills, SLAs, and alerts.
• Deploy and run data workloads using AWS and Docker.
• Troubleshoot production pipeline failures and drive issues to root cause and closure.
• Improve pipeline performance, reliability, scalability, and data quality.
• Work closely with Product, Analytics, and Engineering teams to deliver accurate and timely data.
• Document data pipelines, data models, processes, and technical solutions.