Big Data Developer

Metaplore Solutions Pvt Ltd; Bengaluru; On-site; INR 4,500,000 - 7,500,000; Full time

Job summary

Metaplore Solutions Pvt Ltd in Bengaluru seeks a Senior Data Engineer to design, build, and scale distributed data pipelines and platforms end-to-end for analytics and downstream applications.

You will own ingestion, processing, and storage, leveraging Spark, Kafka, Hadoop/HDFS, Hive, and Databricks across Azure, AWS, and GCP to ensure data quality and performance. Collaboration with data scientists and engineers is essential.

Qualifications

• 6–10 years of data engineering or distributed systems experience.
• Proficient in Scala and Java; Python knowledge; C++ a plus.
• Hands-on Spark (Scala-based) experience for large-scale data processing.
• Experience with Kafka and Kafka Connect for real-time streaming.
• Hadoop/HDFS and Hive experience for data warehousing.
• Databricks experience for data engineering and pipeline orchestration.
• Strong MySQL SQL skills.
• Experience deploying data platforms on Azure, AWS, or GCP.
• Proficient with CI/CD and Git.
• Proficient with development tooling (IntelliJ, Eclipse, PyCharm, VS Code, Vim).

Responsibilities

• Design, build, and maintain large-scale batch and streaming data pipelines.
• Develop and optimize distributed data processing jobs in Scala and Python.
• Build data ingestion pipelines with Kafka Connect and Hadoop/HDFS.
• Design warehousing with Hive and manage Databricks workflows.
• Write production-grade code in Scala, Java, Python, and C++ where needed.
• Design relational data models and queries in MySQL.
• Deploy data solutions across Azure, AWS, and GCP.
• Monitor, test, and validate data quality and performance.
• Collaborate with data scientists, analysts, and engineers to meet data needs.
• Mentor junior data engineers and promote best practices.

Skills

Scala

Java

Python

C++

Distributed systems

SQL

Data engineering

Cloud platforms

Communication

Problem solving

Tools

Apache Spark

Apache Kafka

Kafka Connect

Hadoop/HDFS

Hive

Databricks

MySQL

Git

IntelliJ

Eclipse

PyCharm

VS Code

VIM

Job description

We're looking for a Senior Data Engineer to design, build, and scale distributed data pipelines and platforms. You'll own critical data infrastructure end-to-end — from ingestion to processing to storage — enabling reliable, high-quality data for analytics and downstream applications across the organization.

Responsibilities

• Design, build, and maintain large-scale batch and streaming data pipelines using Apache Spark and Kafka
• Develop and optimize distributed data processing jobs in Scala and Python
• Build and manage data ingestion pipelines using Kafka Connect and Hadoop/HDFS
• Design and maintain data warehousing solutions using Hive
• Develop, tune, and manage workflows on Databricks
• Write efficient, production-grade code in Scala, Java, Python, and where needed, C++
• Design and optimize relational data models and queries in MySQL
• Architect and deploy data solutions across cloud platforms (Azure, AWS, GCP)
• Ensure data quality, reliability, and performance through monitoring, testing, and validation frameworks
• Collaborate with data scientists, analysts, and backend engineers to understand data requirements and deliver scalable solutions
• Troubleshoot and resolve performance bottlenecks in large-scale distributed systems
• Contribute to architectural decisions around data platform design and technology selection
• Mentor junior data engineers and promote best practices in data engineering

Requirements

• 6–10 years of experience in data engineering or distributed systems development
• Strong programming proficiency in Scala and Java; working knowledge of Python; C++ a plus
• Hands-on expertise with Apache Spark (Scala-based) for large-scale data processing
• Strong experience with Apache Kafka and Kafka Connect for real-time data streaming and integration
• Solid experience with Hadoop/HDFS ecosystem and Hive for data warehousing
• Hands-on experience with Databricks for data engineering and pipeline orchestration
• Strong SQL skills and experience with MySQL for relational data management
• Experience deploying and managing data platforms on at least one major cloud provider (Azure, AWS, or GCP); multi-cloud experience preferred
• Proficiency with standard development tooling (IntelliJ, Eclipse, PyCharm, VS Code, VIM)
• Strong understanding of distributed systems concepts, data partitioning, and performance optimization
• Experience with CI/CD practices and version control (Git) for data pipeline deployment
• Excellent problem-solving skills and ability to work with large, complex datasets
• Strong communication skills and ability to collaborate across engineering, analytics, and business teams

Nice to Have

• Experience with data orchestration tools (Airflow, Databricks Workflows, etc.)
• Familiarity with data governance, lineage, and cataloging tools
• Exposure to machine learning pipelines or MLOps
• Experience with infrastructure-as-code (Terraform, CloudFormation)