Big Data Developer

Unify Technologies; Hyderabad; On-site; INR 1,200,000 - 1,800,000; Full time

Job summary

Unify Technologies in Hyderabad is looking for a Senior Big Data Engineer. This role entails creating and maintaining Scala/Spark jobs for complex data transformation and aggregation. You will collaborate with cross-functional teams and develop data processing pipelines while ensuring best practices in data management and security.

The ideal candidate has a minimum of 7 years of experience with Big Data and Hadoop platforms, proficient in Spark and data transformation techniques. A focus on problem-solving and programming skills in Scala or Java is essential.

Qualifications

• Minimum 7 years of experience in Big Data and Hadoop platforms.
• Experience developing Spark Applications using multiple Spark APIs.
• Solid understanding of data structures and algorithms.
• Hands-on experience with programming languages like Scala or Java.

Responsibilities

• Create and maintain Scala/Spark jobs for data transformation.
• Produce unit tests for Spark transformations.
• Develop data processing pipelines and management architecture.
• Work collaboratively with infrastructure teams for complex analysis.

Skills

Spark Applications

Hadoop

Data Structures

Scala

Java

Python

Big Data Platforms

Data Transformation

Problem-solving

Tools

HDFS

AWS S3

Apache Kafka

MongoDB

Job description

Responsibilities

• Create new and maintain existing Scala/Spark jobs for data transformation and aggregation from simple to complex data transformations involving structured and unstructured data.
• Produce unit tests for Spark transformations and helper performance optimisation methods.
• Develop data processing pipelines, data storage, and management architecture.
• Define scalable calculation logic for interactive and batch use cases.
• Interact with infrastructure and data teams to produce complex analysis across data.
• Work on a unique and challenging big data ecosystem with a focus on storage efficiency, data security and privacy, scalable and performant queries, expandability and flexibility, etc., and help better measure the quality of map data.
• Work with engineers to build a big data platform that processes and manages exabytes of data and enables efficient access to that data.

Requirements

• Minimum 7 years of working experience in Big Data and Hadoop platforms, and building large-scale distributed systems with high availability.
• Experience in developing Spark Applications using Spark RDD API, Hadoop, Spark SQL, Spark Streaming API, Spark MLlib API, and DataFrames APIs.
• Should have broad knowledge of Spark advantages, Spark workflows, how to write Spark jobs, Spark query tuning and performance optimisation.
• Solid in data structures and algorithm basics.
• Should have good hands-on experience with any one programming language (Scala/Java 8 - 1st preference, OR Python - 2nd preference OR Java - 3rd preference).
• Strong investigative and problem-solving skills.
• Data ingestion, optimisation techniques, data transformation and aggregation pipeline design/development knowledge are required.
• Experience working on cutting-edge Big Data storage systems and technologies like Hadoop, HDFS, AWS S3 AWS Lambda, Storm/Heron, Cassandra, Apache Kafka, Solr/ElasticSearch, MongoDB, DynamoDB, Postgres, MySQL, etc.