Unify Technologies - Big Data Developer - Hadoop/Spark

Unify Technologies; Hyderabad; On-site; INR 1,200,000 - 1,800,000; Full time

Job summary

Unify Technologies in Hyderabad is looking for an experienced Big Data Engineer with strong Scala/Spark skills. The role involves creating and maintaining data transformation jobs and developing processing pipelines for a big data ecosystem.

The ideal candidate will have a minimum of 7 years in Big Data, a solid foundation in data structure and algorithms, and hands-on experience with multiple programming languages and big data storage systems.

Qualifications

• Minimum 7 years of working experience in Big Data and Hadoop platforms.
• Hands-on experience with programming languages like Scala or Java 8.
• Experience with data ingestion and pipeline design/development.

Responsibilities

• Create and maintain Scala/Spark jobs for data transformations.
• Develop data processing pipelines and architecture.
• Interact with teams for complex data analysis.

Skills

Spark Applications development

Data transformation techniques

Problem-solving skills

Strong knowledge of Hadoop

Tools

Scala

Java 8

Hadoop

AWS S3

Kafka

MongoDB

Job description

Responsibilities

• Create new and maintain existing Scala/Spark jobs for data transformation and aggregation from simple to complex data transformations involving structured and unstructured data.
• Produce unit tests for Spark transformations and helper performance optimisation methods.
• Develop data processing pipelines, data storage, and management architecture.
• Define scalable calculation logic for interactive and batch use cases.
• Interact with infrastructure and data teams to produce complex analysis across data.
• Work on a unique and challenging big data ecosystem with a focus on storage efficiency, data security and privacy, scalable and performant queries, expandability and flexibility, to help measure the quality of map data.
• Work with engineers to build a big data platform that processes and manages exabytes of data and enables efficient access to that data.

Requirements

• Minimum 7 years of working experience in Big Data and Hadoop platforms, and building large-scale distributed systems with high availability.
• Experience in developing Spark Applications using Spark RDD API, Hadoop, Spark SQL, Spark Streaming API, Spark MLlib API, and DataFrames APIs.
• Broad knowledge of Spark advantages, Spark workflows, how to write Spark jobs, Spark query tuning and performance optimisation.
• Solid foundation in data structures and algorithm basics.
• Hands-on experience with at least one programming language (Scala/Java 8 preferred, Python as a secondary option, Java as a tertiary option).
• Strong investigative and problem-solving skills.
• Data ingestion, optimisation techniques, data transformation and aggregation pipeline design/development knowledge.
• Experience with cutting-edge Big Data storage systems and technologies such as Hadoop, HDFS, AWS S3, AWS Lambda, Storm/Heron, Cassandra, Apache Kafka, Solr/Elasticsearch, MongoDB, DynamoDB, PostgreSQL, MySQL, etc.