Senior Hadoop Developer

The Timberline Group; St. Louis (MO); On-site; USD 120,000 - 190,000; Full time

Job summary

The Timberline Group seeks a Senior Hadoop Developer to design and optimize data processing applications within our Big Data stack. You will implement data analytics processing algorithms on batch and stream frameworks and collaborate with cross-functional teams to deliver scalable solutions.

Responsibilities include building and tuning Hadoop jobs, designing Hive/HBase schemas, developing Pig and Hive scripts, and deploying API services in Java Spring.

Qualifications

• Minimum requirements:

Responsibilities

• Implement data analytics processing algorithms on Big Data batch and stream processing frameworks (e.g. Hadoop MapReduce, Python, Spark, Scala, Kafka).
• Perform data acquisition, preparation, and analysis leveraging Spark with Scala.
• Load data from diverse datasets and select efficient file formats for tasks.
• Install, configure, and maintain enterprise Hadoop environment.
• Build distributed data pipelines to ingest and process data in real-time.
• Define Hadoop Job Flows and manage Hadoop jobs using Scheduler.
• Review and manage Hadoop log files.
• Design Hive/HBase schemas within HDFS and create tables with suitable formats.
• Mentor Big Data Developers on best practices.
• Develop Pig and Hive scripts with joins on datasets.
• Apply HDFS formats like Parquet, Avro for analytics speed.
• Fine-tune Hadoop applications for performance.
• Troubleshoot Hadoop ecosystem runtime issues.
• Develop and document data integration solutions (batch and real-time).
• Lead technical meetings and communicate clearly to technical and non-technical audiences.
• Implement Spark Streaming and integrate with JMS queue with custom receivers.
• Develop and deploy API services in Java Spring.

Skills

Hadoop

Python

Spark

Scala

Kafka

Java

Spring

Hive

HBase

Pig

JMS

Parquet

Avro

Tools

Spark Streaming

JMS Queue

Parquet/Avro formats

Job description

Senior Hadoop Developer to develop, create, and modify general computer applications software or specialized utility programs.
Job responsibilities and duties include:

• Implement data analytics processing algorithms on Big Data batch and stream processing frameworks (e.g. Hadoop MapReduce, Python, Spark, Scala, Kafka etc.).
• Perform data acquisition, preparation, and perform analysis leveraging a variety of data programming techniques in Spark using Scala.
• Work on complex issues where analysis of situations and data requires an in-depth evaluation of variable factors.
• Load data from different datasets and decide on which file format is efficient for a task. Hadoop Developers source large volumes of data from diverse data platforms into Hadoop platform.
• Install, configure, and maintain enterprise Hadoop environment.
• Build distributed, reliable, and scalable data pipelines to ingest and process data in real-time. Hadoop Developers deals with fetching impression streams, transaction behaviors, clickstream data, and other unstructured data.
• Define Hadoop Job Flows and manage Hadoop jobs using Scheduler.
• Review and manage Hadoop log files.
• Design and implement column family schemas of Hive and HBase within HDFS
• Assign schemas and create Hive tables with suitable formats and compression techniques.
• Mentor Big Data Developers on best practices and strategic development.
• Develop efficient Pig and Hive scripts with joins on datasets using various techniques.
• Apply different HDFS formats and structure like Parquet, Avro, etc. to speed up analytics.
• Fine tune Hadoop applications for high performance and throughput.
• Troubleshoot and debug any Hadoop ecosystem run time issues.
• Develop and document technical design specifications.
• Design and develop data integration solutions (batch and real-time) to support enterprise data platforms including Hadoop, RDBMS, and NoSQL.
• Lead technical meetings, as required, and convey ideas clearly and tailor communication based on selected audience (technical and non-technical).
• Implement Spark Streaming architecture and integration with JMS queue with custom receivers.
• Develop and deploy API services in Java Spring.
• Create Hive and HBase data source connection to Spring.
• Implement multi-threading in Java/Scala.

This position has no direct reports and does not supervise any other personnel.

Minimum requirements: