Hadoop Scala Developer

Envision Technology Solutions; Charlotte (NC); On-site; USD 110,000 - 150,000; Full time

Job summary

Envision Technology Solutions is seeking a Big Data Engineer in Charlotte to design and maintain large-scale Hadoop-based pipelines. You will build data ingestion and processing frameworks using Scala, Spark, and Hadoop, and automate deployments with Shell scripts.

The role also involves developing robust RESTful services with Java and Spring Boot, optimizing Spark jobs for batch and real-time workloads, and working with HDFS, Hive, Sqoop, Kafka, and Oozie within a collaborative, enterprise

Qualifications

• Design, develop and maintain large-scale Big Data solutions using Hadoop ecosystem technologies.
• Develop data ingestion, transformation and processing pipelines with Scala, Spark and Hadoop.
• Create and maintain Shell scripts for automation and deployment.
• Build scalable backend services with Java and Spring Boot.
• Develop and optimize Spark jobs for batch and real-time processing.
• Work with HDFS, Hive, Sqoop, Kafka and Oozie components.

Responsibilities

• Design, develop, and maintain large-scale Big Data solutions.
• Develop data ingestion, transformation, and processing frameworks.
• Create and maintain Shell scripts for automation and operations.
• Build backend services using Java and Spring Boot.
• Optimize Spark applications for batch and real-time processing.
• Collaborate with analysts, architects, DevOps, and stakeholders.

Skills

Hadoop

Scala

Shell scripting

Java

Spring Boot

Tools

Spark

Hive

Sqoop

Kafka

Oozie

HDFS

Job description

Primary Skill: Hadoop, Scala, Shell scripting

Secondary: Java, Spring Boot

• Design, develop, and maintain large-scale Big Data solutions using Hadoop ecosystem technologies. Develop data ingestion, transformation, and processing frameworks using Scala, Spark, and Hadoop.
• Create and maintain Shell scripts for automation, monitoring, deployment, and operational support.
• Build reusable and scalable backend services using Java and Spring Boot.
• Develop and optimize Spark applications for batch and real-time data processing.
• Design and implement RESTful APIs and microservices using Spring Boot.
• Work with HDFS, Hive, Sqoop, Kafka, Oozie, and other Hadoop ecosystem components.
• Perform performance tuning and optimization of Hadoop and Spark jobs.
• Troubleshoot production issues and provide L2/L3 support for critical applications.
• Collaborate with business analysts, architects, DevOps teams, and other stakeholders.
• Participate in code reviews, unit testing, CI/CD implementation, and release activities.
• Ensure compliance with enterprise security, governance, and data quality standards.