Senior Hadoop Data Engineer: Spark, Kafka & Real-Time Pipelines

Cloudious LLC; Dallas (TX); On-site; USD 120,000 - 180,000; Full time

Job summary

Cloudious LLC in Dallas, TX is seeking an experienced Hadoop Data Engineer with a strong background in Java, Spring Boot, Spark, and Scala to design, build, and optimize scalable data processing solutions for batch and real-time pipelines.

The ideal candidate will work across Hadoop ecosystem components, implement RESTful data services, tune Hive queries, and collaborate with cross-functional teams to translate business requirements into robust data platforms.

Qualifications

• 10+ years hands-on Java development experience.
• Expertise in Spring Boot 2 framework.
• Proficient in Spark and Scala for big data processing.
• Strong Hive SQL optimization and Hadoop MapReduce knowledge.
• Experience with Kafka messaging and streaming.
• RESTful web services design and consumption.
• Version control with GitLab.
• Build tools Maven/Gradle; CI/CD experience.
• Understanding distributed systems and big data architectures.
• Experience with JD Edwards EnterpriseOne is a plus.

Responsibilities

• Design, develop, and maintain scalable data processing apps using Java and Spring Boot.
• Build and optimize big data pipelines with Spark and Scala for large-scale processing.
• Develop and consume RESTful web services for data integration.
• Implement batch and streaming pipelines with Spark Streaming and Kafka.
• Write optimized Hive queries and SQL for performance.
• Work on distributed data platforms leveraging Hadoop components.
• Collaborate with cross-functional teams to translate requirements into solutions.
• Ensure code quality via reviews, version control, and CI/CD practices.
• Troubleshoot performance, scalability, and data reliability issues.

Skills

Java 8+

Spring Boot 2

Spark

Scala

Kafka

RESTful services

GitLab

Maven/Gradle

CI/CD

Distributed systems

Big data architecture

Tools

GitLab

Maven/Gradle

Hive

SQL

Spark Streaming

Job description

Cloudious LLC in Dallas, TX is seeking an experienced Hadoop Data Engineer with a strong background in Java, Spring Boot, Spark, and Scala to design, build, and optimize scalable data processing solutions for batch and real-time pipelines.

The ideal candidate will work across Hadoop ecosystem components, implement RESTful data services, tune Hive queries, and collaborate with cross-functional teams to translate business requirements into robust data platforms.