Spark & Scala Big Data Engineer – Real-Time Hadoop

TechDigital Group; New Jersey; On-site; USD 80,000 - 130,000; Full time

Job summary

An established industry player is seeking a skilled Big Data specialist proficient in Scala and Spark. This exciting role involves leveraging your expertise in the Hadoop ecosystem to drive real-time data processing and analytics. You'll work in a dynamic environment where clear communication and problem-solving skills are essential. If you thrive in challenging situations and are passionate about Big Data technologies, this opportunity is perfect for you to make a significant impact on innovative projects.

Qualifications

• Experience in Big Data technologies and real-time data processing.
• Strong problem-solving and decision-making skills.

Responsibilities

• Work with Linux-based Hadoop ecosystem and demonstrate analytical skills.
• Deliver against agreed objectives while multi-tasking under pressure.

Skills

Spark

Scala

Hive & SQL

Hadoop

Communication

Tools

HDFS

Impala

HBase

Job description

Performs data collection, analysis, validation, cleansing, and reporting. Designs, codes, tests, debugs, and documents ETL processes, SQL queries, and stored procedures. Extracts and analyzes data from various sources, including databases, manual files, and external websites. Responds to data inquiries from various groups within an organization. Creates and publishes regularly scheduled and ad hoc reports. Documents reporting requirements and processes and validates data components as required. Requires experience with relational databases and knowledge of query tools and/or statistical software. Strong analytical and organizational skills also essential. Must possess expert level knowledge of MS Excel.

1. Explain what the scope of the work/project is? Project Overview, deliverables (what is this person responsible for).

This role is a Data Analyst supporting a large regulatory transformation project. The analyst will respond to discovery requests from multiple state commissions, pulling and validating data related to:

Customer accounts

Credit inquiries

Billing data

Usage data

The position supports all towers across the project, not just Customer Experience, and works closely with Compliance Analysts to ensure regulatory questions are answered accurately and efficiently

1. What are the top 3-5 skills and qualifications (technology/application/software, etc.) required?

Advanced Excel skills (pivot tables)

Experience with Power BI or similar BI tools

Data validation and quality assurance

Reporting and analytics experience

Familiarity with systems such as Salesforce

2. What non-technical skills are necessary (i.e., such as communication, problem solving, team player)?

Relationship management

Strong verbal and written communication

Ability to present data to a broad audience

Problem‐solving

Team collaboration

3. Ideal candidate background and how many years required?

Approximately 3–7 years of experience

Prior experience on large transformation projects preferred

4. Are there any specific companies you like to see on a candidate's resume?

N/a

5. Are there any certifications that the candidate must possess?

N/a

6. Are there any preferred or "nice to have " skills?

Previous utility industry experience

Experience working on regulatory or transformation projects

Strong analytical mindset

Ability to tell a clear story with data, not just produce outputs