Big Data Engineer

Vortalsoft Inc; New Jersey; On-site; USD 90,000 - 120,000; Full time

Job summary

Vortalsoft Inc is seeking a skilled data engineer to develop and maintain data pipelines for their Big Data financial crime solution. The ideal candidate will have at least 4 years of experience in Big Data solutions and strong skills in programming, data modeling, and ETL processes.

The candidate should possess a Bachelor's degree in a relevant field and the ability to adapt in a fast-paced, team-oriented environment. Familiarity with AWS and Big Data technologies like Hadoop and Spark is a plus.

Qualifications

• 4+ years of experience in designing, configuring, and implementing Big Data solutions.
• Strong understanding of data warehousing and ETL processes.
• Ability to manage multiple priorities and deadlines.

Responsibilities

• Create and maintain data pipelines that ensure security.
• Develop and standardize ETL processes for reporting.
• Monitor and troubleshoot issues with Big Data solutions.

Skills

Java

Python

Scala

Big Data technologies

Data modeling

ETL processes

Problem-solving

Communication

Education

Bachelor's degree in computer science or related field

Tools

Hadoop

Spark

NoSQL databases

AWS

Job description

Responsibilities

• Create and maintain data pipelines that transfer data from select location (maintaining security protocols) into our Big Data financial crime solution.
• Develop, implement, and standardize ETL processes to transform and cleanse data for pipeline development and reporting.
• Develop and implement data models and schemas to support analytical applications and data visualization tools.
• Monitor and troubleshoot issues with our proprietary Big Data financial crime solution and provide timely resolution.
• Develop and implement security and data privacy measures to protect sensitive data.
• Develop and maintain documentation and training materials for end-users and stakeholders.
• Keep up to date with emerging trends and technologies in Big Data, data analytics, and data engineering.

Requirements

• Bachelor s degree in computer science, Engineering, or related field.
• 4+ years of experience in designing, configuring, and implementing Big Data solutions, preferably with financial crime solutions.
• Strong programming skills in Java, Python, or Scala.
• Strong understanding of data modelling, data warehousing, and ETL processes.
• Strong understanding of Big Data technologies and their architectures, including Hadoop, Spark, and NoSQL databases.
• Strong analytical and problem-solving skills.
• Excellent communication and interpersonal skills.
• Ability to work in a fast-paced, team-oriented environment.
• Ability to manage multiple priorities and deadlines.
• Experience with AWS is a plus.