Big Data Consultant

Open Systems Technologies; Mississauga; Hybrid; CAD 80,000 - 120,000; Full time

Job summary

An innovative company is seeking a Big Data Consultant for a hybrid role in Mississauga. This exciting position involves translating high-level requirements into technical designs, developing scalable data ingestion pipelines, and optimizing distributed data processing systems. The ideal candidate will have extensive experience in Big Data technologies, including Apache Spark and Hadoop, and will play a key role in transforming data handling processes. If you are passionate about data and eager to contribute to impactful projects in the banking sector, this opportunity is perfect for you.

Qualifications

• 6 to 8 years of experience in Big Data development.
• Expertise in Apache Spark and distributed systems.

Responsibilities

• Translate requirements into technical design for data modeling.
• Develop data ingestion pipelines using Spark and Hadoop.
• Participate in UAT/SIT test cycles and resolve issues.

Skills

Apache Spark

Scala

Hadoop

Hive

Kafka

Job description

We are looking for a Big Data Consultant for a contract hybrid role in Mississauga. The ideal candidate will have extensive experience in large-scale data handling and expertise in Big Data transformation programs, from architecture to maintenance. Please apply with your updated resume and contact details.

Key Responsibilities:

1. Translate high-level and functional requirements, including data modeling (dimensional, semi-structured, transactional use cases), into technical design.
2. Develop batch and real-time data ingestion pipelines using technologies like Spark, Hadoop, Hive, and Middleware.
3. Develop programs to migrate historical data from legacy platforms to the Big Data platform.
4. Participate in UAT/SIT test cycles, release cycles, and resolve issues.
5. Perform code reviews and test case reviews to ensure functional and non-functional requirements are met.
6. Analyze platform and software version upgrades, and evaluate new tools and technologies for data management.

Required Skills:

1. 6 to 8 years of hands-on experience in Big Data development, focusing on Apache Spark, Scala, and distributed systems.
2. Experience designing, creating, and optimizing scalable distributed data processing pipelines using the Hadoop ecosystem, including Cloudera platform, Hive, HDFS, and YARN.
3. Experience with real-time data streaming platforms such as Apache Kafka or Spark Streaming.

Additional Details:

• Seniority Level: Mid-Senior level
• Employment Type: Contract
• Job Function: Information Technology
• Industries: Banking and Investment Banking