Data Engineer

ProViso Staffing; Ontario; On-site; CAD 90,000 - 120,000; Full time

Job summary

A leading staffing company is seeking a Senior Developer to lead technical projects, ensuring adherence to best practices while utilizing big data technologies like Scala, Spark, and Akka. The role involves coding, testing, and mentoring junior developers, with a focus on delivering high-quality solutions and documentation. Ideal candidates should have strong experience with Java and SQL, and familiarity with cloud platforms is a plus.

Qualifications

• Hands-on experience with big data technologies.
• Proficient in Java and SQL with strong understanding of ETL processes.
• Experience with distributed systems using Akka.

Responsibilities

• Lead technical aspects of projects from inception to deployment.
• Provide detailed development estimates and track defects.
• Mentor junior developers and ensure knowledge transfer.

Skills

Scala

Spark

Akka

Java

SQL

Tools

Hadoop

Kafka

Hive

HBase

C#

Python

Azure ADLS

Google BigQuery

Databricks

Dremio

Job description

Summary of day-to-day responsibilities :

• Reviews, analyzes, and modifies programming systems including encoding, testing, debugging, and documenting programs.
• Familiar with a variety of the field’s concepts, practices, and procedures.
• Relies on experience and judgment to plan and accomplish goals.
• Performs a variety of complicated tasks.
• May lead and direct the work of others.
• May report directly to a project lead or manager.
• A wide degree of creativity and latitude is expected.
• A vacancy exists for a senior developer to join the team in the delivery and development of this project and to eventually contribute to future enhancement, support, and maintenance post-delivery.
• Prime responsibility will be to develop detailed design specifications, create and review code, and assess the impact on other applications.
• Role will work closely with other team members to ensure consistency in coding standards and approaches and to optimize any synergy and re-use opportunities.

Key accountabilities are :

• Responsible for leading technical aspects of the project from inception through to deployment.
• Ensure that solutions adhere to development best practices and architectural principles.
• Apply analytical skills to resolve problems and deliver viable solutions.
• Anticipate obstacles and execute proactive approaches to issues.
• Active participation in issue resolution.
• Provide detailed development estimates to Project Manager for development effort as well as weekly status reporting and budget forecasting.
• Configuration management for the sub projects within the release.
• Responsible for coding, testing, packaging, deployment, and support of builds as code is promoted from development, system, acceptance, and production environments.
• May lead code reviews.
• Participate in the development of test strategy, test plans, test cases and testing efforts (unit testing and integration testing of code).
• Provide technical support to business and testing team during testing phases.
• Track defects, defect triage, and resolution.
• Data set up and environment shake down. Coaching / mentoring of junior developers.
• Knowledge transfer within the group.
• Ensure required documentation is housed in the project repository and that knowledge transfer has occurred to full time staff upon completion of the contract term.

Must have :

• Maintain a real-time data processing solution built on Scala, Spark, and Akka framework.
• Hands-on experience with big data technologies such as Hadoop, Kafka, Hive, and HBase.
• Hands-on experience with the Akka framework for building distributed systems.
• Proficient in Java.
• Understanding of data communication protocols like REST and GRPC.
• Strong understanding of data warehousing concepts and ETL processes.
• Proficiency in SQL and experience with relational and NoSQL databases.

Nice To Have :

• Knowledge of Capital Markets.
• Proficiency with C# for Excel add-ins.
• Experience with cloud data platforms like Azure ADLS, Google BigQuery, and Databricks, Dremio or equivalent.
• Proficient in Python.
• Experience with CI / CD pipelines and infrastructure as code (IaaC).
• Familiarity with machine learning and data science workflows.
• Experience with real-time analytics and event-driven architectures.