Specialist - Data Engineering

LTM; Mississauga; On-site; CAD 90,000 - 130,000; Full time

Job summary

LTM in Mississauga, Ontario is seeking a Specialist with 5 to 7 years of experience in Scala and Apache Spark to design, develop, and optimize scalable big data processing solutions within the Hadoop Spark SparkSQL ecosystem. You will design, implement, and optimize data pipelines, collaborating with data scientists, analysts, and architects to deliver robust data processing applications.

Responsibilities include leading development of big data solutions, optimizing Spark jobs for production,

Qualifications

• Design, develop and optimize scalable data processing applications using Scala and Apache Spark.
• Collaborate with cross-functional teams to build robust data pipelines.
• Analyze large datasets to extract actionable business insights.

Responsibilities

• Lead the development and deployment of big data solutions using Scala, Spark and SparkSQL.
• Collaborate with data scientists, analysts and architects to gather requirements.
• Optimize Spark jobs for performance and scalability in production environments.
• Mentor junior team members and provide technical guidance on best practices.
• Perform code reviews to ensure quality and adherence to standards.
• Monitor system performance and address bottlenecks or failures.
• Participate in sprint planning and agile ceremonies.

Skills

Scala

Apache Spark

SparkSQL

Big data

Hadoop ecosystem

Tools

Apache Spark

Hadoop

SparkSQL

Job description

Job Summary

Seeking a Specialist with 5 to 7 years of experience in Scala and Apache Spark to design develop and optimize scalable big data processing solutions within the Hadoop Spark SparkSQL ecosystem

Role Description

Job role : Spark and scala developer

Location : Mississauga ,Ontario

Job Summary

Seeking a Specialist with 5 to 7 years of experience in Scala and Apache Spark to design develop and optimize scalable big data processing solutions within the Hadoop Spark SparkSQL ecosystem

Job Description

Design develop and optimize scalable data processing applications using Scala and Apache Spark Collaborate with crossfunctional teams including data scientists analysts and architects to build and maintain robust data pipelines Analyze large and complex datasets to extract actionable insights supporting business objectives Develop highperformance applications in distributed computing environments leveraging SparkSQL Ensure code quality maintainability and adherence to Scala programming best practices Continuously evaluate and integrate emerging big data technologies and tools to enhance platform capabilities Troubleshoot and resolve complex data processing issues to maintain system reliability Document technical designs and implementation details for effective knowledge sharing

Roles and Responsibilities

• Lead the development and deployment of big data solutions leveraging Scala Apache Spark and SparkSQL
• Collaborate with stakeholders to gather requirements and deliver effective data solutions
• Optimize Spark jobs for performance and scalability in production environments
• Mentor junior team members and provide technical guidance on best practices
• Conduct code reviews to ensure adherence to coding standards and promote continuous improvement
• Monitor system performance proactively and address bottlenecks or failures
• Participate in sprint planning estimation and agile ceremonies to deliver highquality software
• Ensure compliance with data governance and security policies