Senior Python/PySpark Engineer for AWS Data Platform

Luxoft; Toronto; On-site; CAD 90,000 - 130,000; Full time

Job summary

Luxoft in Toronto is seeking a senior data engineer to support the Analytical Work Bench integration, including AWS-based data pipelines and Jupyter Lab IDE work. You will design and implement scalable data flows, collaborate with end users to gather requirements, and ensure code quality with standard testing practices.

Join a team that leverages PySpark, PyTest, and Pylint, while integrating with CI/CD tooling such as Jenkins and Urban Code to deliver robust, maintainable solutions in a

Qualifications

• Bachelor's degree in Engineering or Computer Science required or preferred.
• 8+ years of hands-on Python and PySpark development experience.
• Experience with AWS cloud services and AWS EMR for big data processing.
• Solid testing & quality tooling with PyTest and Pylint.
• Experience with CI/CD pipelines (Jenkins, Urban Code) and end‑to‑end automation.
• Strong ability to gather requirements and collaborate with end users.

Responsibilities

• Integrate large datasets into the research ecosystem via the resource platform.
• Develop and maintain data pipelines and PySpark workflows.
• Collaborate with stakeholders to understand complex business needs and translate them into technical solutions.

Skills

Python Core

PySpark

PyTest

Pylint

CI/CD collaboration

End-user communication

Education

Bachelor's degree in Engineering or Computer Science

Tools

AWS

AWS EMR

PyPI

Gulp

Jenkins

Urban Code

React

JupiterLab

Job description

Project description

The project involves the data engineering organization’s Analytical Work Bench , which is an integrated development environment. The objective is to rationalize and integrate AWB with another internally developed application. Additionally, the project includes creating an IDE based on the Jupyter Lab environment using custom code developed by Client.

Responsibilities

• - Integrate large datasets into the research ecosystem via the resource platform.

SKILLS

Must have

• A university degree in Engineering or Computer Science is preferred.Hard skills:
• Proven experience working with cloud-based technologies, specifically AWS.
• Solid expertise in Big Data solutions utilizing AWS EMR.
• Advanced proficiency in Python Core and PySpark, with over 8 years of hands-on experience.
• Practical knowledge of Pylint and Pytest for code quality and testing, with at least two years of experience.
• Extensive experience with package management and automation tools such as PyPI and Gulp for over five years.
• Familiarity with continuous integration and continuous delivery pipelines using Jenkins and Urban Code, with more than two years of experience.Soft skills:
• Excellent communication skills demonstrated through collaboration with end users to gather and discuss complex business requirements effectively.

Nice to have

- UI development with React: Familiarity with building user interfaces using React is advantageous.- JupiterLab: Experience working with JupiterLab is a beneficial asset.- Data Engineering and/or Data Science experience: Having skills or experience in data engineering or data science is a plus.