Senior Software Engineer - Data Engineer: Python/Scala Developer with Spark and Snowflake

Capgemini; Brampton; On-site; CAD 85,000 - 110,000; Full time

Job summary

A global technology consulting firm is seeking a Senior Software Engineer specialized in data engineering, particularly with expertise in Python/Scala and big data technologies like Spark and Snowflake. The ideal candidate will have 7-10 years of experience, strong analytical skills, and a Bachelor's degree. This role involves implementing AWS Big Data solutions while leveraging CI/CD practices. This firm values diversity and offers opportunities for growth in a supportive environment.

Qualifications

• 7-10 years of experience, with at least 3 years in a relevant role.
• Certification: Should have SE Level 1 and seeking Level 2.
• Excellent analytical and problem-solving skills.

Responsibilities

• Implement AWS Big Data Lake using EMR and Spark.
• Develop data pipelines using Python, Scala, SQL.
• Deploy real-time data streams in a cloud-native infrastructure.

Skills

Business Analysis

Software Engineering

Data Management

Technical Solution Design

Education

Bachelor’s degree

Tools

Spark

Snowflake

AWS

GitLab

Job description

Job Description

Senior Software Engineer – Data Engineer: Python/Scala Developer with Spark and Snowflake (053491)

Responsibilities

• Hands‑on experience implementing an AWS Big Data Lake using EMR and Spark.
• Working experience with Spark, Hive, message queue or Pub/Sub, and streaming technologies (3+ years).
• Experience developing data pipelines using languages such as Python, Scala, SQL and open‑source frameworks for data ingest, processing, and analytics.
• Leveraging open‑source big data processing frameworks such as Apache Spark, Hadoop, and streaming technologies such as Kafka.
• Hands‑on experience with newer technologies relevant to the data space such as Spark, Airflow, Apache Druid, Snowflake (or any other OLAP database).
• Developing and deploying data pipelines and real‑time data streams within a cloud‑native infrastructure, preferably AWS.
• Using CI/CD pipelines (GitLab).
• Implementing code quality tools such as PEP8/Pylint or other code quality tools.
• Experience with Python plugins/operators such as FTP sensor, Oracle operator, etc.
• Implementing industry standards/best practices.
• Excellent analytical and problem‑solving skills.
• Excellent verbal and written communication skills.
• Flexibility to work across this delivery landscape which includes and is not limited to Agile application development, support and deployment.

Qualifications

• 7‑10 years of experience (3 years minimum relevant experience in the role) and a Bachelor’s degree.
• Certification: Should have SE Level 1 and seeking Level 2.
• Proficient in Business Analysis, Business Knowledge, Software Engineering, Testing, Data Management, Architecture Knowledge, and Technical Solution Design.

Legal and Equal Opportunity Statement

Applicants for employment in the US must have valid work authorization that does not now and/or will not in the future require sponsorship of a visa for employment authorization in the US by Capgemini.

Capgemini is an Equal Opportunity Employer encouraging diversity in the workplace. All qualified applicants will receive consideration for employment without regard to race, national origin, gender identity or expression, age, religion, disability, sexual orientation, genetics, veteran status, marital status or any other characteristic protected by law.

This is a general description of the duties, responsibilities and qualifications required for this position. Physical, mental, sensory or environmental demands may be referenced to communicate the manner in which this position is traditionally performed. Whenever necessary, Capgemini will consider reasonable accommodations that might involve varying job requirements and/or changing the way this job is performed, provided that such accommodations do not pose an undue hardship.

Capgemini is a global leader in consulting, digital transformation, technology and engineering services. The Group is at the forefront of innovation to address the entire breadth of clients’ opportunities in the evolving world of cloud, digital and platforms. Building on its strong 50‑year+ heritage and deep industry‑specific expertise, Capgemini enables organizations to realize their business ambitions through an array of services from strategy to operations. Capgemini is driven by the conviction that the business value of technology comes from and through people. Today, it is a multicultural company of 270,000 team members in almost 50 countries. With Altran, the Group reported 2019 combined revenues of €17 billion.