Data Engineer Lead - GCP

Scotiabank; Canada; Hybrid; CAD 100,000 - 140,000; Full time

Job summary

A leading financial institution seeks a Lead Data Engineer to join their Wealth Data engineering team. The position involves designing and implementing advanced data solutions, optimizing performance for large-scale applications, and leveraging Google Cloud and Hadoop technologies. Candidates should have extensive experience in data engineering and performance optimization, along with strong problem-solving and communication skills. The role supports a hybrid working environment located in Toronto, Ontario, within a culture that values diversity and inclusion.

Benefits

Inclusive culture emphasizing diversity, equity, and inclusion.

Accessible workplace with accommodations available.

Opportunities for upskilling and competitive rewards.

Dynamic work environment with community engagement programs.

Qualifications

• 10+ years in data engineering with strong performance optimization skills.
• Expertise in GCP data services and Hadoop technologies.
• Strong background in data streaming, Java microservices, and architecture design.

Responsibilities

• Lead development of data ingestion and transformation processes.
• Build and operationalize data platforms using GCP services.
• Guide data engineers in building robust data pipelines.

Skills

Performance optimization

Data ingestion patterns

Problem-solving

Communication

Tools

Hadoop

GCP

Terraform

Java

Scala

SQL

Kafka

Spark

Job description

Requisition ID: #

Join a purpose-driven, high-performing team committed to results in an inclusive culture.

The Wealth Data engineering team within Global Wealth Engineering (GWE) plays a key role in meeting operational data needs for Wealth Management stakeholders. The Lead Data Engineer will design and implement data solutions using Big Data, Hadoop (Cloudera), and Google Cloud, collaborating closely with enterprise data teams, architects, business analysts, and data engineers.

Key Responsibilities:

1. Lead development efforts in ingesting and transforming data from various sources, writing code, building scripts, and ensuring end-to-end delivery in the Enterprise Data Lake environment.
2. Build distributed, reliable, and scalable data pipelines for multiple data sources.
3. Design and operationalize data platforms using GCP services like DataProc, Dataflow, BigQuery, CloudSQL, CloudSpanner, and third-party tools such as Spark, DBT, Kafka, and Cloud Storage.
4. Implement data ingestion patterns supporting batch, streaming, and API interfaces for ingress and egress.
5. Guide and work hands-on with a team of data engineers, developing frameworks and custom code using best practices for high performance.
6. Design and build production data pipelines from ingestion to consumption using GCP services, Java, Python, Scala, BigQuery, and SQL.
7. Deploy streaming and batch jobs in GCP using Cloud Dataflow, Java/Python, and write data to BigQuery.
8. Manage data pipelines with workflow orchestration, task scheduling, and dependency management.
9. Provide technical guidance on leveraging Google Cloud to build effective solutions, applying cloud infrastructure and platform services creatively.
10. Implement application logging, notifications, job monitoring, and performance monitoring.

Required Skills and Experience:

• 10+ years in data engineering and performance optimization for large OLTP applications.
• Expertise with Hadoop HDFS, Hive, Pig, Flume, and Sqoop.
• Experience with GCP managed data services: DataProc, Dataflow, BigQuery/DBT, Cloud Spanner, Cloud SQL, Pub/Sub.
• Knowledge of Google Cloud Databases (SQL, Spanner, PostgreSQL).
• Proficiency in HQL, relational/NoSQL databases, data streaming (Kafka, Spark Streaming).
• Experience with Infrastructure as Code (Terraform).
• Java microservices, Spring Boot, RESTful services, and architecture design.
• Strong problem-solving and communication skills.

Nice-to-Have Skills:

Understanding of Wealth domain data and solutions.

Benefits and Culture:

• Inclusive culture emphasizing diversity, equity, and inclusion.
• Accessible workplace with accommodations available.
• Opportunities for upskilling, development, and competitive rewards.
• Dynamic work environment with community engagement programs.

Location:

Hybrid in Toronto, Ontario, Canada.

Scotiabank is committed to diversity and inclusion. Accommodations are available during the recruitment process. Only shortlisted candidates will be contacted.