Senior Data Engineer – IoT

Pearlfisher; Vancouver; Hybrid; CAD 120,000 - 180,000; Full time

Benefits offered by this job

RRSP matching

Unlimited vacation

Work-from-home flexibility

Spacious downtown Vancouver office

Job summary

Pearlfisher in Vancouver is seeking a senior Data Engineer to join a high-growth Clean Energy software team. You’ll design and build data pipelines, manage large time-series datasets, and deploy ML-enabled data intelligence in the cloud.

You will collaborate with data scientists and engineers to deliver scalable, real-time analytics using Kafka, Kinesis, Spark, and AWS services. A passion for data quality and shipping robust systems is essential.

Qualifications

• Bachelor’s Degree in Computer Science or related field.
• Experience building scalable infrastructure software or distributed systems for commercial online services.
• Production experience with orchestration and batch processing technologies like Apache Airflow.
• 3+ years of production experience with Python; Java is a plus.
• 3+ years of experience with large-scale data warehousing, mining or analytic systems.
• 3+ years of building big data pipelines using Kafka or Kinesis.
• 3+ years of Experience with AWS services (EC2, S3, Lambda, Kinesis, EMR, Redshift).
• Production experience with real-time streaming technologies (Flink, Spark, Samza).

Responsibilities

• Establish scalable, automated processes for large-scale data analyses, model development, validation and implementation.
• Productize ML applications.
• Propose and validate batch and real-time streaming solutions.
• Analyze and extract information from large streaming data to create an event-driven data pipeline.
• Collaborate with scientists and engineers to deploy new features.

Skills

Python

SQL

NoSQL

Data pipelines

Machine Learning

Leadership

AWS

Kubernetes

Education

Bachelor's degree in Computer Science or related field

Tools

Airflow

Kafka

Kinesis

Spark

Flink

Kubernetes

Job description

Are you interested in joining a high-growth company and software team in the Clean Energy industry? Do you want to work with massive amounts of real-time energy data, and the latest technologies in IoT, machine learning, big data, and SaaS?

Our client and it’s people help make the world safer, brighter and more productive. They are a team of builders and doers who share a passion for innovation and a desire to outpace their competition. The organization offers an inclusive workplace, empowers its employees to embrace diversity in all forms, celebrates differences, and treats everyone with equity and respect.

This is a great opportunity to join the client’s clean energy team in their mission to accelerate the adoption of renewable energy and create a more intelligent home!

The goal is to reduce global carbon emissions through the manufacturing and sale of renewable power backup systems. They are looking for Data Engineers to contribute to their technical vision and design and build their data intelligence/machine learning pipeline and cloud data store.

The company has recently experience explosive growth from its start-up origin and they need to scale up their technology while continuing to drive innovation of consumer power electronics. They are growing extremely quickly and looking to build out reliable, robust, IoT systems with feature-rich cloud services.

What will you do?

As a senior member of the team, you will have significant responsibility and influence in shaping its future direction. You will iterate quickly on all stages of data pipeline, extracting, transforming, loading data and scheduling tasks to bring data intelligence products to production.

You should have deep expertise in the design, creation, management, and business use of large datasets, across a variety of data platforms. You should have excellent business and interpersonal skills to be able to work with business owners to understand data requirements, and to build highly scalable time-series and data lake systems.

Successful candidates will have strong engineering and communication skills, as well as, a belief that data driven processes lead to great products. You will need to have a passion for quality and an ability to understand complex systems.

Above all you should be passionate about working with huge data sets and be someone who loves to bring datasets together to answer business questions and drive growth.

As part of this role, you will be required to:

• Establish scalable, efficient, automated processes for large scale data analyses, model development, validation and implementation,
• specifically productizing ML applications
• Propose and validate solutions for batch and real-time streaming solutions
• Analyze and extract relevant information from large amounts of streaming data to create an event driven data pipeline.
• Work closely with scientists and engineers to create and deploy new features

Ideal Skills

• Bachelor’s Degree in Computer Science or related field
• Experience building scalable infrastructure software or distributed systems for commercial online services
• Production experience with various orchestration, batch processing technologies like Apache Airflow.
• 3+ years of production experience with Python, experience in Java is a big plus.
• 3+ years of experience with large scale data warehousing, mining or analytic systems.
• 3+ years of Expertise in developing big data pipelines using technologies like Kafka or Kinesis
• 3+ years of Experience with various AWS services (EC2, S3, Lambda, Kinesis, EMR, Redshift)
• Production experience with various real time streaming technologies (Flink, Spark, Samza)

What else will you need to be successful?

• Background in Machine Learning and productizing Data Intelligence pipelines
• Strong knowledge of SQL and NoSQL data stores
• Experience with YARN, Kubernetes
• Strong communication skills and commitment to teamwork
• Sharp analytical abilities and proven design skills
• Strong sense of ownership, urgency, and drive
• Proven leadership abilities in an engineering environment in driving operational excellence and best practices
• Employer RRSP contribution matching
• Unlimited vacation (and they genuinely encourage you to use it)
• Work-from-home flexibility
• Spacious new office in the heart of downtown Vancouver