DATAOPS - Cloud Data Engineer

V-Soft Consulting; Toronto; On-site; CAD 90,000 - 120,000; Full time

Job summary

A leading IT consulting firm is seeking a Cloud Data Engineer in Toronto, ON. The role involves designing and optimizing data pipelines, ensuring effective data management, and collaborating in an agile environment. The ideal candidate should be fluent in Python, proficient in SQL, and experienced with cloud data platforms. This position offers competitive compensation and comprehensive benefits.

Benefits

Medical, Dental, Vision benefits

401(k) plan

Competitive compensation

Qualifications

• Demonstrated fluency in Python, with knowledge of its best practices.
• Understanding of iterative product development cycles.
• Experience with version-controlled, shared codebases using git.
• Proficiency in SQL and Python, with hands-on experience using Databricks.
• Experience with data pipeline and workflow development.

Responsibilities

• Design and develop data pipelines from source to end user.
• Optimize data pipelines using various tools and technologies.
• Ensure effective data management and data exchange.
• Collaborate in an agile, sprint-based development environment.

Skills

Fluency in Python

SQL proficiency

Experience with Git

Data pipeline development

Agile product development

Cloud data platforms

Tools

Databricks

Spark SQL

Job description

Overview

Cloud Data Engineer

Primary Location: Toronto, ON

V-Soft Consulting is currently hiring for a Cloud Data Engineer for our premier client in Toronto, ON.

What You’ll Do

Job Responsibilities

• Designing and developing data pipelines from source to end user
• Optimizing data pipelines General Skills
• Experience with Cloud data platforms, data management and data exchange tools and technologies
• Experience with Commercial and opensource data and database development and management and specializing in data storage setting and managing cloud Data As a Service (DaaS), application Database as a Service (DBaaS), Data Warehouses as a Service (DWaaS), and other storage platforms (both in the cloud and on-premise). Experience in DataOPS principle, best practice, and implementation and Agile project development and deployment
• Experience in Continuous Integration / Continuous Development / Deployment (CI / CD) and Data provisioning Automation
• Experience with digital product, data analysis, data exchange, data provisioning, and data security
• Extensive expert experience in designing / developing and implementing data conversation and migration of Very Large Data (VLD) of Online analytical processing (OLAP) and online transaction processing (OLTP) environments to Cloud Software-as-a-service (SaaS), Platform-as-a-service (PaaS) and Infrastructure-as-a-service (IaaS) environments.

Qualifications

• Must Have: Demonstrated fluency in Python, with knowledge of its best practices, coding conventions, and application in building robust, scalable data pipelines.
• Understanding and working knowledge of iterative product development cycles (Discovery, Agile, Beta, Live)
• Experience contributing to version-controlled, shared codebases using git (Azure DevOps, GitHub, Bitbucket) and participating in pull request code reviews.
• Experience in Continuous Integration / Continuous Development / Deployment (CI / CD) and Data provisioning Automation.
• Proficiency in SQL and Python, with hands-on experience using Databricks and Spark SQL for data modeling and transformation tasks.
• Experience with Data pipeline and workflow development, orchestration, deployment, and automation and specializing in programming and pipelines to create and manage the dataflow and movement of data.
• Experience with Cloud data platforms, data management and data exchange tools and technologies.
• Experience with structured, semi-structured, unstructured data collection, ingestion, provisioning and exchange technological development of enterprise data warehouse and data lake and data lakehouse solutions and operational support.

Cloud Knowledge and Experience (25%)

• Experience with Cloud data platforms, data management and data exchange tools and technologies.
• Experience with Commercial and open-source data and database development and management and specializing in data storage setting and managing cloud Data As a Service (DaaS), application Database as a Service (DBaaS), Data Warehouses as a Service (DWaaS), and other storage platforms (both in the cloud and on-premise).
• Experience managing Cloud Data services for project delivery, including storage, repositories, Data Lake, Data Lakehouse, key vault, virtual machine, disk, etc.

Agile Product Development (25%)

• Experience working in an agile, sprint-based development environment
• Understanding and working knowledge of iterative product development cycles (Discovery, Agile, Beta, Live)
• Experience collaborating and sharing tasks with multiple developers on complex data product deliveries
• Experience contributing to version-controlled, shared codebases using git (Azure DevOps, GitHub, Bitbucket) and participating in pull request code reviews.
• Experience in Continuous Integration / Continuous Development / Deployment (CI / CD) and Data provisioning Automation.
• Demonstrated experience in defining and executing tests across the development lifecycle (unit testing, system testing, user acceptance testing) and using results to refine database / store design.

V-Soft Consulting Group is recognized among the top 100 fastest growing staffing companies in North America. V-Soft Consulting Group is headquartered in Louisville, KY with strategic locations in India, Canada and the U.S. V-Soft is known as an agile, innovative technology services company with awards and partnerships across diverse technology stacks.

As a valued V-Soft Consultant, you’re eligible for full benefits (Medical, Dental, Vision), a 401(k) plan, competitive compensation and more. V-Soft is partnered with numerous Fortune 500 companies, exceptionally positioned to advance your career growth.

EEO Statement: V-Soft is an equal opportunity employer. All qualified applicants will receive consideration for employment without regard to race, color, religion, age, sex, national origin, disability status, genetics, protected veteran status, sexual orientation, gender identity or expression, or any other characteristic protected by federal, state or local laws.

LI-LP1