RQ11576 - DataOps/Cloud Data Engineer - Senior

Flexstaf IT; Toronto; On-site; CAD 90,000 - 130,000; Full time

Job summary

Flexstaf IT seeks a Data Engineer to design, build and operate cloud-based data platforms and pipelines using Databricks, Delta Lake and Databricks SQL within a modern DataOps framework. You will implement metadata, lineage and data quality capabilities across Azure Data Services, with a focus on governance and scalable data movement.

The role emphasizes collaboration with architects, governance teams and data modellers to operationalize data standards and data models, while enabling data

Qualifications

• Demonstrated experience designing, implementing and supporting cloud-based data platforms and services.
• Experience developing scalable ETL/ELT processes, data pipelines and integration solutions.
• Experience implementing DataOps practices including deployment automation, testing, monitoring and operational support.
• Experience with cloud-native data engineering and modern data platform architectures.
• Experience implementing metadata, lineage and data quality capabilities within cloud data environments.
• Experience developing integration solutions using APIs, event-driven architectures and batch or streaming technologies.
• Knowledge of data governance, security and operational best practices for cloud platforms.
• Experience automating metadata collection, data movement and operational processes.
• Experience supporting cloud modernization and data migration initiatives.
• Experience collaborating with architects, modellers and governance teams to operationalize data standards, metadata and common data models.
• Knowledge of performance optimization and scalability considerations for cloud data platforms.
• Ability to troubleshoot complex technical problems and develop practical solutions.
• Strong analytical, organizational and communication skills.

Responsibilities

• Data pipelines and integration services
• Automated metadata collection and synchronization processes
• Cloud data platform implementation and configuration
• Data quality and lineage enablement
• CI/CD and DataOps automation
• Data migration and interoperability services
• Cloud-hosted data products and reusable data services
• Metadata and governance platform integrations

Skills

Databricks

Delta Lake

Databricks SQL

Medallion Architecture

Azure Data Services

Azure Data Factory

Informatica

Metadata repositories

DataOps

APIs

Event-driven architectures

Cloud data platforms

Data governance

Data quality

CI/CD

Data migration

Tools

Microsoft Purview

Microsoft Fabric

Azure Data Factory

Job description

HM is looking for a Data Engineer - with the tech stack below:

Must Have:

• Experience with Databricks, Delta Lake and Databricks SQL.
• Knowledge of Medallion Architecture.
• Experience with Azure Data Services/Azure Data Factory
• Experience with Informatica or equivalent integration platforms.
• Experience operationalizing common data models and metadata repositories.

Experience and Skill Set Requirements

General Skills (80%)

• Demonstrated experience designing, implementing and supporting cloud-based data platforms and services.
• Experience developing scalable ETL/ELT processes, data pipelines and integration solutions.
• Experience implementing DataOps practices including deployment automation, testing, monitoring and operational support.
• Experience with cloud-native data engineering and modern data platform architectures.
• Experience implementing metadata, lineage and data quality capabilities within cloud data environments.
• Experience developing integration solutions using APIs, event-driven architectures and batch or streaming technologies.
• Knowledge of data governance, security and operational best practices for cloud platforms.
• Experience automating metadata collection, data movement and operational processes.
• Experience supporting cloud modernization and data migration initiatives.
• Experience collaborating with architects, modellers and governance teams to operationalize data standards, metadata and common data models.
• Knowledge of performance optimization and scalability considerations for cloud data platforms.
• Ability to troubleshoot complex technical problems and develop practical solutions.
• Strong analytical, organizational and communication skills.

Resource will support the following deliverables:

• Data pipelines and integration services
• Automated metadata collection and synchronization processes
• Cloud data platform implementation and configuration
• Data quality and lineage enablement
• CI/CD and DataOps automation
• Data migration and interoperability services
• Cloud-hosted data products and reusable data services
• Metadata and governance platform integrations

Desirable Skills (20%)

• Experience with Databricks, Delta Lake and Databricks SQL.
• Knowledge of Medallion Architecture.
• Experience with Microsoft Fabric and Azure Data Services.
• Experience with Microsoft Purview.
• Experience with Informatica or equivalent integration platforms.
• Experience operationalizing common data models and metadata repositories.
• Experience with graph technologies and semantic repositories.
• Knowledge and understanding of AODA and related standards.
• Ability to meet tight deadlines and adapt to changing project priorities.

Nice to have:

• Prior OPS/Public sector exp.
• Experience with Microsoft Purview and Microsoft Fabric