DataOps Engineer — Build Scalable ETL Pipelines

Citylogix; Remote; CAD 85,000 - 120,000; Full time

Job summary

A data analytics firm in Canada is seeking a DataOps Engineer to develop sophisticated data processing pipelines and ensure robust performance. The successful candidate will design scalable ETL/ELT workflows using big data frameworks like Spark and Hadoop, along with workflow orchestration tools such as Airflow or Prefect. This role offers the opportunity to integrate diverse data sources and create monitoring solutions within a dynamic and collaborative environment.

Qualifications

• 3+ years building production data pipelines.
• Expertise in big data frameworks such as Spark and Hadoop.
• Experience with workflow orchestration tools like Airflow or Prefect.

Responsibilities

• Implement orchestration with Airflow/Prefect.
• Build reusable components and shared libraries.
• Optimize processing performance and costs.
• Manage schema evolution and data versioning.
• Integrate diverse data sources and formats.
• Create monitoring dashboards and metrics.

Skills

Building production data pipelines

Expertise in big data frameworks

Experience with workflow orchestration tools

Tools

Apache Spark

Hadoop

Airflow

Prefect

Job description

A data analytics firm in Canada is seeking a DataOps Engineer to develop sophisticated data processing pipelines and ensure robust performance. The successful candidate will design scalable ETL/ELT workflows using big data frameworks like Spark and Hadoop, along with workflow orchestration tools such as Airflow or Prefect. This role offers the opportunity to integrate diverse data sources and create monitoring solutions within a dynamic and collaborative environment.