Big Data Developer

Cogency; Toronto; Hybrid; CAD 110,000 - 150,000; Full time

Benefits offered by this job

Hybrid work model

Job summary

Cogency Inc. is seeking a skilled Big Data Developer/Data Engineer to design and optimize scalable data pipelines using Spark, Snowflake, AWS, and Airflow. The role involves delivering robust data solutions to support analytics and BI in an agile environment.

You will collaborate with cross-functional teams, ensure data quality and governance, and contribute to code reviews, testing, and production support. Hybrid onsite work applies in Toronto, Canada.

Qualifications

• 5+ years of experience in Data Engineering, Big Data, or ETL development.
• Strong hands-on experience with Hadoop, Hive, Snowflake, Spark, and Airflow.
• Excellent SQL and data modeling skills; experience with APIs and data integrations.

Responsibilities

• Design, develop, and maintain scalable data ingestion pipelines and ETL workflows.
• Build and optimize data pipelines, transformation frameworks, and processing workflows.
• Develop and optimize Apache Spark applications for large-scale data processing.
• Work with Hadoop, Spark, and Hive within Big Data environments.
• Design and optimize Snowflake data solutions, data models, and SQL workloads.
• Develop complex SQL queries, stored procedures, transformations, and data models.
• Integrate Snowflake with enterprise data sources and BI/reporting platforms.
• Develop ETL workflows using Informatica, Talend, Apache Airflow, or similar technologies.
• Build and manage workflow orchestration using Apache Airflow.
• Develop data solutions on AWS, leveraging services such as S3, Glue, and Lambda.
• Monitor, troubleshoot, and resolve data pipeline and platform performance issues.
• Implement data quality, integrity, security, and governance controls.
• Develop APIs and data integrations using Scala or Java.
• Implement CI/CD, DevSecOps, and Infrastructure-as-Code practices.
• Maintain technical documentation for data pipelines, transformations, and data models.
• Collaborate with Data Architects, Developers, DevOps teams, Business Analysts, and other stakeholders.
• Participate in Agile ceremonies, code reviews, testing, deployment, and production support.
• Leverage GenAI and AI-assisted development tools to improve developer productivity and code quality.

Skills

5+ years experience

SQL

Data Modeling

Scala/Java

APIs & Data Integrations

CI/CD

Agile Delivery

Education

Bachelor's or Master's in CS/IT or related

Tools

Hadoop

Hive

Snowflake

Apache Airflow

Spark

Informatica

Talend

Job description

Job Title: Big Data Developer – Spark, AWS, Airflow & Snowflake

Company: Cogency Inc.

Work Model: Hybrid – 4 Days Onsite

Job Type: Full-Time

Job Summary

Cogency Inc. is seeking a skilled Big Data Developer / Data Engineer with strong expertise in Spark, Snowflake, AWS, Airflow, Big Data technologies, and ETL. The successful candidate will design, develop, and optimize scalable data pipelines and ETL processes while ensuring data quality, security, reliability, and performance.

The role involves working closely with cross-functional teams in an Agile environment to deliver robust data solutions supporting analytics, reporting, and business intelligence initiatives.

Key Responsibilities

• Design, develop, and maintain scalable data ingestion pipelines and ETL workflows.
• Build and optimize data pipelines, transformation frameworks, and processing workflows.
• Develop and optimize Apache Spark applications for large-scale data processing.
• Work with Hadoop, Spark, and Hive within Big Data environments.
• Design and optimize Snowflake data solutions, data models, and SQL workloads.
• Develop complex SQL queries, stored procedures, transformations, and data models.
• Integrate Snowflake with enterprise data sources and BI/reporting platforms.
• Develop ETL workflows using Informatica, Talend, Apache Airflow, or similar technologies.
• Build and manage workflow orchestration using Apache Airflow.
• Develop data solutions on AWS, leveraging services such as S3, Glue, and Lambda.
• Monitor, troubleshoot, and resolve data pipeline and platform performance issues.
• Implement data quality, integrity, security, and governance controls.
• Develop APIs and data integrations using Scala or Java.
• Implement CI/CD, DevSecOps, and Infrastructure-as-Code practices.
• Maintain technical documentation for data pipelines, transformations, and data models.
• Collaborate with Data Architects, Developers, DevOps teams, Business Analysts, and other stakeholders.
• Participate in Agile ceremonies, code reviews, testing, deployment, and production support.
• Leverage GenAI and AI-assisted development tools to improve developer productivity and code quality.

Required Skills & Experience

• 5+ years of experience in Data Engineering, Big Data, or ETL development.
• Strong hands-on experience with:
• Hadoop
• Hive
• Snowflake
• Strong SQL and data modeling skills.
• Programming experience in Scala or Java.
• Experience developing APIs and enterprise data integrations.
• Experience with ETL technologies such as Informatica, Talend, or Apache Airflow.
• Strong understanding of data ingestion, transformation, processing, and pipeline optimization.
• Experience working with cloud platforms, preferably AWS.
• Understanding of CI/CD, DevSecOps, and Infrastructure-as-Code practices.
• Experience working in Agile delivery environments.
• Strong troubleshooting, analytical, and problem-solving skills.
• Excellent communication and collaboration skills.

Preferred / Nice-to-Have

• Hands-on experience with AWS Glue, S3, Lambda, and other AWS data services.
• Strong experience with Apache Airflow or similar orchestration platforms.
• Experience with GitHub Actions, Git, and automated testing.
• Knowledge of Python or other scripting languages.
• Experience with Shell/Bash scripting.
• Exposure to Docker, Kubernetes, or OpenShift.
• Experience with Infrastructure-as-Code tools such as Terraform.
• Experience with GenAI tools for code generation, code review, and developer productivity.

Education

• Bachelor's or Master's degree in Computer Science, Data Engineering, Information Technology, or a related discipline.
• Snowflake
• AWS Cloud
• ETL/ELT Development
• SQL & Data Modeling
• Scala / Java
• API Integration
• DevSecOps & CI/CD
• Data Quality & Governance
• Agile Delivery
• Problem Solving & Troubleshooting