Big Data Engineer

Centraprise; Toronto; On-site; CAD 110,000 - 160,000; Full time

Job summary

Centraprise is seeking a Senior Data Engineer to design, develop, and maintain data ingestion flows and ETL processes within an Agile cross-functional team. The role focuses on building scalable data pipelines, ensuring data quality, and integrating Snowflake with BI tools.

Candidates should have 8+ years in Big Data ETL, strong SQL and data modeling skills, and programming experience in Scala or Java. Cloud familiarity (AWS preferred) and DevSecOps practices are a plus.

Qualifications

• 8+ years of experience with Big Data ETL technologies (Hadoop, Spark, Hive).
• Strong Snowflake platform expertise and SQL/data modeling skills.
• Programming in Scala or Java; API development experience.
• Experience with ETL tools (Informatica, Talend, Apache Airflow).
• Familiarity with cloud platforms (prefer AWS) and DevOps practices.

Responsibilities

• Design, develop and maintain data ingestion flows within an Agile team.
• Build and optimize data pipelines, ETL processes and data frameworks.
• Ensure data quality, security, and performance across platforms.
• Develop and optimize SQL queries, stored procedures and views.
• Integrate Snowflake with data sources and BI tools.
• Monitor, troubleshoot and document data jobs and pipelines.
• Maintain comprehensive documentation for data models and pipelines.

Skills

Big Data

ETL

SQL

Scala/Java

DevSecOps

Python

Data modeling

Education

Bachelor’s or Master’s in Computer Science or Data Engineering

Tools

Informatica

Talend

Apache Airflow

Snowflake

AWS

Docker

Kubernetes

Job description

Skills –

Well versed with Data , ETL Processes, SQL and high-level programming languages such Scala, Java, DevSecOps

Detailed Job Description:

What will you do?

• Work within an Agile cross functional team to design, develop and maintain data ingestion flows and evolve the platform to orchestrate them.
• Design and implement data pipelines, framework and ETL processes.
• Ensure data quality, security, and performance.
• Develop and optimize SQL queries, stored procedures, and views.
• Integrate Snowflake with other data sources and BI tools.
• Monitor and troubleshoot data jobs and platform issues.
• Develop and maintain comprehensive documentation for data pipelines, transformations, and data models.

What do you need to succeed?

• 8+ years of Experience with Big Data technologies used for ETL: Hadoop, Spark, Hive
• Well versed with Snowflake Platform.
• Strong SQL skills and knowledge of data modeling.
• Programming Experience with Scala or Java an API Development.
• Experience with ETL tools (e.g., Informatica, Talend, Apache Airflow).
• Familiarity with any of the cloud platforms (Preferably AWS).
• Knowledge on Gen AI to generate code and improve developer proficiency
• Knowledge of Python or other scripting languages is a plus.
• Excellent problem-solving and communication skills.
• Knowledge of SCM, Infrastructure-as-code, and CI/CD pipelines.

Nice to Have

• Experience with workflow management tools like: Apache Airflow
• Experience with Continuous Integration tools – Git Actions, GitHub, Automated Testing tools, Git, or similar tools.
• Experience with Docker/Kubernetes/Containers/OCP4
• Shell Scripting
• Bachelor’s or master’s degree in computer science, Data Engineering and, or a related field.