Senior Data Engineer

Recru-it; Cape Town; On-site; ZAR 800,000 - 1,200,000; Full time

Job summary

A leading company is seeking a highly experienced Senior Data Engineer to lead the design, development, and optimization of data pipelines and APIs. This pivotal role focuses on ETL/ELT processes using Matillion and Snowflake while collaborating with data scientists to integrate machine learning workflows.

Qualifications

• 8+ years of experience in data modeling, data warehousing, and data integration role.
• Strong proficiency in SQL and at least one programming language (e.g., Python, Java, or Scala).

Responsibilities

• Design, build, and maintain ETL/ELT pipelines using Matillion and Snowflake.
• Develop and manage RESTful APIs for data access and integration.
• Collaborate with data scientists and ML engineers to integrate data pipelines with machine learning workflows.

Skills

Matillion ETL

Snowflake

Databricks

API Development

Data Warehousing

SQL

Cloud Platforms

Education

Bachelors or Masters degree in Computer Science, Engineering, or a related field

Job description

Job Description
Summary
Our client is seeking a highly experienced Senior Data Engineer to lead the design, development, and optimization of data pipelines, APIs, and data platforms. This role will focus on ETL/ELT processes using Matillion and Snowflake, API development, and integration with machine learning workflows and Databricks. The ideal candidate will have a strong background in data engineering, cloud platforms, and modern data architecture.

Responsibilities

• Design, build, and maintain ETL/ELT pipelines using Matillion and Snowflake.
• Develop and manage RESTful APIs for data access and integration.
• Collaborate with data scientists and ML engineers to integrate data pipelines with machine learning workflows.
• Optimize Snowflake data warehouse performance and manage data models.
• Implement data quality, governance, and security best practices.
• Work with Databricks for data processing, transformation, and ML model support.
• Automate data validation and reconciliation processes.
• Document data architecture, pipelines, and integration processes.

Qualifications

• Bachelors or Masters degree in Computer Science, Engineering, or a related field.
• 8+ years of experience in a data modeling, data warehousing, and data integration role.

Required Skills

• Strong experience with Matillion ETL and Snowflake.
• Proficiency in Databricks
• Experience in data management requirements, integration platforms, and have expertise with APIs.
• Experience with API development using frameworks.
• Familiarity with Databricks and integrating Machine Learning workflows.
• Solid understanding of data warehousing, data modeling, and ELT best practices.
• Experience with CI/CD pipelines, version control (Git), and DevOps practices.
• Strong proficiency in SQL and at least one programming language (e.g., Python, Java, or Scala).
• Experience with data modeling, data warehousing, and data integration.
• Familiarity with cloud platforms and data services.
• Understanding of data governance, security, and compliance requirements

Job Description
Summary
Our client is seeking a highly experienced Senior Data Engineer to lead the design, development, and optimization of data pipelines, APIs, and data platforms. This role will focus on ETL/ELT processes using Matillion and Snowflake, API development, and integration with machine learning workflows and Databricks. The ideal candidate will have a strong background in data engineering, cloud platforms, and modern data architecture.

Responsibilities

• Design, build, and maintain ETL/ELT pipelines using Matillion and Snowflake.
• Develop and manage RESTful APIs for data access and integration.
• Collaborate with data scientists and ML engineers to integrate data pipelines with machine learning workflows.
• Optimize Snowflake data warehouse performance and manage data models.
• Implement data quality, governance, and security best practices.
• Work with Databricks for data processing, transformation, and ML model support.
• Automate data validation and reconciliation processes.
• Document data architecture, pipelines, and integration processes.

Qualifications

• Bachelors or Masters degree in Computer Science, Engineering, or a related field.
• 8+ years of experience in a data modeling, data warehousing, and data integration role.

Required Skills

• Strong experience with Matillion ETL and Snowflake.
• Proficiency in Databricks
• Experience in data management requirements, integration platforms, and have expertise with APIs.
• Experience with API development using frameworks.
• Familiarity with Databricks and integrating Machine Learning workflows.
• Solid understanding of data warehousing, data modeling, and ELT best practices.
• Experience with CI/CD pipelines, version control (Git), and DevOps practices.
• Strong proficiency in SQL and at least one programming language (e.g., Python, Java, or Scala).
• Experience with data modeling, data warehousing, and data integration.
• Familiarity with cloud platforms and data services.
• Understanding of data governance, security, and compliance requirements

Desired Skills:

• ETL and Snowflake
• Databricks
• APIs
• CI/CD pipelines
• Strong SQL
• Cloud platforms and data services