Senior Data Engineer

Recru-it; Johannesburg; On-site; ZAR 45,000 - 65,000; Full time

Job summary

A leading company is seeking a highly experienced Senior Data Engineer to oversee the design and development of data pipelines and integration with machine learning workflows. This role focuses on enhancing data platforms using Matillion and Snowflake while ensuring data quality and governance. Candidates should have a strong data engineering background, ideally with extensive cloud platform experience.

Qualifications

• Bachelor's or Master's degree in Computer Science, Engineering, or related.
• 8+ years of experience in data modeling, warehousing, and integration.

Responsibilities

• Design, build, and maintain ETL/ELT pipelines using Matillion and Snowflake.
• Develop and manage RESTful APIs for data access.
• Collaborate with data scientists and ML engineers.

Skills

Data modeling

Data warehousing

Data integration

Matillion ETL

Snowflake

Databricks

CI/CD pipelines

Version control (Git)

Education

Bachelor's or Master's degree in Computer Science, Engineering, or a related field

Job description

Job Description
Summary
Our client is seeking a highly experienced Senior Data Engineer to lead the design, development, and optimization of data pipelines, APIs, and data platforms. This role will focus on ETL/ELT processes using Matillion and Snowflake, API development, and integration with machine learning workflows and Databricks. The ideal candidate will have a strong background in data engineering, cloud platforms, and modern data architecture.

Responsibilities

• Design, build, and maintain ETL/ELT pipelines using Matillion and Snowflake.
• Develop and manage RESTful APIs for data access and integration.
• Collaborate with data scientists and ML engineers to integrate data pipelines with machine learning workflows.
• Optimize Snowflake data warehouse performance and manage data models.
• Implement data quality, governance, and security best practices.
• Work with Databricks for data processing, transformation, and ML model support.
• Automate data validation and reconciliation processes.
• Document data architecture, pipelines, and integration processes.

Qualifications

• Bachelors or Masters degree in Computer Science, Engineering, or a related field.
• 8+ years of experience in a data modeling, data warehousing, and data integration role.

Required Skills

• Strong experience with Matillion ETL and Snowflake.
• Proficiency in Databricks
• Experience in data management requirements, integration platforms, and have expertise with APIs.
• Experience with API development using frameworks.
• Familiarity with Databricks and integrating Machine Learning workflows.
• Solid understanding of data warehousing, data modeling, and ELT best practices.
• Experience with CI/CD pipelines, version control (Git), and DevOps practices.
• Strong proficiency in SQL and at least one programming language (e.g., Python, Java, or Scala).
• Experience with data modeling, data warehousing, and data integration.
• Familiarity with cloud platforms and data services.
• Understanding of data governance, security, and compliance requirements

Job Description
Summary
Our client is seeking a highly experienced Senior Data Engineer to lead the design, development, and optimization of data pipelines, APIs, and data platforms. This role will focus on ETL/ELT processes using Matillion and Snowflake, API development, and integration with machine learning workflows and Databricks. The ideal candidate will have a strong background in data engineering, cloud platforms, and modern data architecture.

Responsibilities

• Design, build, and maintain ETL/ELT pipelines using Matillion and Snowflake.
• Develop and manage RESTful APIs for data access and integration.
• Collaborate with data scientists and ML engineers to integrate data pipelines with machine learning workflows.
• Optimize Snowflake data warehouse performance and manage data models.
• Implement data quality, governance, and security best practices.
• Work with Databricks for data processing, transformation, and ML model support.
• Automate data validation and reconciliation processes.
• Document data architecture, pipelines, and integration processes.

Qualifications

• Bachelors or Masters degree in Computer Science, Engineering, or a related field.
• 8+ years of experience in a data modeling, data warehousing, and data integration role.

Required Skills

• Strong experience with Matillion ETL and Snowflake.
• Proficiency in Databricks
• Experience in data management requirements, integration platforms, and have expertise with APIs.
• Experience with API development using frameworks.
• Familiarity with Databricks and integrating Machine Learning workflows.
• Solid understanding of data warehousing, data modeling, and ELT best practices.
• Experience with CI/CD pipelines, version control (Git), and DevOps practices.
• Strong proficiency in SQL and at least one programming language (e.g., Python, Java, or Scala).
• Experience with data modeling, data warehousing, and data integration.
• Familiarity with cloud platforms and data services.
• Understanding of data governance, security, and compliance requirements

Desired Skills:

• data modeling
• data warehousing
• data integration role.
• Matillion ETL and Snowflake
• Databricks
• CI/CD pipelines
• version control Git