Data Engineer

Blue Pearl Pty Ltd; KwaZulu-Natal; On-site; ZAR 30 000 - 70 000; Full time

Job summary

An innovative firm is seeking a skilled Data Engineer to enhance data pipeline architecture and optimize data flow for cross-functional teams. This role involves collaborating with software developers, database architects, and data scientists to ensure consistent data delivery across projects. You'll be responsible for building the infrastructure for data extraction and transformation, while also creating analytics tools that generate actionable insights. If you have a passion for data and a knack for problem-solving, this opportunity is perfect for you.

Qualifications

• 3+ years as a Database Administrator with strong query writing skills.
• Experience in managing large data volumes and ETL processes.

Responsibilities

• Create and maintain optimal data pipeline architecture.
• Build infrastructure for data extraction, transformation, and loading.
• Collaborate with stakeholders to resolve data-related issues.

Skills

SQL

Azure Big Data Technologies

Data Pipeline Architecture

Data Extraction

Data Scrubbing

ETL Processes

Relational Databases

NoSQL Databases

Education

Qualification in Data Informatics

Tools

Data Warehousing Tools

Job description

Data Engineer Job Description

The data engineer is responsible for expanding and optimizing our data and data pipeline architecture, as well as optimizing data flow and collection for cross-functional teams.

The Data Engineer will support software developers, database architects, data warehouse managers, data analysts, and data scientists on data initiatives and will ensure that the data delivery architecture is consistent across ongoing projects.

Duties & Responsibilities

1. Create and maintain optimal data pipeline architecture.
2. Assemble large, complex data sets that meet functional and non-functional business requirements.
3. Build the infrastructure required for optimal extraction, transformation, and loading of data from various sources using SQL and Azure 'big data' technologies.
4. Collaborate with stakeholders including Executive, Service, Data, and Design teams to resolve data-related technical issues and support their infrastructure needs.
5. Create data tools for analytics and data science teams.
6. Work with data and analytics experts to enhance data system functionality.
7. Map data between source systems, data warehouses, and data marts.
8. Ensure data security and separation across national boundaries through multiple data centers and cloud regions.
9. Implement business rules via stored procedures, middleware, or other technologies.
10. Develop and execute data extraction procedures from various systems such as call centers, HR, or external sources.
11. Create supporting documentation, including metadata, entity relationship diagrams, business processes, and flowcharts.
12. Design and test database scripts and files, from unit to integration testing.
13. Identify, design, and implement internal process improvements, such as automating manual processes, optimizing data delivery, and redesigning infrastructure for scalability.
14. Select methods and techniques for database evaluation.
15. Build analytics tools utilizing data pipelines to generate actionable insights into customer acquisition, operational efficiency, and key business metrics.
16. Design and implement reporting systems for various processes.
17. Analyze system requirements and gather business information.
18. Research and adopt technologies to ensure optimal operation of the data warehouse.

Desired Experience & Qualifications

1. Currently a Database Administrator with at least 3 years of experience in query writing, data extraction, data scrubbing, and managing large data volumes.
2. Experience with relational and NoSQL databases.
3. Experience with ETL processes.
4. Qualification in Data Informatics or equivalent experience in large data environments.
5. Experience managing a team of analysts and demonstrating technical expertise in complex database queries.