Senior Data Engineer

Cf Consulting; Gauteng; On-site; ZAR 600 000 - 900 000; Full time

Job summary

A leading company in South Africa is seeking a Data Engineer with a strong background in data analysis, manipulation, and modelling. The successful candidate will drive technical vision and translate business needs into actionable strategies. Candidates should possess a relevant tertiary degree and extensive experience in data integration and Big Data technologies. This role involves collaborating with teams to implement solutions in cloud environments and optimize data processes.

Qualifications

• Tertiary degree in Computer Science or related field required.
• 8-10 years of experience as a Data Engineer / Database Developer.
• Proficiency in ETL solutions and data integration.

Responsibilities

• Identify valuable data sources and automate collection processes.
• Analyze large amounts of information to discover trends and patterns.
• Collaborate with engineering and product development teams.

Skills

Data analysis

Data manipulation

Data modelling

Data integration

ETL solutions

Big Data technologies

Database query languages

Analytical mind

Education

BSc Computer Science

B.IT

Informatics related degrees

Tools

SQL Server

Oracle

Azure Data Factory

Hadoop

Spark

Kafka

Tableau

Power BI

Job description

The Data Engineer will have a knack for data analysis, data manipulation, and data modelling and will be responsible for understanding and driving the overall technical vision and planning of a client's organisation and translating business needs into technical strategy.

Required Qualifications :

Tertiary degree, diploma or certificate in a related field (BSc Computer Science, B.IT or Informatics related degrees).

Experience and Knowledge :

• 8-10 years' working experience as a Data Engineer / Database Developer.
• Experience in data mining, large scale data modelling, and business requirements gathering / analysis.
• Understanding and working experience in data integration and transformation.
• Experience implementing data modelling methodologies like Dimensional Modeling and / or Data Vault.
• Working knowledge of data quality processes and master data management.
• Experience implementing design support systems using Database Management Systems (DBMS) such as SQL Server or Oracle.
• Proficiency in designing and implementing data integration and ETL solutions using SSIS, Azure Data Factory and / or SQL Server stored procedures.
• Understanding of several Big Data technologies like Hadoop, MapReduce, and Spark as well as event processing or message ingestion services like Kafka, Event Hub, and Stream Analytics.
• Experience in database query languages such as T-SQL, ANSI SQL, PL / SQL.
• Some experience developing software solutions using Visual Basic, C++, C#, Java, or Python.
• Experience using SQL Server Management Studio and Visual Studio.
• Experience implementing solutions using Azure SQL databases, Azure Synapse (Previously SQL Data Warehouse), Azure Storage Accounts (Data Lake), and / or Databricks.
• Analytical mind and business acumen.
• Additional skills in Tableau, Power BI, strong math skills (statistics, algebra), Scala, Python, or R will be considered an asset.

Key Responsibilities :

• Identify valuable data sources and automate collection processes.
• Undertake preprocessing of structured and unstructured data.
• Analyze large amounts of information to discover trends and patterns.
• Data Modelling (Relational and Star Schema).
• Data Warehouse Design - Build and Development.
• Database Administration.
• Database Performance Tuning and Optimisation.
• Present information using data visualization techniques.
• Propose solutions and strategies to business challenges.
• Collaborate with engineering and product development teams.
• Have hands-on experience with Hadoop / Spark based distributed storage and computing frameworks.
• Implement real-time analytics and batch processing.
• Architect analytical applications in cloud environments such as Amazon Web Services and Microsoft Azure.