Data Operations Engineer

Visa Hunt; South Africa; On-site; ZAR 900,000 - 1,300,000; Full time

Job summary

Visa Hunt is seeking a senior data engineering specialist to process large data sets and implement end-to-end data pipelines in AWS. You will design scalable ingestion, transformation, and loading processes while aligning with the organisation's data governance and cloud strategy.

The role requires extensive AWS experience, Python proficiency, and familiarity with CI/CD and DevOps practices. Collaboration with stakeholders across the business is essential for delivering data-driven insights.

Qualifications

• Experience processing large volumes of structured and unstructured data.
• Proficiency in Python for data engineering and ML/PM tasks.
• Experience with AWS services and infrastructure as code.
• Familiarity with data warehousing ETL/ELT architectures.
• Knowledge of data governance and security practices.

Responsibilities

• Process structured and unstructured data for data science activities.
• Support data analysis to identify trends and patterns.
• Design scalable data ingestion and transformation pipelines.
• Create and maintain data pipelines from multiple sources.
• Ingest large, complex datasets meeting requirements.
• Manage RFCs and provisioning in AWS via RFC process.
• Develop and maintain CICD pipelines.
• Administer Linux and Windows in AWS (SSH, RDP).
• Build bulk and delta data lift patterns for ETL/ELT.
• Align data architecture with governance and cloud strategy.
• Collaborate with stakeholders to understand data needs.
• Provide operational support with data services teams.

Skills

Big data processing

Python programming

SAFe/Scrum/Kanban experience

Data governance knowledge

Data pipeline design

Cloud architecture

DevOps practices

Linux/Windows administration

Education

Degree in Computer Science / Data Engineering / Business Informatics

Tools

AWS

CloudFormation

CodeCommit

CodePipeline

CodeBuild

DynamoDB

RDS

S3

IAM

EMR

VPC

AMS

CloudFormation templates

Job description

Key roles and responsibilities:

• Undertake the processing of structured and unstructured data to utilize in data science activities.
• Support the data analysis of large information to discover trends and patterns and provide feedback to the business.
• Assist in designing and implementing scalable and robust processes for ingesting and transforming datasets.
• Design, implement, and support the creation and maintenance of data pipelines from a multitude of sources.
• Ingest large, complex data sets that meet functional and non-functional requirements.
• Request For Change (RFC) management in Amazon Managed Services (AMS) (logging, escalation, resolution)
• Provisioning infrastructure in AWS (via RFC process, CloudFormation, Console)
• AWS Security Management (authoring IAM security policies, logging security requests, user management/permissions)
• Infrastructure As Code (development of CloudFormation templates)
• Development, deployment, and management of CICD (Continuous Integration Continuous Deployment ) pipelines.
• Linux and Windows System Administration within AWS (SSH, RDP).
• Design and build bulk and delta data lift patterns for optimal extraction, transformation, and loading of data.
• Supports the organisations cloud strategy and aligns to the data architecture and governance including the implementation of these data governance practices.
• Work with various stakeholders across the organization to understand data requirements and apply technical knowledge of data management to solve key business problems.
• Provide support in the operational environment with all relevant support teams for data services.

Knowledge, skills and attributes

• Working with large volumes of structured and unstructured data and leveraging them to build artificial intelligence (AI)/machine learning (ML) and predictive modelling (PM) solutions through end-to-end automated data pipelines.
• Deep and Extensive AWS knowledge and skills: EMR, Lambda, EC2, VPC, Transit Gateway, S3,IAM, Infrastructure As Code (e.g. CloudFormation), Code Commit, Code Pipeline and Code Build.
• Advantageous AWS experience with DynamoDB, RDS (MS SQL, Oracle), Landing Zone, Control Tower and AMS
• The candidate having DBA ability and knowledge across at least 2 platforms (for example TSQL, SAS, PSQL, IBM VSAM DB2 etc.) will also be beneficial.
• Experience with the Python programming language.
• Experience with designing and implementing Cloud (AWS) solutions including the use of APIs available.
• Experience with Dev/OPS architecture, implementation, and operation would be advantageous.
• Some experience in applying SAFe/Scrum/Kanban methodologies would be advantageous.
• Knowledge of Engineering and Operational Excellence using standard methodologies. Best practices in software engineering, data management, data storage, data computing and distributed systems to solve business problems with data.
• Strong AWS cost management, billing resolution and cost optimization

Academic qualifications and certifications:

• Matric, with a degree in Computer Science, Business Informatics or Data Engineering
• 5+ years IT work experience and 2+ years of experience in a senior technical position in a relevant role e.g. System Administrator, Network Engineer, DevOps Engineer, Cloud Engineer.
• 5 + years of data engineering experience
• 3+ years of experience with any data warehouse technical architectures, ETL/ELT, and reporting/analytics tools including, but not limited to, any of the following combinations (1) SSIS and SSRS, (2) SAS ETL Framework, (3) SAP ETL Framework, (4) MongoDB ETL deployments, (5) Apache Spark and Apache Hive deployments will be beneficial.