Aws Data Engineer

Linkedin - Jobboard; Pretoria; On-site; ZAR 600,000 - 1,000,000; Full time

Job summary

A leading company is seeking an AWS Data Engineer to create and manage data infrastructure. The role involves designing scalable architectures, developing ETL pipelines, and ensuring data security and compliance. Candidates are required to have an honors degree in Computer Science or Engineering, along with strong experience in AWS services and Python. This position offers ample opportunities for mentorship and collaboration with data teams.

Qualifications

• 5+ years working experience in Data Engineering.
• Strong skills in Python (especially PySpark for AWS Glue).
• Experience with AWS services used for data warehousing.

Responsibilities

• Design and maintain scalable data architectures using AWS services.
• Develop and manage metadata repositories and data dictionaries.
• Design, develop, and optimize scalable ETL pipelines.

Skills

Python

Data Modeling

Schema Design

AWS Services

SQL

NoSQL Databases

ETL Processes

Education

Honors degree in Computer Science or Engineering

AWS Certified Solutions Architect

AWS Certified Data Engineer

Job description

Join to apply for the AWS Data Engineer role at HuntWave.Get AI-powered advice on this job and more exclusive features.Job Purpose : Responsible for creating and managing the technological part of data infrastructure at every step of data flow.

From configuring data sources to integrating analytical tools, all these systems would be architected, built, and managed by a general-role data engineer.Minimum education (essential) : Honors degree in Computer Science or Engineering (or similar)AWS Certified Solutions Architect orAWS Certified Data AnalystMinimum education (desirable) : Bachelor's degree in Computer Science or Engineering (or similar)AWS Certified Data Engineer orAWS Certified Solutions Architect orAWS Certified Data AnalystMinimum applicable experience (years) : 5+ years working experienceRequired nature of experience : Data Engineering developmentExperience with AWS services used for data warehousing, computing, and transformations such as AWS Glue (crawlers, jobs, triggers, and catalog), AWS S3, AWS Lambda, AWS Step Functions, AWS Athena, and AWS CloudWatchExperience with SQL and NoSQL databases (e.g., PostgreSQL, MySQL, DynamoDB)Experience with SQL for querying and transformation of dataSkills and Knowledge (essential) : Strong skills in Python (especially PySpark for AWS Glue)Strong knowledge of data modeling, schema design, and database optimizationProficiency with AWS and infrastructure as codeSkills and Knowledge (desirable) : Knowledge of SQL, Python, AWS serverless microservicesDeploying and managing ML models in productionVersion control (Git), unit testing, and agile methodologiesData Architecture and Management 20%Design and maintain scalable data architectures using AWS services like AWS S3, AWS Glue, and AWS AthenaImplement data partitioning and cataloging strategiesWork with schema evolution and versioning to ensure data consistencyDevelop and manage metadata repositories and data dictionariesSupport setup and maintenance of data access roles and privilegesPipeline Development and ETL 30%Design, develop, and optimize scalable ETL pipelines with batch and real-time frameworks (AWS Glue and PySpark)Implement data extraction, transformation, and loading processes from various sourcesOptimize ETL jobs for performance, cost, and scalabilityDevelop and integrate APIs for data workflowsEnable deployment of ML models by integrating data pipelines with ML workflowsAutomation, Monitoring, and Optimization 30%Automate data workflows ensuring fault tolerance and optimizationImplement logging, monitoring, and alertingTune ETL configurations and analyze resource usage for performanceOptimize data storage solutions for performance and costEnsure scalability of AWS resources for data ingestion and outputsDeploy ML models into production using AWS SagemakerSecurity, Compliance, and Best Practices 10%Ensure API security, authentication, and access control best practicesImplement data encryption, access control, and compliance with GDPR, HIPAA, SOC2, etc.Establish data governance policiesDevelopment, Team Mentorship, and Collaboration 5%Collaborate with data scientists, analysts, and business teamsWork with backend teams on CI / CD integrationMentor team members through coaching and code reviewsAlign with B2C division strategy and client visionQMS and Compliance 5%Document data processes and architectural decisionsMaintain software quality and compliance with QMS, data security, and industry standards (ISO, CE, FDA)Safeguard confidential information