Are you passionate about data, coding and creating innovative technical solutions? Do you enjoy solving problems, exploring new technologies and seeing your work make a real impact?
About the Role
Our client is a dynamic data analytics firm seeking a skilled Senior Data Engineer to join their team in Sandton, Gauteng. The role offers a hybrid work model, blending office collaboration with remote flexibility. You will be responsible for designing, building, and maintaining scalable data pipelines and infrastructure that enable efficient data processing and analysis. This is a critical role in ensuring the availability and integrity of data for business intelligence, machine learning, and reporting initiatives.
Key Responsibilities
- Design and build scalable batch and near‑real‑time data pipelines.
- Experience specifically with Claude Code.
- Ingest and process data from APIs, databases, files, event streams, and web sources.
- Develop and maintain large‑scale scraping and data acquisition systems.
- Clean, validate, transform, and model data for analytics and AI applications.
- Build LLM‑powered workflows and agentic AI solutions using structured outputs, retrieval‑augmented generation (RAG), API and database integrations.
- Support modern data lake, warehouse, and lakehouse architectures.
- Implement monitoring, observability, schema validation, and data quality controls.
- Collaborate with engineering teams to deliver secure, scalable, and maintainable production systems.
- Plan, estimate, break features into tasks, and hit the quality bar agreed with the product owner.
- Design and build ingestion and transformation pipelines across systems and domains, including batch and incremental processes. Handle error, monitoring, alerting, validation and reconciliation.
- Shape solutions that fit the target data platform. Consider cost, security, performance and support impact before building.
- Coach less experienced data engineers.
- Give analytics, BI and data science a clear view of structures, availability and how a pipeline actually behaves.
- Review code and configuration. Simplify what is already there.
- Use Git, automated testing, CI/CD and structured releases.
- Document enough that someone else can support the work.
- Use AI in the engineering workflow on approved tools only, validate output and avoid putting client or proprietary data into a public model.
- Stay accountable for what goes into production.
- Join incident response and root‑cause work when a pipeline fails. Build privacy, access control and regulatory requirements from the start.
Requirements and Experience
- A bachelor’s degree in Computer Science, Information Systems, Engineering or a related field. A relevant certification helps.
- 4–7+ years in data engineering, with production pipelines and platforms you can talk through.
- Strong AWS data architecture experience.
- Advanced Python and SQL.
- Data modelling, analytics‑oriented schema design and warehousing.
- Ingestion from relational databases, cloud storage, APIs and files.
- Git, CI/CD and automation (GitHub Actions, Azure DevOps, Docker, Kubernetes, ArgoCD, Nexus).
- Agile or SAFe delivery experience.
- Experience using AI in engineering work and knowledge of how to check it.
- Power BI or a similar BI tool is useful.
- Ability to explain a constraint to a business stakeholder and a design choice to another engineer.
- Experience with LLM platforms such as OpenAI, Anthropic, Gemini, Bedrock.
- Experience building agentic workflows involving tool use, retrieval systems, memory, and scraping frameworks such as Playwright, Puppeteer, Scrapy or Selenium.
- Strong understanding of APIs, containers, CI/CD, testing and secure engineering practices.
- Experience with orchestrating data pipelines using cloud services (AWS S3, Lambda, Glue, Azure Data Factory, Synapse, GCP BigQuery).
Salary and Contract
Reference number: NG61354. Contract: remote role. Salary: up to R650 per hour, salary negotiable based on experience.
Benefits
- Competitive salary and potential for performance bonuses.
- Hybrid work arrangement for a flexible work-life balance.
- Opportunities for professional development and acquiring new skills.
- Work on challenging and impactful data projects.
- Collaborative environment with experienced data professionals.
- Exposure to modern big data technologies and cloud solutions.
- Comprehensive health and retirement benefits.
- We pay cash for successful referrals.
Qualifications
- IT Degree / Diploma / Equivalent experience.
- Strong AWS data architecture experience.
- Advanced Python and SQL skills.
- Data modelling, analytics‑oriented schema design and warehousing experience.
- Ingestion from relational databases, cloud storage, APIs and files.
- Git, CI/CD and automation experience.
- Agile or SAFe delivery experience.
- Experience with AI in engineering work.
- Power BI or similar BI tool experience.
- Excellent communication and stakeholder engagement abilities.
Ideal Candidate
- Strong academic background.
- Stable career history demonstrating commitment and consistent growth.
- Passion for leveraging data to drive business decisions.
- Ability to work independently while collaborating effectively with cross‑functional teams.