IT Data Engineer (Remote)

HR Talent Partner SA; United States; Hybrid; USD 36,000 - 54,000; Full time

Job summary

HR Talent Partner SA is seeking an IT Data Engineer for a fully remote role with the option to work from the office once a week in South Africa. The position supports permanent or contract employment and requires 8-hour work days.

You will design, develop, maintain and govern data solutions to collect, integrate, transform and enable data-driven decision making across the business. The role focuses on building data pipelines, modelling data, applying SQL transformations, and ensuring data

Qualifications

• Design, build and maintain data pipelines for reliable data movement.
• Develop data models, lakehouse structures and reporting datasets.
• Apply SQL, data transformation and automation to prepare business data.
• Monitor pipelines, troubleshoot failures and optimise performance.

Responsibilities

• Design, build and maintain data pipelines that support data movement from local systems into Microsoft Fabric.
• Develop data models, lakehouse structures and reporting datasets for operational and management reporting.
• Transform and prepare data for reporting, analytics and decision making.
• Integrate data from ERP, MES, SCADA and other systems where applicable.
• Support batch, near real-time and streaming data ingestion patterns.
• Monitor data pipelines and resolve data load issues and processing exceptions.
• Optimise data pipelines and models for performance, reliability and cost efficiency.

Skills

Data pipelines

SQL & data modelling

Data governance

Monitoring & troubleshooting

Tools

Microsoft Fabric

Job description

IT Data Engineer – Remote job vacancy in South Africa.

A well establish international software development company is looking for a IT Data Engineer to join their team in either Somerset West, Pretoria or Johannesburg.

This is a fully remote position, with the option to work from the office once a week if you prefer a hybrid working arrangement. You also have the option of choosing between permanent or contract employment, with a requirement to work 8 normal working hours per day.

Position Overview:

To design, develop, maintain and support reliable, scalable and governed data solutions that enable the organisation to collect, integrate, transform and make effective use of business and operational data.

The individual in this role will assist with building and maintaining data pipelines, supporting reporting and analytics requirements, ensuring data quality and integrity, and working with business, IT, OT and engineering stakeholders to deliver accurate, secure and accessible data for operational and strategic decision-making.

Requirements / Knowledge:

• Data engineering principles, including ingestion, transformation, modelling and pipeline monitoring.
• Microsoft Fabric concepts, including lakehouses, warehouses, data pipelines, semantic models and workspaces.
• SQL databases, data modelling, data quality, metadata, lineage and database management principles.
• Data governance, access control, information security and protection of confidential business data.
• Change management principles for data platforms, reports, pipelines and business critical information systems.
• Business process knowledge to understand data sources, reporting requirements and operational decision making needs.
• Testing and validation methods used to confirm data accuracy, completeness and consistency.
• Industrial and operational data concepts, including ERP, MES, SCADA, IoT, CMMS and manufacturing KPI data sources where applicable

Skills Required:

• Design, build and maintain data pipelines and data integration processes.
• Develop data models, lakehouse structures and reporting datasets for analytics and reporting.
• Apply SQL, data transformation and automation skills to prepare reliable business data.
• Apply pipeline optimisation, monitoring and troubleshooting skills to improve performance, reliability and cost efficiency.

Duties and Responsibilities:

• SHEQ
  • Conduct continuous risk assessments in the workplace
  • Conduct planned task observations in the workplace
  • Identify potential hazards and critical safety issues in the workplace
  • Address workplace hazards and risks
  • Apply safety, health and environment practices and legal requirements when working on machines and equipment
• Policies, procedures
  • Abide by the IT policies and procedures
  • Recommend changes to the IT policies, procedures, standards, protocols, work instructions, etc.
• Financial management
  • Participate in the audit process
  • Adhere to financial policies and procedures
  • Protect company assets (working capital and fixed assets)
  • Detect and prevent fraud
  • Promote good corporate governance
• Customer relations
  • Manage relationships with key customers
  • Supplier relations
  • Manage relationships with key suppliers
• IT strategy
  • Provide input to the IT strategy in line with business requirements
  • Identify, mitigate and manage IT risk
• Data engineering
  • Design, build and maintain data pipelines that support reliable movement of data from local business systems into Microsoft Fabric.
  • Develop data models, lakehouse structures and reporting datasets that support operational, financial and management reporting requirements.
  • Transform and prepare data for reporting, analytics and business decision-making.
  • Integrate data from ERP, MES, SCADA, IoT, CMMS and other business or operational systems where applicable.
  • Support batch, scheduled, near real-time and streaming data ingestion patterns where required by business or operational reporting needs.
  • Monitor data pipelines and resolve failures, data load issues and processing exceptions.
  • Optimise data pipelines and models for performance, reliability, maintainability and cost efficiency.
  • Develop reusable and governed data products that can be consumed by reporting, analytics and business process improvement initiatives.
  • Maintain technical documentation for data sources, pipelines, transformations, models and reporting datasets.
• Data governance
  • Support the implementation of data ownership, stewardship and classification across business data domains.
  • Maintain data lineage and traceability from source systems through ingestion, transformation and reporting layers.
  • Apply agreed data standards, naming conventions and validation rules across data engineering solutions.
  • Identify and elevate data quality issues, duplicate records, incomplete data and control weaknesses.
  • Support audit evidence requirements for data movement, access control, change management and information security controls.
• Fabric migration
  • Assist with the migration of local data sources, legacy extracts and reporting processes into Microsoft Fabric.
  • Support the design and maintenance of Fabric lakehouse, warehouse, pipeline and semantic model components.
  • Contribute to scalable lakehouse, warehouse and data platform architecture that supports reusable data products and governed analytics.
  • Validate migrated data against source systems to confirm completeness, accuracy and consistency.
  • Work with business users, IT, OT, engineering and analytics stakeholders to prioritise migration activities, data product requirements and reporting needs.
  • Support the reduction of manual spreadsheets, duplicated reports and uncontrolled local data stores.
• Reporting and analytics
  • Develop and maintain reliable reporting datasets for business intelligence and operational reporting.
  • Support Power BI reporting by preparing accurate, structured and governed data models.
  • Assist business users with reporting requirements, data definitions and interpretation of available data.
  • Support reporting and analytics for manufacturing and operational KPIs, including OEE, downtime, yield, throughput, asset reliability, supply chain efficiency, safety and sustainability where applicable.
  • Review reports and datasets to identify improvement opportunities, inconsistencies and obsolete content.
  • Ensure reporting outputs are aligned with approved data sources and business rules.
• IT security
  • Apply least-privilege access to data platforms, pipelines, Fabric workspaces and reporting datasets.
  • Maintain data classification, ownership and lineage when migrating local data into Microsoft Fabric.
  • Protect confidential, personal and business-critical data across pipelines, lakehouses, models and reports.
  • Maintain audit trails for data movement, transformations, access changes and administrative activities.
  • Escalate unauthorised access, data security incidents, data quality risks and control weaknesses.
• Continuous improvement
  • Investigate opportunities to improve data collection, integration, processing and reporting through appropriate data technologies and automation
  • Identify opportunities to improve operational visibility, data reliability and decision-making through reusable data products, automation and advanced analytics enablement.
  • Participate in continuous improvement drives
• Administration
  • Compile IT reports
  • Participate in software and licence discussions to ensure appropriate tools are selected, licensed, and aligned with business and data requirements
• People management
  • Ensure an enabling climate and culture

Interview Process:

They do not have an extensive interview process and normally it is only three parts:

• A technical interview.
• An online interview.
• After successful completion, contract negotiation will take place.