DATA ENGINEER
This is for a full-time position.
Job Summary
The Data Engineer will be responsible for designing and implementing data pipelines using the Microsoft Fabric Data Platform. This involves creating Notebooks to extract, transform, and load data. The role will receive technical guidance from a Senior Data Engineer based in Sydney and day-to-day support from the Data Team Leader.
Responsibilities
- Design, build, and maintain data pipelines to support data ingestion, preparation, and transformation processes across multiple layers.
- Develop and manage Notebooks for transforming data between Bronze, Silver, and Gold layers, ensuring proper version control and adherence to best practices.
- Implement and maintain diverse ingestion methods, such as Shortcuts and Database Mirroring, to enable efficient data integration.
- Conduct quality assurance and testing on ingestion processes, validating data accuracy and consistency between source systems and Microsoft Fabric.
- Monitor and optimize pipeline performance, identifying inefficiencies and implementing improvements to enhance reliability and scalability.
- Troubleshoot and resolve pipeline errors and alerts, ensuring minimal downtime and smooth data flow.
- Document and maintain existing data pipelines, including creating clear technical documentation and optimizing workflows for long-term sustainability.
Qualifications
- Bachelor’s degree in Computer Science, Data Engineering, or related field
- At least 2 years’ experience in data engineering or ETL development
- Proficient in Python and SQL for data processing and analysis.
- Skilled in working with REST APIs for data integration.
- Hands‑on experience with Microsoft Fabric or similar data platforms.
- Strong understanding of data modeling and transformation techniques.
- Strong problem-solving and troubleshooting ability.
- Effective communication and documentation skills.
- Ability to work collaboratively in a team environment.
- Willingness to learn and try new things
- Amenable to work on a day shift work schedule in Ortigas (AU working hours, hybrid work setup)