Data Engineer

Bravissimo Resourcing Inc.; Makati; On-site; PHP 1,792,114 - 3,584,229; Full time

Job summary

A tech-focused resourcing company is seeking a Data Engineer based in Makati, Philippines. The ideal candidate will design, develop, and maintain data pipelines using Microsoft Azure services. Responsibilities include collaborating with stakeholders, ensuring data integrity, and optimizing pipelines for performance. A Bachelor's degree in computer science or related field is required, along with proven experience in ETL processes and strong knowledge of Azure tools. This role emphasizes both technical and soft skills, alongside a focus on continuous learning and innovation.

Qualifications

• Demonstrate high proficiency in programming fundamentals.
• Proven experience dealing with data and ETL processes.
• Strong experience in Python preferred, with knowledge in Scala, Java, C#.

Responsibilities

• Design and develop data pipelines using Microsoft Azure services.
• Collaborate with stakeholders to understand data requirements.
• Optimize data pipelines for performance, scalability, and reliability.

Skills

Data engineering principles

Azure Data Factory

SQL DML

Big data technologies

Python

Problem-solving skills

Communication skills

Education

Bachelor’s degree in computer science, Engineering, or related field

Tools

Azure Data Lake Storage Gen 2

Azure Blob Storage

Azure DevOps

Spark

Git

Ansible

Job description

Responsibilities

• Design, develop, and maintain data pipelines and ETL processes using Microsoft Azure services (e.g., Azure Data Factory, Azure Synapse, Azure Databricks, Azure Fabric).
• Utilize Azure data storage accounts for organizing and maintaining data pipeline outputs. (e.g., Azure Data Lake Storage Gen 2 & Azure Blob storage).
• Collaborate with data scientists, data analysts, data architects and other stakeholders to understand data requirements and deliver high-quality data solutions.
• Optimize data pipelines in the Azure environment for performance, scalability, and reliability.
• Ensure data quality and integrity through data validation techniques and frameworks.
• Develop and maintain documentation for data processes, configurations, and best practices.
• Monitor and troubleshoot data pipeline issues to ensure timely resolution.
• Stay current with industry trends and emerging technologies to ensure our data solutions remain cutting-edge.
• Manage the CI/CD process for deploying and maintaining data solutions.

Qualifications

• Bachelor’s degree in computer science, Engineering, or a related field (or equivalent experience) and able to demonstrate high proficiency in programming fundamentals.
• Proven experience as a Data Engineer or similar role dealing with data and ETL processes.
• Strong knowledge of Microsoft Azure services, including Azure Data Factory, Azure Synapse, Azure Databricks, Azure Blob Storage and Azure Data Lake Gen 2.
• Experience utilizing SQL DML to query modern RDBMS in an efficient manner (e.g., SQL Server, PostgreSQL).
• Strong understanding of Software Engineering principles and how they apply to Data Engineering (e.g., CI/CD, version control, testing).
• Experience with big data technologies (e.g., Spark).
• Strong problem-solving skills and attention to detail.
• Excellent communication and collaboration skills.
• Learning agility
• Technical Leadership
• Consulting and managing business needs
• Strong experience in Python is preferred but experience in other languages such as Scala, Java, C#, etc. is accepted.
• Experience building spark applications utilizing PySpark.
• Experience with file formats such as Parquet, Delta, Avro.
• Experience efficiently querying API endpoints as a data source.
• Understanding of the Azure environment and related services such as subscriptions, resource groups, etc.
• Understanding of Git workflows in software development.
• Using Azure DevOps pipeline and repositories to deploy and maintain solutions.
• Understanding of Ansible and how to use it in Azure DevOps pipelines.