Senior Data Engineer - Scala/Spark

ASTEK SINGAPORE INNOVATION TECHNOLOGY PTE. LTD.; Singapore; On-site; SGD 80,000 - 120,000; Full time

Job summary

A leading technology company in Singapore seeks a Data Software Engineer to work with cutting-edge Big Data services including Spark, Scala, and Databricks. The role requires 6-10 years of experience in data engineering, familiarity with CI/CD pipelines, and a strong understanding of SQL. Candidates should thrive in an Agile/SCRUM environment and have experience with version control and automation testing tools. Join a passionate team to help optimize data-driven projects and enhance data accuracy and traceability.

Qualifications

• 6-10 years of experience as a Data engineer.
• Intermediate level of Database (SQL) skills.
• Demonstrated ability to thrive in an enterprise Agile/SCRUM environment.

Responsibilities

• Monitor and optimize data pipeline performance.
• Implement data governance practices.
• Develop unit and integration test cases.

Skills

Data engineering

SQL

CI/CD

Agile/SCRUM

Linux/Unix

GitHub

Object-oriented programming (Java)

Testing tools

Tools

Spark

Scala

Databricks

PySpark

Jira

Confluence

Job description

Data Software Engineer, you'll be part of a team who are passionate about learning and prototyping cutting-edge technologies and convergence initiatives with latest Big Data services on Spark, Scala, Databricks, PySpark and leverage GitHub to support things like CI/CD and integrations

• 6-10 years of experience as a Data engineer who works on enterprise standard frameworks like Spark, Scala, Databricks, PySpark.
• Development experience in unit and integration test cases
• Intermediate level of Database (SQL) skills to develop SQL queries, functions
• Good Understanding on CI/CD Pipeline i.e. GitHub Actions
• Strong knowledge of version control tools, preferably GitHub
• Basic Knowledge on Linux/Unix environment (basic commands, shell scripting, etc.)
• Demonstrated ability to thrive in an enterprise Agile/SCRUM environment
• Implement data governance practices, data lineage, and metadata management to ensure data accuracy, traceability, and compliance
• Monitor and optimize data pipeline performance, troubleshoot issues, and implement necessary enhancements
• Implement monitoring and logging mechanisms to ensure the health, availability, and performance of the data infrastructure
• Experience using Collaboration Technologies: GitHub, Jira, Confluence
• Experience using Object-oriented languages Java
• Experience working with testing tools and Automation test needs