Senior Data Engineer

Shuru Technologies; Singapore; Hybrid; SGD 120,000 - 180,000; Full time

Job summary

Shuru Technologies in Singapore seeks a Senior Data Engineer to build, test, and run data pipelines on Azure Databricks, turning raw source data into governed datasets using Delta Lake.

You will collaborate with a Forward Deployed Data Lead and Lead Data Engineer to translate requirements into production plans, ensuring adherence to platform standards and reliability.

Qualifications

• Working experience with Delta Lake and lakehouse (bronze/silver/gold) architecture.
• Solid understanding of data modelling, including dimensional modelling.
• Experience implementing automated data validation, testing, and monitoring for pipelines.
• Familiarity with data cataloguing, lineage, and metadata concepts.
• Exposure to Structured Streaming or real-time processing is a plus.
• Familiarity with dbt and Git-based CI/CD workflows is a plus.

Responsibilities

• Build, test, and maintain data pipelines on Azure Databricks for the assigned business line, using PySpark and SQL.
• Implement bronze, silver, and gold layer transformations using Delta Lake, following the platform's established lakehouse standards.
• Ingest structured and semi-structured data from sources such as MariaDB, Azure Event Hubs, APIs, and file systems.

Skills

Python/ PySpark

SQL

Delta Lake

Data modelling

Automated data validation

Data pipeline monitoring

Git-based CI/CD

Azure Databricks

Education

Bachelor's degree in CS/Engineering or related field

Tools

dbt

Git

Job description

The Senior Data Engineer is a hands-on builder dedicated to a specific business line, responsible for developing, testing, and running the data pipelines and models that turn that line's raw source data into reliable, governed datasets on Azure Databricks.

The role works day-to-day with the Forward Deployed Data Lead for that business line to translate scoped requirements into production pipelines, and with the Lead Data Engineer to make sure what's built follows the platform's shared standards rather than becoming a one-off, line-specific solution. This is an execution-focused role: less about setting architecture direction and more about building it well, reliably, and on time, with a clear growth path toward Lead Data Engineer for those who want to take on cross-line or platform-wide ownership.

The candidate will have responsibilities across the following functions:

Pipeline Development:

• Build, test, and maintain data pipelines on Azure Databricks for the assigned business line, using PySpark and SQL.
• Implement bronze, silver, and gold layer transformations using Delta Lake, following the platform's established lakehouse standards.
• Ingest structured and semi-structured data from sources such as MariaDB, Azure Event Hubs, APIs, and file systems.

Data Modelling:

• Design and build data models (dimensional models, gold-layer tables) for the business line's use cases, in line with patterns set by the Lead Data Engineer.
• Flag where a business line's data model needs a shared, cross-line solution rather than a local one, and raise it to the Lead Data Engineer.

Quality and Reliability:

• Implement automated testing, data validation, and pipeline monitoring for everything owned.
• Respond to and troubleshoot pipeline incidents and data quality issues for the assigned business line, escalating where needed.
• Follow the platform's cataloguing, lineage, and metadata practices so the data produced is discoverable and trustworthy.

Collaboration and Delivery:

• Work directly with the Forward Deployed Data Lead to turn scoped business requirements into build plans and delivery timelines.
• Partner with analysts and, where relevant, data scientists and AI engineers consuming the pipelines built, to make sure outputs meet their needs.
• Participate in code reviews and platform standards discussions led by the Lead Data Engineer.

Compliance:

• Ensure pipelines and datasets meet data security, privacy, and regulatory requirements (e. g., GDPR, PII handling) relevant to the business line.

Requirements:

• Working experience with Delta Lake and lakehouse (bronze/silver/gold) architecture.
• Solid understanding of data modelling, including dimensional modelling.
• Experience implementing automated data validation, testing, and monitoring for pipelines.
• Familiarity with data cataloguing, lineage, and metadata concepts.
• Exposure to Structured Streaming or real-time processing is a plus.
• Familiarity with dbt and Git-based CI/CD workflows is a plus.

Collaboration and Communication:

• Speaks with clarity and confidence, able to discuss initiatives with Business and Leads with full end-to-end context
• Comfortable working directly with a business-facing lead (Forward Deployed Data Lead) to clarify requirements and scope.
• Able to explain technical trade-offs clearly to both engineering peers and less technical stakeholders.

Experience:

• 5-8 years of experience in data engineering, including hands-on work with cloud data platforms.
• Experience building pipelines on Azure Databricks, Delta Lake, or comparable lakehouse/cloud-native tools.
• Exposure to version control (Git), Agile methodologies, and CI/CD practices.
• Experience in fintech, broking, or trading platforms is a plus, not required.
• Bachelor's degree in Computer Science, Engineering, or a related field.