Senior AWS Data Engineer (6 Months Contract)

PALO IT; Singapore; On-site; SGD 80,000 - 120,000; Full time

Job summary

A leading global technology consultancy is seeking a Senior Data Engineer to manage and optimize data pipelines. The role involves working with large datasets, cloud migration, and collaborating with cross-functional teams to deliver impactful solutions. Ideal candidates will have extensive experience in data engineering and a passion for technology.

Benefits

Stimulating working environments

Unique career path

International mobility

Personalized training

Qualifications

• At least 7 years of experience in data engineering.
• Good knowledge of big data technology landscape.
• Experience with batch & ETL jobs.

Responsibilities

• Execute a data pipeline for downloading, processing, and error handling.
• Migrate data pipeline to AWS Cloud.
• Enhance existing data pipeline using industry-standard tools.

Skills

Python

SQL

NoSQL

DevOps

Data Science

Education

Bachelor's degree in IT

Master's degree in Information Management

Ph.D. in Computer Science

Tools

AWS

Airflow

Hadoop

Spark

Power BI

Job description

WHO WE ARE:

Build. Scale. Sustain.

PALO IT is a global technology consultancy that crafts tech as a force for good. We design, develop and scale digital and sustainable products and services to unlock value across the triple bottom line: people, planet, profit.We do the right thing, and we do it right.We're proud to be a World Economic Forum New Champion, and a B Corp-certified company.

• We aresmall enough to care locally, big enough to deliver globally(5 continents, 18 offices, +650 experts from +50 nationalities)
• We arerobust and resilient(100% independent and 0 debt)
• We areentrepreneurs and passionate experts: We invest in what we believe genuinely and work as a collective intelligence
• We arepositive, courageous, caring, doers and committed to excellence

YOUR ROLE:

The Senior Data Engineer will work on acquiring, storing, governing and processing huge sets of structured / unstructured data. You will bring a vision of the big data solutions landscape for choosing optimal architecture components to use for these purposes as well as a set of data engineering skills for implementing enterprise data foundations such as data lake. Finally, you will work closely with other PALO IT experts in Data Intelligence, Research, UX Design, Digital Technology and Agile teams to enable intelligence in the digital mesh of our clients and deliver big impact to them.

YOUR RESPONSIBILITIES:

• The project is executing a data pipeline, which handles data download, decryption, quality checking, cleaning, processing, and prediction including comprehensive error handling, error reporting, and self-recovery mechanisms. Currently, the data pipeline is deployed on an on-premises server, requiring the project team to closely monitor its operations and performance. The project team seeks an experienced data engineer to enhance the existing data pipeline using industry-standard tools and mechanisms.
• Experience with migrating data pipeline to AWS Cloud using AWS managed components
• Proficient in Python, SQL Knowledge, Schell scripts (Linux)
• Proficient in Airflow or similar tool to create pipelines
• Proficient in AWS tools like S3, Glue, AWS EMR, Lambda, Step functions, Fargate, AWS Batch
• Familiar with tools like AWS Batch, Athena, file formats like Parquet, NoSQL knowledge and tools like AWS OpenSearch

WHO YOU ARE:

• You hold a Bachelor's, Master's, or Ph.D. degree in IT, Information Management, and/or Computer Science with at least 7 years of experience.
• Good knowledge of the big data technology landscape and concepts related to distributed storage/computing
• Experience with big data frameworks (e.g. Hadoop, Spark) and distributions (Cloudera, Hortonworks, MapR)
• Experience with batch & ETL jobs to ingest and process data from multiple data sources
• Experience withNoSQLdatabases (e.g. Cassandra, MongoDB, Neo4J, ElasticSearch)
• Experience with querying tools (e.gHive,SparkSQL,Impala)
• Experience with Power BI
• Experience or willingness to go in real-time stream processing, using solutions such as Kafka, AWS Kinesis, Flume, and/or Spark Streaming
• Experience or willingness to learn about DevOps and DataOps principles (e.g Infrastructure as Code, automating different parts of the data pipeline)
• High-level understanding of Data Science concepts and methodologies (how models are built, trained, and deployed)
• You are passionate about technology and continuous learning comes naturally to you

We will be prioritizing applicants who have a current right to work in Singapore, and do not require sponsorship of a visa.

Please note that this will be a 6 months contract, potential of being extended.

MORE ABOUT PALO IT:

We’re eager to adapt to change, learn from our experiences and move to meet our planet’s urgent needs. We are continuously taking action to:

• Attain 50% of revenue from projects with a positive impact
• Train 100% of our workforce on impact
• Achieve B Corp certification among all our offices across the globe
• Continuously measure & improve employee happiness

Our clients are amongst the world's most successful companies. We innovate with both established Fortune 1000s, SMEs and start-ups who aim to make an impact, become global leaders and address the world's most complex challenges.

What We Offer

• Stimulating working environments
• Unique career path
• International mobility
• Internal R&D projects
• Knowledge sharing
• Personalized training

For more on our team culture and benefits, check out ourcareers page .

PALO IT Singapore is an equal opportunity employer. Employment decisions will be based on merit, qualifications and abilities. Palo IT SG does not discriminate in employment opportunities or practices on the basis of race, colour, religion, sex, sexuality, national origin, age, disability, marital status or any other characteristics protected by law.

Protecting your privacy and the security of yourdataare longstanding top priorities for Palo-IT.

Your personaldatawill be processed for the purposes of managing Palo-IT’s recruitment related activities, which include setting up and conducting interviews and tests for applicants, evaluating and assessing the results, and as is otherwise needed in the recruitment and hiring processes.

Please consult ourPrivacy Notice to know more about how we collect, use, and transfer the personaldataof our candidates. Here you can find how you can request for access, correction and/or withdrawal of your PersonalData.

Apply for this job

*

indicates a required field

First Name *

Last Name *

Preferred First Name

Email *

Phone

Resume/CV

Enter manually

Accepted file types: pdf, doc, docx, txt, rtf

Enter manually

Accepted file types: pdf, doc, docx, txt, rtf

LinkedIn Profile

Website

How did you hear about this job?

Are you eligible to work in the country of the role that you applied for? * Select...

Select...

Protecting your privacy and the security of your data are longstanding top priorities for Palo-IT.

Your personal data will be processed for the purposes of managing Palo-IT’s recruitment related activities, which include setting up and conducting interviews and tests for applicants, evaluating and assessing the results, and as is otherwise needed in the recruitment and hiring processes.

Please consult our Privacy Notice to know more about how we collect, use, and transfer the personal data of our candidates.

Here you can find how you can request for access, correction and/or withdrawal of your Personal Data.