Big Data Engineer, Data Ingestion Singapore Regular

Pangle; Singapore; On-site; SGD 90,000 - 140,000; Full time

Job summary

ByteDance in Singapore is seeking a Software Engineer for the Data Management Suite to design and optimize large-scale data ingestion systems, enabling real-time pipelines and fault tolerance.

You will work with cross-functional teams to deliver event data ingestion, transformation, and storage solutions, applying knowledge of big data tech and distributed systems to impact hundreds of millions of users.

Qualifications

• Bachelor's degree in Computer Science or equivalent practical experience.
• 2+ years of experience in software development with proficiency in one or more programming languages (Java/Scala/Python/Go).
• Strong understanding of data structures, algorithms, and distributed systems principles.
• Hands-on experience with big data technologies such as Hadoop, Flink, Kafka, or similar frameworks.

Responsibilities

• Design, develop, and optimize the architecture of large-scale data ingestion systems to support real-time data pipelines, ensuring high throughput, low latency, and fault tolerance.
• Enhance the performance, scalability, and reliability of data ingestion pipelines.
• Develop and implement automated and intelligent operation and maintenance systems to monitor, diagnose, and ensure the stability and reliability of data ingestion pipelines.
• Collaborate with cross-functional teams to deliver event data ingestion, transformation, and storage solutions that meet diverse business requirements.
• Troubleshoot and resolve complex issues in production systems, ensuring minimal downtime and optimal performance.

Skills

Java

Scala

Python

Go

Distributed systems

Education

Bachelor's degree in Computer Science

Tools

Hadoop

Flink

Kafka

Job description

Join us as we work together to inspire creativity and enrich life around the globe.

The Data Management Suite team is building products that cover the whole lifecycle of data pipeline, including data ingestion and Integration, data development, data catalog, data security and data governance. These products support various businesses, so data engineers and data scientists could greatly boost their productivity.As a software engineer in the data management suite team, you will have the opportunity to build, optimize and grow one of the largest data platforms in the world. You'll have the opportunity to gain hands-on experience on core systems in the data platform ecosystem. Your work will have a direct and huge impact on the company's core products as well as hundreds of millions of users.

Responsibilities

• Design, develop, and optimize the architecture of large-scale data ingestion systems to support real-time data pipelines, ensuring high throughput, low latency, and fault tolerance.
• Enhance the performance, scalability, and reliability of data ingestion pipelines.
• Develop and implement automated and intelligent operation and maintenance systems to monitor, diagnose, and ensure the stability and reliability of data ingestion pipelines.
• Collaborate with cross-functional teams to deliver event data ingestion, transformation, and storage solutions that meet diverse business requirements.
• Troubleshoot and resolve complex issues in production systems, ensuring minimal downtime and optimal performance.

Qualifications

Minimum Qualification(s)

• Bachelor's degree in Computer Science or equivalent practical experience.
• 2+ years of experience in software development with proficiency in one or more programming languages, such as Java, Scala, Python, or Go.
• Strong understanding of data structures, algorithms, and distributed systems principles.
• Hands-on experience with big data technologies, such as Hadoop, Flink, Kafka, or similar frameworks.

Preferred Qualification(s)

• 2+ years of experience in real-time data processing frameworks like Flink. Contributions to open-source projects are a plus.
• Experience in the event tracking domain, including event collection, real-time processing, governance, quality assurance and cost optimization.
• Deep understanding of data lakehouse architectures and the integration of stream and batch processing.
• Strong problem-solving skills and the ability to work in a collaborative environment.

Job Information

About Us

Why Join ByteDance

Inspiring creativity is at the core of ByteDance's mission. Our innovative products are built to help people authentically express themselves, discover and connect – and our global, diverse teams make that possible. Together, we create value for our communities, inspire creativity and enrich life - a mission we work towards every day.

As ByteDancers, we strive to do great things with great people. We lead with curiosity, humility, and a desire to make impact in a rapidly growing tech company. By constantly iterating and fostering an "Always Day 1" mindset, we achieve meaningful breakthroughs for ourselves, our Company, and our users. When we create and grow together, the possibilities are limitless. Join us.

Diversity & Inclusion

ByteDance is committed to creating an inclusive space where employees are valued for their skills, experiences, and unique perspectives. Our platform connects people from across the globe and so does our workplace. At ByteDance, our mission is to inspire creativity and enrich life. To achieve that goal, we are committed to celebrating our diverse voices and to creating an environment that reflects the many communities we reach. We are passionate about this and hope you are too.