Data Engineer - Big Data Platforms

Placements24; United States; Hybrid; USD 54,000 - 97,000; Full time

Benefits offered by this job

Medical aid contributions

Provident fund

Generous paid vacation time

Training programs

Industry conferences

Performance bonuses

Job summary

Placements24 is seeking an experienced Data Engineer to join a rapidly growing technology firm focused on data solutions in Paarl, Western Cape. You will design, develop, and maintain scalable big data platforms and data pipelines that ingest, transform, and store large volumes of data to enable analytics and machine learning initiatives.

The ideal candidate has hands-on experience with distributed systems, cloud data services, and programming languages such as Python, Scala, or Java.

Qualifications

• Bachelor’s degree in Computer Science or related field.
• Proven data engineering experience with big data tech.
• Strong SQL and database knowledge.

Responsibilities

• Design, build, and maintain scalable data pipelines.
• Develop data ingestion, transformation, and loading processes.
• Implement data warehousing and manage data storage systems.
• Ensure data quality, integrity and security across platforms.
• Collaborate with data scientists and engineers to meet data requirements.
• Monitor and troubleshoot pipeline performance; resolve issues.

Skills

SQL

Big data

Python

Scala

Java

Airflow

Education

Bachelor's degree in Computer Science/Engineering/IT

Tools

Apache Spark

Hadoop

Kafka

S3

BigQuery

Job description

About the Role

Our client, a rapidly growing technology firm specializing in data solutions, is seeking an experienced Data Engineer to join their team. This position is based in Paarl, Western Cape , and involves the design, development, and maintenance of robust, scalable big data platforms. You will play a key role in building and optimizing data pipelines that ingest, transform, and store vast amounts of data, enabling advanced analytics and machine learning initiatives across the organization. The ideal candidate has a strong background in data architecture, distributed systems, and programming, with hands-on experience with cloud-based data technologies. This is an excellent opportunity to work with cutting-edge technologies in a dynamic and collaborative environment, contributing directly to the company's data infrastructure and strategy. You will be instrumental in ensuring data availability and quality for critical business decisions.

Key Responsibilities

• Design, build, and maintain scalable and reliable big data pipelines using distributed computing frameworks.
• Develop and optimize data ingestion, transformation, and loading processes.
• Implement data warehousing solutions and manage data storage systems.
• Ensure data quality, integrity, and security across all data platforms.
• Collaborate with data scientists, analysts, and other engineers to understand data requirements and deliver solutions.
• Monitor and troubleshoot data pipeline performance, identifying and resolving issues proactively.
• Implement and manage ETL/ELT processes.
• Stay current with emerging big data technologies and best practices.
• Document data architectures, processes, and standards.

Requirements

• Bachelor's degree in Computer Science, Engineering, Information Technology, or a related field.
• Proven experience as a Data Engineer or in a similar role, with a focus on big data technologies.
• Strong proficiency in SQL and experience with various database technologies (relational, NoSQL).
• Hands-on experience with big data technologies such as Apache Spark, Hadoop, Kafka, or similar.
• Experience with cloud platforms (AWS, Azure, or GCP) and their data services (e.g., S3, Redshift, Data Factory, BigQuery).
• Proficiency in programming languages like Python, Scala, or Java.
• Understanding of data modeling, data warehousing concepts, and ETL/ELT principles.
• Experience with workflow management tools (e.g., Airflow) is a plus.
• Excellent problem-solving and analytical skills.
• Strong communication and collaboration abilities.

Benefits

Our client offers a competitive salary package, complemented by a comprehensive benefits scheme. This includes medical aid contributions, a provident fund, and generous paid vacation time. We encourage professional development through training programs, workshops, and opportunities to attend industry conferences. The role is situated in the beautiful Paarl, Western Cape region, offering a stimulating work environment. The hybrid nature of this position allows for flexibility between working from home and collaborating in the office. Additional benefits include performance bonuses and a supportive team culture focused on innovation and continuous improvement.