Real-Time Data Engineer: Kafka & Spark, REST APIs

Stepping Up Solutions; San Francisco (CA); On-site; USD 120,000 - 150,000; Full time

Job summary

Stepping Up Solutions is seeking a skilled Data Engineer to join their Data Technology team. The successful candidate will design and implement scalable data infrastructure while ensuring the efficiency of data transformations developed by data scientists.

The role involves optimizing a data processing portal for real-time analytics and maintaining data in both MySQL and NoSQL systems. Candidates should have significant experience in a data-intensive environment and proficiency in Java or Python.

Qualifications

• 5+ years working in a data-intensive environment.
• 2+ years experience in distributed data platforms.
• Expert in SQL - MySQL preferred.
• 4+ years experience in Java or Python.
• Experience building RESTful APIs.

Responsibilities

• Take ownership of the data processing portal for real-time analytics.
• Migrate data processing from AWS to in-house servers.
• Script and analyze data, and query databases.
• Optimize and maintain the portal.

Skills

Data warehouse experience

Distributed data platforms

SQL proficiency

Java or Python programming

RESTful API development

Agile development practices

Tools

Hadoop

Kafka

Apache Spark

MySQL

NoSQL

Job description

Stepping Up Solutions is seeking a skilled Data Engineer to join their Data Technology team. The successful candidate will design and implement scalable data infrastructure while ensuring the efficiency of data transformations developed by data scientists.

The role involves optimizing a data processing portal for real-time analytics and maintaining data in both MySQL and NoSQL systems. Candidates should have significant experience in a data-intensive environment and proficiency in Java or Python.