Data Engineer - Real-Time, 24/7 Big Data Pipelines

Roberts Recruiting, LLC; Boston (MA); On-site; USD 80,000 - 110,000; Full time

Job summary

A recruiting company is seeking a data processing expert to design and develop scalable data and reporting services. Candidates should have strong SQL skills and experience with Hadoop and distributed computing technologies, along with a background in producing data solutions for 24×7 environments. The position offers challenges in data processing with high volumes and complex reporting needs.

Qualifications

• Hands-on data processing experience required.
• Familiarity with distributed computing technologies is essential.
• Experience with 24×7 production environments necessary.

Responsibilities

• Design and develop scalable data and reporting services.
• Develop enhancements to Hadoop/Hive data pipeline.
• Implement new services and algorithms for product improvement.

Skills

Strong SQL skills

Data processing experience

Experience with Linux/Unix

Strong software development skills

Education

Degree in Computer Science or related field

Tools

Hadoop

Java

MySQL

AWS

Job description

The ideal candidate has hands-on data processing experience. Candidate must be familiar with current distributed computing data technologies on commodity servers and have experience with 24×7 production needs.

Requirements

• Strong SQL skills on large-scale databases, knowledge of any of the following RDBMS such as MySQL, Postgres, Redshift, Hive or others.
• Experience with Linux or Unix based systems – including Bourne shell, cron and other Unix utilities.
• Strong software development skills, including experience with Java.
• Degree in Computer Science or a related field

Desired Technology Experience

• Experience with at least one of Ruby/Python, Bourne Shell or other scripting languages.
• Experience with Hadoop/MapReduce and/or EMR, including experience in developing MapReduce Jobs in Java or developing Hive UDF.
• ETL experience maintaining multiple data systems.
• Experience with Ooozie or other Hadoop workflow solutions and experience developing complex data processing pipelines, including experience developing regressions tests and deployment strategies for such environments.
• Experience with data reporting solutions – either developed in house or with 3rd party solutions.
• Working experience developing and supporting 24×7 production data services and pipelines on Linux systems – including experience being on-call supporting such services. Experience with AWS preferred.

Challenges we are tackling

• Reliably processing billions of events per day with no data loss 24×7 on commodity hardware.
• Processing events in near real-time
• Building for the fragility of cloud and distributed services
• Complex processing of large amounts of data in an efficient manner.
• Reporting, distribution of data, data analysis, data visualization and machine learning algorithms.
• Low latency data stores for use in bidding or algo optimization.