Senior Big Data Engineer – HDFS/Spark/Hive/Oozie

SmartRecruiters, Inc.; Durham (NC); On-site; USD 110,000 - 170,000; Full time

Job summary

Sonsoft , Inc. is seeking a senior data/ETL engineer with 7+ years of experience in big data environments including HDFS, Hive, Spark, and Oozie. Hands-on expertise with MapReduce, Sqoop and Cloudera distribution is required.

Candidates should be proficient in Java, Python, and Perl and understand warehousing concepts like de-duplication, cleansing, lookups and data transformation. U.S. work eligibility is required; sponsorship is not provided.

Qualifications

• 7+ years of overall IT experience.
• Hands-on with HDFS architecture and concepts: YARN, ZooKeeper, MapReduce.
• Hands-on with Hive: queries, UDFs, ORC/AVRO/Parquet formats.
• Hands-on with Sqoop and Spark.
• Experience with warehousing concepts: de-duplication, cleansing, lookups, transformation, data versioning.
• Hands-on with Oozie workflow definitions.
• Knowledge of Cloudera distribution; monitoring/operational capabilities.
• Knowledge of Flume and Kafka (plus).
• Proficiency in Java, Python, and Perl.

Skills

7+ years experience

Java/Python/Perl

Education

Bachelor’s degree or foreign equivalent

Three years of progressive experience in lieu of education

Tools

HDFS

YARN

ZooKeeper

MapReduce

Hive

UDFs

ORC/AVRO/Parquet

Sqoop

Spark

Oozie

Cloudera

Flume

Kafka

Job description

Sonsoft , Inc. is seeking a senior data/ETL engineer with 7+ years of experience in big data environments including HDFS, Hive, Spark, and Oozie. Hands-on expertise with MapReduce, Sqoop and Cloudera distribution is required.

Candidates should be proficient in Java, Python, and Perl and understand warehousing concepts like de-duplication, cleansing, lookups and data transformation. U.S. work eligibility is required; sponsorship is not provided.