Big Data Hadoop

Arete Technologies, Inc.; Atlanta (GA); On-site; USD 100,000 - 140,000; Full time

Job summary

Arete Technologies, Inc. is seeking a Big Data Engineering Lead in Atlanta, GA, specializing in data ingestion from diverse systems. The role involves collaboration in developing the NCR Data Lake and inspiring the engineering team towards technical excellence.

The ideal candidate will have a Bachelor's degree in Computer Science and extensive experience with tools like Apache NiFi and Kafka. Familiarity with Agile practices and DevOps is also required.

Qualifications

• Good understanding of distributed computing and big-data architectures.
• Proven experience in developing Big Data solutions in the Hadoop Ecosystem.
• Experience with at least one NoSQL database like HBase or Cassandra.
• DevOps experience with CI and CD principles.

Responsibilities

• Lead a Big Data engineering team with data ingestion focus.
• Participate in collaborative design of the NCR Data Lake.
• Inspire team with an engineering mindset and systems thinking.

Skills

Apache NiFi

Kafka

Flume

Sqoop

Hive

HDFS

HBase

Spark

Java

Scala

DevOps

Education

Bachelor’s degree in Computer Science or related field

Tools

Maven

Git

Docker

Talend

Job description

Arete Technologies, Inc. offers a set of innovative consulting and outsourcing services, bridging the gap between requirements and outputs for companies worldwide. We provide global deliverables with a focus on delivering unparalleled services combined with cost‑saving solutions.

Job Description

Apache NiFi, Kafka, Flume, Sqoop, Apache Atlas, Hive, HDFS, HBase and Spark (Hortonworks HDP and HDF preferred).

Lead Big Data engineering team with specialization on data ingestion (from 100+ source systems in batch and near real‑time), egestion and governance.

Participate in collaborative software and system design and development of the new NCR Data Lake on Hortonworks HDP and HDF distributions.

Manage own learning and contribute to technical skill building of the team.

Inspire and cultivate the engineering mindset and systems thinking.

Gain deep technical expertise in the data movement patterns, practices and tools.

Play an active role in Big Data Communities of Practice.

Put the minimal system needed into production.

Required Qualifications

• Bachelor’s degree or higher in Computer Science or a related field.
• Good understanding of distributed computing and big‑data architectures.
• Passion for software engineering and craftsman‑like coding prowess.
• Proven experience in developing Big Data solutions in the Hadoop Ecosystem using Apache NiFi, Kafka, Flume, Sqoop, Apache Atlas, Hive, HDFS, HBase and Spark (Hortonworks HDP and HDF preferred).
• Experience with at least one of the leading CDC (Change Data Capture) tools like Informatica PowerCenter.
• Development experience with at least one NoSQL database. HBase or Cassandra preferred.
• Polyglot development (4–5 years+): Capable of developing in Java and Scala with good understanding of functional programming, SOLID principles, concurrency models and modularization.
• DevOps: Appreciates the CI and CD model and always builds to ease consumption and monitoring of the system. Experience with Maven (or Gradle or SBT) and Git preferred.
• Experience in Agile development including Scrum and other lean techniques.
• Belief in “You Build! You Ship! And You Run!” philosophy.
• Personal qualities such as creativity, tenacity, curiosity, and passion for deep technical excellence.

Desired Qualifications

• Experience with Big Data migrations/transformations programs in the Data Warehousing and/or Business Intelligence areas.
• Experience with ETL tools like Talend, Pentaho, Attunity etc.
• Knowledge of Teradata, Netezza etc.
• Good grounding in NoSQL data stores such as Cassandra, Neo4j etc.
• Strong knowledge of computer algorithms.
• Experience with workload orchestration and automation tools like Oozie, Control‑M etc.
• Experience in building self‑contained applications using Docker, Vagrant, Chef.

Additional Information

All your information will be kept confidential according to EEO guidelines.