Big Data Engineer

Recooty; Chicago (IL); Remote; USD 90,000 - 150,000; Full time

Job summary

An established industry player is seeking a talented Sr. Big Data Engineer to join their innovative team. In this dynamic role, you'll leverage your expertise in Spark, Scala, and Azure to build robust analytics and machine learning platforms. Your contributions will directly impact data-driven products and micro-services, enabling the organization to harness vast datasets effectively. This position offers a unique opportunity to work on exciting projects, including personalization initiatives and the migration to a new data platform. If you're passionate about big data and eager to make a difference, this is the perfect opportunity for you!

Qualifications

• 5-8 Jahre Erfahrung in der Softwareentwicklung, insbesondere in Big Data Engineering.
• Starke Kenntnisse in Spark, Scala, SQL und Azure sind erforderlich.

Responsibilities

• Entwicklung und Verwaltung von Datenpipelines für die Verarbeitung von Daten in Echtzeit.
• Zusammenarbeit mit Produktmanagern und Data Scientists zur Umsetzung von Anforderungen.

Skills

Big Data Engineering

Spark

Scala

SQL

Azure

Java

Machine Learning

NoSQL Databases

Kafka

Hadoop Ecosystem

Education

Bachelor's Degree in Computer Science or related field

Tools

Azure Cloud

Spark Streaming

Flink

Apache Beam

Cassandra

HBase

MongoDB

Couchbase

Job description

Job Title: Sr. Big Data Engineer

Location: San Francisco, CA (open to remote)

Duration: 6 months (will extend; we have multiple consultants on this team that have been there 2+ years)

Interview: 2 rounds (1st round 1-hour video technical interview, 2nd round 30 min formality personality call)

Looking for a strong Big Data Engineer with experience in Spark, Scala, SQL, and Azure. The Architecture and Platform Organizations are seeking an experienced Big Data Engineer to build analytics and ML platforms to collect, store, process, and analyze huge sets of data spread across the organization. The platform will provide frameworks for quickly rolling out new data analysis for data-driven products and micro-services.

The platform will enable machine/deep learning infrastructure that operationalizes data science models for broad consumption. You'll partner with end-to-end Product Managers and Data Scientists to understand customer requirements and design prototypes and bring ideas into production. You need to be an expert in design, coding, and scripting, writing high-quality code consistent with our standards, creating new standards as necessary, and demonstrating correctness with pragmatic automated tests. You'll review the work of other engineers to improve quality and engineering practices and participate in continuing education programs to grow your skills as a member of an Agile Engineering team.

Ideally, you should have 5-8 years of experience as a Software Engineer, with experience in building distributed, scalable, and reliable data pipelines that ingest and process data at scale, both in batch and real-time. Strong knowledge of programming languages/tools including Java, Scala, Spark, SQL, Hive, and ElasticSearch is essential. Familiarity with most tools within the Hadoop Ecosystem is necessary, particularly Spark and Scala (Java if not Scala). Experience with streaming technologies such as Spark Streaming, Flink, or Apache Beam, along with Kafka, is a plus. Working experience with various NoSQL databases such as Cassandra, HBase, MongoDB, and/or Couchbase would be beneficial. Prior knowledge in Machine Learning or Deep Learning is a plus (this will be learned on the job).

You will be working with the Marketing and Supply Chain side on a Personalization initiative, managing data feeds to and from 3rd party vendors doing analytics, marketing, and operations for email and catalog campaigns. Eventually, you will get into Machine Learning in areas of Product Recommendations on the site.

The team is working in Spark in Scala to ingest transaction and clickstream data to generate associations and product recommendations. You will be involved in batch processing and real-time streaming projects, creating Spark Jobs & Azure Cloud using Azure tools for scheduling and workflow management for batch jobs. The team is currently migrating from Teradata to Microsoft Azure, building a new Data Platform using Spark and developing a data pipeline from transactional systems processed in Spark (framework written in Scala or Java).

Key Responsibilities:

1. Basic Transformations like filter, map & Actions like count, Group by, etc using Dataframe API
2. Iterating over Scala collections
3. Spark Parallelism – Data Ingestion from External RDBMS, Local Transformations
4. Datawarehouse – Dimensions, Facts when to do full load vs Incremental, etc.
5. Basic software engineering principles.

Regards,
Rakesh Kumar