Sr. Data Engineer

Elliot Systems; United States; Remote; USD 90,000 - 130,000; Full time

Job summary

Join a forward-thinking company as a Data Engineer, where you will leverage your extensive experience in Spark and Scala to build robust data pipelines. This role involves working with cutting-edge technologies like Apache Kafka and AWS services to process large-scale data in real-time. You will collaborate within an Agile framework, contributing to innovative solutions that drive business success. If you are passionate about data engineering and eager to make an impact, this opportunity offers a dynamic environment for growth and creativity.

Qualifications

• 5+ years experience in Spark & Scala for big data pipelines.
• Experience with NoSQL systems and AWS services.
• Strong communication skills and work ethic.

Skills

Spark & Scala

Apache Flink

Kafka/Pulsar

Data Pipeline Development

ETL

NoSQL (MongoDB, DynamoDB)

AWS Services

Java

Git

Agile Framework

Job description

Skills, accomplishments, interests you should have:

• 5+ years core Spark & Scala experience: building business logic layers and high-volume/low latency/big data pipelines.
• 5+ years of experience in large scale real-time stream processing: using Apache Flink or Apache Spark with messaging infrastructure like Kafka/Pulsar.
• 7+ years of experience in Data Pipeline development: ETL and processing of structured and unstructured data.
• 2+ years of experience with NoSQL systems: such as MongoDB, DynamoDB, and Relational SQL Database systems like PostgreSQL and Athena.
• 2+ years of experience with AWS services: including Lambda, SNS, S3, Batch Jobs, Step Functions, EC2, ECR, etc.
• Excellent written and verbal communication skills in English.
• Strong work ethic and entrepreneurial spirit.

Additional skills set that are a must:

• Spark with Scala
• Apache Kafka
• NoSQL DB experience like MongoDB
• Experience in AWS cloud services.
• Java (5 years)
• Git (GitLab preferred)
• Experience working in an Agile framework/team