Senior Big Data Engineer — AWS, Spark & Snowflake Pipelines

Compunnel, Inc.; Costa Mesa (CA); On-site; USD 120,000 - 160,000; Full time

Job summary

Compunnel, Inc. is seeking a Big Data Engineer responsible for designing and implementing scalable platforms within AWS. The role demands strong expertise in technologies including Spark, Hadoop, Scala, and Java, focusing on processing large datasets and building data pipelines.

The ideal candidate will have extensive experience in big data application development, ensuring reliability and performance. Responsibilities include developing microservices architectures and collaborating with cross-functional teams.

Qualifications

• 7+ years of experience in Big Data application development and testing.
• Strong experience working in AWS environments and developing applications.
• Proficiency in Scala, PySpark, and Apache Spark essential for the role.

Responsibilities

• Design and develop scalable big data applications and platforms.
• Implement and maintain data ingestion and processing pipelines.
• Collaborate with teams to capture requirements and deliver solutions.

Skills

Big Data applications development

AWS cloud environments

Scala programming

PySpark

Apache Spark

Java development

Hadoop ecosystem

Cassandra

Data pipeline building

API development

Automation with Python

Microservices architectures

Tools

AWS services

Apache Airflow

CI/CD tools

Job description

Compunnel, Inc. is seeking a Big Data Engineer responsible for designing and implementing scalable platforms within AWS. The role demands strong expertise in technologies including Spark, Hadoop, Scala, and Java, focusing on processing large datasets and building data pipelines.

The ideal candidate will have extensive experience in big data application development, ensuring reliability and performance. Responsibilities include developing microservices architectures and collaborating with cross-functional teams.