Senior Data Engineer: Real-Time Pipelines with Spark&Kafka

Compunnel, Inc.; Bentonville (AR); On-site; USD 100,000 - 130,000; Full time

Job summary

Compunnel, Inc. is looking for a Data Engineer in Bentonville, Arkansas to develop scalable data pipelines using Spark, Kafka, and GCP. The role involves optimizing data architectures and implementing quality checks for accurate data analytics and machine learning.

The ideal candidate should have at least 8 years of IT experience, with strong skills in Spark, Python, SQL, and a solid understanding of cloud services. Familiarity with tools like Airflow and Docker is preferred.

Qualifications

• Minimum 8 years of IT experience.
• At least 2 years of Scala experience.
• 4+ years of recent GCP experience.
• Strong proficiency in Python and SQL.
• Expertise in Apache Spark and DataFrames.
• Experience with Apache Kafka or Pub/Sub.
• Familiarity with GCP and Azure data services.

Responsibilities

• Design and maintain ETL/ELT data pipelines using Spark and Kafka.
• Build scalable data architectures and optimize Spark jobs.
• Implement data quality checks and monitoring systems.

Skills

Spark

Streaming

Python

SQL

Apache Kafka

Data Lakes

BigQuery

Airflow

Databricks

Docker

Kubernetes

Job description

Compunnel, Inc. is looking for a Data Engineer in Bentonville, Arkansas to develop scalable data pipelines using Spark, Kafka, and GCP. The role involves optimizing data architectures and implementing quality checks for accurate data analytics and machine learning.

The ideal candidate should have at least 8 years of IT experience, with strong skills in Spark, Python, SQL, and a solid understanding of cloud services. Familiarity with tools like Airflow and Docker is preferred.