Senior Data Engineer

Compunnel, Inc.; Bentonville (AR); On-site; USD 100,000 - 130,000; Full time

Job summary

Compunnel, Inc. is looking for a Data Engineer in Bentonville, Arkansas to develop scalable data pipelines using Spark, Kafka, and GCP. The role involves optimizing data architectures and implementing quality checks for accurate data analytics and machine learning.

The ideal candidate should have at least 8 years of IT experience, with strong skills in Spark, Python, SQL, and a solid understanding of cloud services. Familiarity with tools like Airflow and Docker is preferred.

Qualifications

• Minimum 8 years of IT experience.
• At least 2 years of Scala experience.
• 4+ years of recent GCP experience.
• Strong proficiency in Python and SQL.
• Expertise in Apache Spark and DataFrames.
• Experience with Apache Kafka or Pub/Sub.
• Familiarity with GCP and Azure data services.

Responsibilities

• Design and maintain ETL/ELT data pipelines using Spark and Kafka.
• Build scalable data architectures and optimize Spark jobs.
• Implement data quality checks and monitoring systems.

Skills

Spark

Streaming

Python

SQL

Apache Kafka

Data Lakes

BigQuery

Airflow

Databricks

Docker

Kubernetes

Job description

JOB SUMMARY

Seeking a Data Engineer with Spark & Streaming skills to build real-time, scalable data pipelines using tools like Spark, Kafka, and cloud services (GCP) to ingest, transform, and deliver data for analytics and ML.

Key Responsibilities

• Design, develop, and maintain ETL/ELT data pipelines for batch and real-time data ingestion, transformation, and loading using Spark (PySpark/Scala) and streaming technologies (Kafka, Flink).
• Build and optimize scalable data architectures, including data lakes, data warehouses (BigQuery), and streaming platforms.
• Optimize Spark jobs, SQL queries, and data processing workflows for speed, efficiency, and cost-effectiveness.
• Implement data quality checks, monitoring, and alerting systems to ensure data accuracy and consistency.

Required Qualifications

• Total IT Experience: Minimum 8 years.
• Scala: Minimum 2 years of experience.
• GCP: 4+ years of recent GCP experience.
• Programming: Strong proficiency in Python, SQL.
• Big Data: Expertise in Apache Spark (Spark SQL, DataFrames, Streaming).
• Streaming: Experience with messaging queues like Apache Kafka, or Pub/Sub.
• Cloud: Familiarity with GCP, Azure data services.
• Databases: Knowledge of data warehousing (Snowflake, Redshift) and NoSQL databases.

Preferred Qualifications

• Programming: Proficiency in Scala/Java.
• Tools: Experience with Airflow, Databricks, Docker, Kubernetes.