Data Engineer

Compunnel Inc.; Bentonville (AR); On-site; USD 110,000 - 150,000; Full time

Job summary

Compunnel Inc. in Bentonville, AR is looking for a Data Engineer to design, develop, and maintain data pipelines for large-scale analytics.

The role requires strong Python and PySpark skills, Airflow orchestration, and experience with BigQuery and Spark to enable reliable data access for analysts and data scientists.

Qualifications

• Proven experience in Python programming for data engineering tasks.
• Strong hands-on experience with PySpark and distributed data processing.
• Expertise in Airflow for workflow orchestration.
• Proficiency with BigQuery and Spark for large-scale data analytics.
• Solid understanding of data modeling, ETL concepts, and performance optimization.

Responsibilities

• Develop and optimize data pipelines using Python and PySpark.
• Implement workflow orchestration using Airflow for ETL processes.
• Design and maintain scalable data solutions leveraging BigQuery and Spark.
• Collaborate with data scientists and analysts to ensure efficient data access and transformation.
• Ensure data quality, integrity, and compliance with governance standards.
• Monitor and troubleshoot data workflows for performance and reliability.

Skills

Python

PySpark

Airflow

BigQuery

Spark

Data modeling

ETL concepts

Performance optimization

Tools

Kafka

CI/CD pipelines

Job description

We are seeking a skilled Data Engineer to design, develop, and maintain data pipelines and processing frameworks for large-scale analytics.

The ideal candidate will have strong expertise in Python, PySpark, and modern data orchestration tools, with experience in cloud-based data platforms.

Key Responsibilities

• Develop and optimize data pipelines using Python and PySpark.


• Implement workflow orchestration using Airflow for ETL processes.


• Design and maintain scalable data solutions leveraging BigQuery and Spark.


• Collaborate with data scientists and analysts to ensure efficient data access and transformation.


• Ensure data quality, integrity, and compliance with governance standards.


• Monitor and troubleshoot data workflows for performance and reliability.

Required Qualifications

• Proven experience in Python programming for data engineering tasks.


• Strong hands-on experience with PySpark and distributed data processing.


• Expertise in Airflow for workflow orchestration.


• Proficiency with BigQuery and Spark for large-scale data analytics.


• Solid understanding of data modeling, ETL concepts, and performance optimization.

Preferred Qualifications

• Experience with cloud platforms (AWS, GCP, or Azure).


• Familiarity with CI/CD pipelines and DevOps practices for data engineering.


• Knowledge of data streaming tools like Kafka.