Onsite Data Engineer — Spark, Kafka & AWS Pipelines

Pyramid Consulting, Inc; North Carolina; On-site; USD 69,000 - 83,000; Full time

Benefits offered by this job

Health insurance

401(k) plan

Paid sick leave

Job summary

Pyramid Consulting, Inc. is seeking a Data Engineer for a 12+ month contract opportunity located onsite in the United States. The role involves designing and maintaining scalable data pipelines using Python, Spark, Kafka, Airflow, and AWS services, with emphasis on Databricks and data warehousing.

Responsibilities include building ETL/ELT pipelines, real-time and batch processing, and collaboration with data scientists and stakeholders to ensure data quality and performance across platforms.

Qualifications

• Must have experience designing scalable data pipelines with Python and Spark.
• Experience with Kafka, Airflow, and AWS data services.
• Experience with Databricks and data warehousing.
• Onsite from Day 1 in United States.

Responsibilities

• Design, develop, and maintain scalable data pipelines using Python, PySpark, and Apache Spark.
• Build and optimize ETL/ELT pipelines to ingest, transform, and process large volumes of structured and unstructured data.
• Develop real-time and batch data pipelines using Apache Kafka and Airflow for data ingestion and workflow orchestration.
• Build and manage cloud-based data solutions using AWS S3, Glue, EMR, and Redshift.
• Develop and optimize data processing workloads using Databricks and PySpark.
• Write complex SQL queries and optimize data processing for performance and scalability.
• Design and maintain data warehouses, data models, and scalable data architectures.
• Develop reusable data ingestion and transformation frameworks to support analytics and reporting requirements.
• Implement workflow scheduling, dependency management, monitoring, and failure handling using Airflow and other orchestration tools.
• Ensure data quality, accuracy, reliability, and consistency across data pipelines and downstream systems.
• Collaborate with data scientists, analysts, architects, and business stakeholders to understand data requirements and deliver scalable data solutions.
• Troubleshoot pipeline failures and performance issues and continuously improve the reliability and efficiency of data platforms.

Skills

Python

Apache Spark

SQL

Kafka

Airflow

AWS

Databricks

ETL/ELT pipelines

data warehousing

data modeling

orchestration tools

Tools

PySpark

Job description

Pyramid Consulting, Inc. is seeking a Data Engineer for a 12+ month contract opportunity located onsite in the United States. The role involves designing and maintaining scalable data pipelines using Python, Spark, Kafka, Airflow, and AWS services, with emphasis on Databricks and data warehousing.

Responsibilities include building ETL/ELT pipelines, real-time and batch processing, and collaboration with data scientists and stakeholders to ensure data quality and performance across platforms.