Data Engineer

Pyramid Consulting, Inc; North Carolina; On-site; USD 69,000 - 83,000; Full time

Benefits offered by this job

Health insurance

401(k) plan

Paid sick leave

Job summary

Pyramid Consulting, Inc. is seeking a Data Engineer for a 12+ month contract opportunity located onsite in the United States. The role involves designing and maintaining scalable data pipelines using Python, Spark, Kafka, Airflow, and AWS services, with emphasis on Databricks and data warehousing.

Responsibilities include building ETL/ELT pipelines, real-time and batch processing, and collaboration with data scientists and stakeholders to ensure data quality and performance across platforms.

Qualifications

• Must have experience designing scalable data pipelines with Python and Spark.
• Experience with Kafka, Airflow, and AWS data services.
• Experience with Databricks and data warehousing.
• Onsite from Day 1 in United States.

Responsibilities

• Design, develop, and maintain scalable data pipelines using Python, PySpark, and Apache Spark.
• Build and optimize ETL/ELT pipelines to ingest, transform, and process large volumes of structured and unstructured data.
• Develop real-time and batch data pipelines using Apache Kafka and Airflow for data ingestion and workflow orchestration.
• Build and manage cloud-based data solutions using AWS S3, Glue, EMR, and Redshift.
• Develop and optimize data processing workloads using Databricks and PySpark.
• Write complex SQL queries and optimize data processing for performance and scalability.
• Design and maintain data warehouses, data models, and scalable data architectures.
• Develop reusable data ingestion and transformation frameworks to support analytics and reporting requirements.
• Implement workflow scheduling, dependency management, monitoring, and failure handling using Airflow and other orchestration tools.
• Ensure data quality, accuracy, reliability, and consistency across data pipelines and downstream systems.
• Collaborate with data scientists, analysts, architects, and business stakeholders to understand data requirements and deliver scalable data solutions.
• Troubleshoot pipeline failures and performance issues and continuously improve the reliability and efficiency of data platforms.

Skills

Python

Apache Spark

SQL

Kafka

Airflow

AWS

Databricks

ETL/ELT pipelines

data warehousing

data modeling

orchestration tools

Tools

PySpark

Job description

Immediate need for a talented Data Engineer. This is a 12+ months contract opportunity with long-term potential and is located in United States (onsite). Please review the job description below and contact me ASAP if you are interested.

Job Diva Id- 26-25001

Pay Range: $50 - $60/hour. Employee benefits include, but are not limited to, health insurance (medical, dental, vision), 401(k) plan, and paid sick leave (depending on work location).

Key Responsibilities

• Note: Candidates must be willing to work onsite from Day 1
• Design, develop, and maintain scalable data pipelines using Python, PySpark, and Apache Spark.
• Build and optimize ETL/ELT pipelines to ingest, transform, and process large volumes of structured and unstructured data.
• Develop real-time and batch data pipelines using Apache Kafka and Airflow for data ingestion and workflow orchestration.
• Build and manage cloud-based data solutions using AWS S3, Glue, EMR, and Redshift.
• Develop and optimize data processing workloads using Databricks and PySpark.
• Write complex SQL queries and optimize data processing for performance and scalability.
• Design and maintain data warehouses, data models, and scalable data architectures.
• Develop reusable data ingestion and transformation frameworks to support analytics and reporting requirements.
• Implement workflow scheduling, dependency management, monitoring, and failure handling using Airflow and other orchestration tools.
• Ensure data quality, accuracy, reliability, and consistency across data pipelines and downstream systems.
• Collaborate with data scientists, analysts, architects, and business stakeholders to understand data requirements and deliver scalable data solutions.
• Troubleshoot pipeline failures and performance issues and continuously improve the reliability and efficiency of data platforms.

Key Requirements and Technology Experience

• Must have skills: - Python, Apache Spark, SQL, Kafka, Airflow, AWS (S3, Glue, EMR, Redshift), Databricks, ETL/ELT pipelines, data warehousing, PySpark, data modeling, orchestration tools

Our client is a leading IT Industry, and we are currently interviewing to fill this and other similar contract positions.

Pyramid Consulting, Inc. provides equal employment opportunities to all employees and applicants for employment and prohibits discrimination and harassment of any type without regard to race, colour, religion, age, sex, national origin, disability status, genetics, protected veteran status, sexual orientation, gender identity or expression, or any other characteristic protected by federal, state, or local laws.