Python Developer: Spark, Airflow & AWS Data Pipelines

Ethereum Technologies; Wilmington (DE); On-site; USD 120,000 - 170,000; Full time

Job summary

Ethereum Technologies seeks a skilled Python data engineer to design and optimize data pipelines, leveraging PySpark, Spark SQL and AWS data services to build scalable, reliable data processing solutions in a fast-paced environment.

You will collaborate with ML and DevOps teams, implement CI/CD, and ensure code quality with PyTest. Prior experience with Airflow, Kafka and cloud data warehouses will help accelerate delivery and drive data-driven decisions.

Qualifications

• Expertise in Python and data processing.
• Strong PySpark and Spark SQL experience.
• Proficiency with Pandas/Numpy; familiar with Flask or Django.
• Hands-on with Airflow, Kafka, Jenkins, Spinnaker.
• AWS cloud services (S3, EC2, Lambda, ECS, EventBridge).
• CI/CD and DevOps practices for automated deployments.
• Cloud data warehouses like Snowflake/Databricks preferred.
• Exposure to ML frameworks is a plus.
• Familiar with AI tools such as GitHub Copilot and LLMs.

Responsibilities

• Develop and maintain Python data processing pipelines.
• Design scalable, high-performance data solutions on AWS.
• Collaborate with ML and DevOps teams to deploy features.

Skills

Python

PySpark

Spark SQL

PyTest

Pandas/Numpy

Flask/Django

Airflow

Kafka

Jenkins

Spinnaker

AWS S3/EC2/Lambda/ECS/EventBridge

CI/CD & DevOps

Snowflake/Databricks

ML exposure

GitHub Copilot/LLMs

Tools

Airflow

Kafka

Jenkins

Spinnaker

Snowflake

Databricks

GitHub Copilot

Job description

Ethereum Technologies seeks a skilled Python data engineer to design and optimize data pipelines, leveraging PySpark, Spark SQL and AWS data services to build scalable, reliable data processing solutions in a fast-paced environment.

You will collaborate with ML and DevOps teams, implement CI/CD, and ensure code quality with PyTest. Prior experience with Airflow, Kafka and cloud data warehouses will help accelerate delivery and drive data-driven decisions.