Python Engineer: Data Pipelines, Spark & Cloud

Machine Learning Technologies, LLC; Wilmington (DE); On-site; USD 120,000 - 160,000; Full time

Job summary

Machine Learning Technologies, LLC seeks a highly skilled Python developer with hands-on PySpark, AWS, and data-pipeline expertise to design scalable, high-performance applications. You will contribute to testing, debugging, and deploying data-driven solutions using Airflow, Kafka, Jenkins, and Spinnaker, while leveraging Pandas/Numpy, Flask or Django for data processing and APIs.

The role involves working with cloud services (S3, EC2, Lambda, ECS, EventBridge) and modern CI/CD practices, with

Qualifications

• Expert proficiency in Python programming language with strong coding and problem-solving skills.
• Strong experience in PySpark, including Spark SQL and distributed data processing using Spark APIs.
• Experience in testing and debugging applications using Python testing frameworks such as PyTest, PyUnit, etc.
• In-depth knowledge of Python libraries and frameworks like Pandas, Numpy, Flask, or Django.
• Hands-on experience with data pipeline and orchestration tools such as Airflow, Kafka, Jenkins, and Spinnaker.
• Strong knowledge of designing scalable, reliable, and high-performance applications.
• Experience working with AWS cloud services such as S3, EC2, Lambda, ECS, Event Bridge, etc.
• Experience with CI/CD pipelines and DevOps practices for automated deployments.
• Good to have experience with cloud data warehousing platforms such as Snowflake, Databricks.
• Nice to have exposure to Machine Learning and AI frameworks.
• Familiarity with AI-assisted development tools such as GitHub Copilot and understanding of Large Language Models (LLMs) for improving developer productivity and building intelligent features.

Skills

Python

PySpark

PyTest

Pandas

Numpy

Flask

Django

Airflow

Kafka

Jenkins

Spinnaker

AWS

S3

EC2

Lambda

ECS

EventBridge

CI/CD

DevOps

Snowflake

Databricks

ML frameworks

AI frameworks

GitHub Copilot

LLMs

Tools

Python libraries (Pandas, Numpy)

Flask

Django

Job description

Machine Learning Technologies, LLC seeks a highly skilled Python developer with hands-on PySpark, AWS, and data-pipeline expertise to design scalable, high-performance applications. You will contribute to testing, debugging, and deploying data-driven solutions using Airflow, Kafka, Jenkins, and Spinnaker, while leveraging Pandas/Numpy, Flask or Django for data processing and APIs.

The role involves working with cloud services (S3, EC2, Lambda, ECS, EventBridge) and modern CI/CD practices, with