Lead Python Platform Engineer - API Layer for Spark & GCP

VMC Soft Technologies, Inc; Charlotte (NC); On-site; USD 120,000 - 180,000; Full time

Job summary

VMC Soft Technologies, Inc. in Charlotte, NC is seeking a Lead Python Platform Engineer to design and build a containerized API layer over Apache Spark, delivering secure, scalable access to Spark-based data pipelines.

You will lead the development of production-grade Python APIs, containerized via Docker/Kubernetes, and implement CI/CD with GitHub Actions while coordinating with onsite and offshore teams.

Qualifications

• At least 10 years of IT experience.
• Proven track record building production-grade Python APIs for data platforms.
• Experience packaging services with containers (Docker/Kubernetes) and running as platform services.
• Strong CI/CD automation experience, e.g., GitHub Actions.
• GCP expertise and data pipeline interaction.

Responsibilities

• Work with onsite team and client to design and govern Spark-based data pipelines via a stable API.
• Lead containerized API layer development and secure, scalable platform services.
• Coordinate offshore resources and ensure timely delivery of complex software projects.

Skills

Python

Pyspark

API

CI/CD

GCP

ETL

Tools

Kubernetes

FastAPI

Pytest

ARQ

Job description

VMC Soft Technologies, Inc. in Charlotte, NC is seeking a Lead Python Platform Engineer to design and build a containerized API layer over Apache Spark, delivering secure, scalable access to Spark-based data pipelines.

You will lead the development of production-grade Python APIs, containerized via Docker/Kubernetes, and implement CI/CD with GitHub Actions while coordinating with onsite and offshore teams.