Lead Python Platform Engineer

VMC Soft Technologies, Inc; Charlotte (NC); On-site; USD 120,000 - 180,000; Full time

Job summary

VMC Soft Technologies, Inc. in Charlotte, NC is seeking a Lead Python Platform Engineer to design and build a containerized API layer over Apache Spark, delivering secure, scalable access to Spark-based data pipelines.

You will lead the development of production-grade Python APIs, containerized via Docker/Kubernetes, and implement CI/CD with GitHub Actions while coordinating with onsite and offshore teams.

Qualifications

• At least 10 years of IT experience.
• Proven track record building production-grade Python APIs for data platforms.
• Experience packaging services with containers (Docker/Kubernetes) and running as platform services.
• Strong CI/CD automation experience, e.g., GitHub Actions.
• GCP expertise and data pipeline interaction.

Responsibilities

• Work with onsite team and client to design and govern Spark-based data pipelines via a stable API.
• Lead containerized API layer development and secure, scalable platform services.
• Coordinate offshore resources and ensure timely delivery of complex software projects.

Skills

Python

Pyspark

API

CI/CD

GCP

ETL

Tools

Kubernetes

FastAPI

Pytest

ARQ

Job description

Job Title: Lead Python Platform Engineer PySpark, FastAPI & GCP

Work Location: Charlotte, NC 28202

Job Details Must Have Skills

Python, Pyspark, API, Microservices CI/CD, GCP, ETL

Nice to have skills

ETL

Detailed Job Description At least 10 years of experience in IT industry.

We are seeking a Python-based platform engineer to design and build a containerized API layer that abstracts and governs interactions with Apache Spark through a well-defined API contract. This role focuses on building platform capabilities, not simply consuming existing data tools enabling consistent, secure, and scalable access to Spark-based data pipelines.

The ideal candidate has strong experience developing production-grade APIs in Python that interface with data frameworks or pipeline orchestration systems, packaging services using containers (Docker/Kubernetes), and operating them as reusable platform services. A proven background in CI/CD automation using GitHub Actions is required, along with solid software engineering practices around testing, versioning, and deployment.

Should have good hands-on experience in GCP skills. Should be able to interact, coordinate with the client.

Should be able to manage offshore resources also.

Should be able to work with other onsite team members and client.

This role is suited for engineers who have built data or compute platforms, service layers, or internal frameworks translating complex Spark capabilities into stable, contract-driven APIs for broad enterprise use.

Tech Skills

Python,

Pyspark

Experience building control planes

Design patterns

• Modular containerized services
• Asynchronous processing
• Job-based work queue execution
• Immutability and idempotency
• Metadata management
• Policy enforcement
• Semantic modeling
• Observability and telemetry
• Side effect capability injection

GCP

Testing

Functional testing

Behavioral testing

Languages

Python

Frameworks and Platforms

Fast API

ARQ

Kubernetes

Pytest

Minimum Years Of Experience >10 years

Certifications Needed :No

Top 3 responsibilities you would expect the Subcon to shoulder and execute

Should be able to work with other onsite team members and client

Should be able to manage offshore resources also.

Should be able to manage complex software application development and support projects