Python/PySpark Developer

Software Guidance & Assistance, Inc. (SGA, Inc.); Rutherford (NJ); On-site; USD 120,000 - 180,000; Full time

Job summary

Software Guidance & Assistance, Inc. (SGA, Inc.) seeks an experienced Data Engineering Lead to design, develop, and deploy scalable AI agents within enterprise ELT/ETL pipelines.

You will drive microservice integrations, cloud-native architectures, and CI/CD automation to ensure quality and scalability. Ideal candidates have 8+ years in large-scale app development, strong Python/PySpark experience, and deep knowledge of data modeling, data warehousing, and Data Mesh concepts.

Qualifications

• 8+ years of experience in large-scale application development with secure AI agent deployment.
• 5+ years in Python and PySpark engineering leadership for high-volume ELT/ETL.
• Hands-on with YAML, JSON, FAST API or Spring Boot, GitHub Copilot.
• Proven microservice integration experience for data-intensive apps.
• Proficiency in Python or Scala and strong SQL skills.
• Deep knowledge of data modeling, warehousing, Data Mesh, and data federation.
• Excellent communication, collaboration, and problem-solving abilities.

Responsibilities

• Design, develop, and maintain scalable, enterprise AI agents for ELT/ETL with Python, FAST API, and related tech.
• Develop, deploy, and automate microservice integrations using cloud-native infra and Kubernetes/OpenShift.
• Ensure data quality, integrity, and security across the data lifecycle.
• Contribute to continuous improvement of data engineering processes, standards, and best practices.

Skills

Large-scale app development

Python

PySpark

YAML/JSON

FAST API

Spring Boot

SQL

Data modeling

Data warehousing

Data Mesh

CI/CD

Tools

GitHub Copilot

OpenShift

Kubernetes

Job description

Responsibilities

• Design, develop, and maintain scalable, enterprise-grade AI agents , supporting ELT/ETL processes to handle large data volumes using the Python, FAST API, Microservices , PySpark, Kafka and Databricks ecosystem.
• Develop, deploy, and automate microservice integrations to support data-intensive applications, ensuring scalability, resilience, and maintainability using cloud native infrastructure and openshift or Kubernates architecture including CI/CD pipelines.
• Ensure data quality, integrity, and security throughout the entire data lifecycle.
• Contribute to the continuous improvement of data engineering processes, standards, and best practices within the team.

Required Skills

• 8+ years of overall experience in large-scale application development with recent mandatory platform for the secure and scalable deployment of AI agents into application contexts
• Minimum of 5+ years of proven experience in a Python and pyspark Engineering lead role focused on building enterprise-grade, high-volume ELT/ETL processes using the PySpark.
• Hands-on experience with YAML, JSON, FAST API or Spring boot, Github Copilot
• Proven experience developing and automating microservice integrations to support data-intensive applications.
• Proficiency in at least one programming language commonly used for data analytics, engineering, such as Python or Scala.
• Strong SQL skills and experience with various relational databases.
• Deep understanding of data modeling, data warehousing concepts, Data Mesh architecture, and data federation.
• Excellent communication, collaboration, and problem-solving skills

SGA is a technology and resource solutions provider driven to stand out. We are a women-owned business. Our mission: to solve big IT problems with a more personal, boutique approach. Each year, we match consultants like you to more than 1,000 engagements. When we say let's work better together, we mean it. You'll join a diverse team built on these core values: customer service, employee development, and quality and integrity in everything we do. Be yourself, love what you do and find your passion at work. Please find us at https://sgainc.com/.

SGA is an Equal Opportunity Employer and does not discriminate on the basis of Race, Color, Sex, Sexual Orientation, Gender Identity, Religion, National Origin, Disability, Veteran Status, Age, Marital Status, Pregnancy, Genetic Information, or Other Legally Protected Status. We are committed to providing access, equal opportunity, and reasonable accommodation for individuals with disabilities in employment, and our services, programs, and activities. Please visit our company EEO page to request an accommodation or assistance regarding our policy.