Senior Python & PySpark Data Engineer (AI/ELT)

Software Guidance & Assistance, Inc. (SGA, Inc.); Rutherford (NJ); On-site; USD 120,000 - 180,000; Full time

Job summary

Software Guidance & Assistance, Inc. (SGA, Inc.) seeks an experienced Data Engineering Lead to design, develop, and deploy scalable AI agents within enterprise ELT/ETL pipelines.

You will drive microservice integrations, cloud-native architectures, and CI/CD automation to ensure quality and scalability. Ideal candidates have 8+ years in large-scale app development, strong Python/PySpark experience, and deep knowledge of data modeling, data warehousing, and Data Mesh concepts.

Qualifications

• 8+ years of experience in large-scale application development with secure AI agent deployment.
• 5+ years in Python and PySpark engineering leadership for high-volume ELT/ETL.
• Hands-on with YAML, JSON, FAST API or Spring Boot, GitHub Copilot.
• Proven microservice integration experience for data-intensive apps.
• Proficiency in Python or Scala and strong SQL skills.
• Deep knowledge of data modeling, warehousing, Data Mesh, and data federation.
• Excellent communication, collaboration, and problem-solving abilities.

Responsibilities

• Design, develop, and maintain scalable, enterprise AI agents for ELT/ETL with Python, FAST API, and related tech.
• Develop, deploy, and automate microservice integrations using cloud-native infra and Kubernetes/OpenShift.
• Ensure data quality, integrity, and security across the data lifecycle.
• Contribute to continuous improvement of data engineering processes, standards, and best practices.

Skills

Large-scale app development

Python

PySpark

YAML/JSON

FAST API

Spring Boot

SQL

Data modeling

Data warehousing

Data Mesh

CI/CD

Tools

GitHub Copilot

OpenShift

Kubernetes

Job description

Software Guidance & Assistance, Inc. (SGA, Inc.) seeks an experienced Data Engineering Lead to design, develop, and deploy scalable AI agents within enterprise ELT/ETL pipelines.

You will drive microservice integrations, cloud-native architectures, and CI/CD automation to ensure quality and scalability. Ideal candidates have 8+ years in large-scale app development, strong Python/PySpark experience, and deep knowledge of data modeling, data warehousing, and Data Mesh concepts.