Technical Lead: AWS PySpark & GenAI Data Pipelines

Apptad Inc; California (MO); Hybrid; USD 150,000 - 210,000; Full time

Job summary

Apptad Inc. is seeking a highly skilled Technical Lead AWS PySpark to drive data engineering initiatives on AWS and lead a team of data engineers in building scalable data pipelines.

You will optimize PySpark workloads, govern data quality, and enable GenAI-enabled ingestion and metadata extraction. The role requires strong AWS expertise, vector database experience, and a proven track record of delivering in Agile environments with CI/CD, while mentoring engineers and driving data governance

Qualifications

• 6+ years in Data Engineering, Big Data, or Cloud Architecture.
• Hands-on unstructured data processing and metadata extraction.
• Experience with LLMs for parsing unstructured data.
• Expertise in Vector Databases (e.g., Pinecone, Milvus, Qdrant).
• AWS Data Engineer or AWS Solutions Architect with active certs preferred.
• Proficient in PySpark and high-performance SQL tuning.
• Experience with AWS EMR, Glue, DataBrew, Lambda, Redshift, S3, IAM, Step Functions.
• Strong Agile experience; skilled in JIRA reporting.
• CI/CD using Git/GitHub Actions; AI-assisted development tools like Copilot.

Responsibilities

• Design and implement large-scale, high-performance data pipelines on AWS.
• Architect end-to-end ETL for unstructured data and LLM integration.
• Manage vector data storage for advanced search and AI workloads.
• Mentor a team of data engineers; drive Agile sprints and backlog tracking.
• Provide regular reporting to stakeholders; ensure data governance and quality.
• Optimize PySpark workloads and SQL performance; implement serverless data processing.

Skills

PySpark

AWS Cloud

Data Engineering

Unstructured Data

LLMs / GenAI

Vector DBs

SQL Tuning

Agile / JIRA

CI/CD

AI in Dev

Education

Bachelor's/Master's in IT/CS

Tools

AWS EMR

AWS Glue

AWS DataBrew

AWS Lambda

Redshift

S3

GitHub Actions

Git

Job description

Apptad Inc. is seeking a highly skilled Technical Lead AWS PySpark to drive data engineering initiatives on AWS and lead a team of data engineers in building scalable data pipelines.

You will optimize PySpark workloads, govern data quality, and enable GenAI-enabled ingestion and metadata extraction. The role requires strong AWS expertise, vector database experience, and a proven track record of delivering in Agile environments with CI/CD, while mentoring engineers and driving data governance