AWS PySpark Tech Lead - Data Pipelines & GenAI

Argyllinfotech; South San Francisco (CA); Hybrid; USD 150,000 - 190,000; Full time

Benefits offered by this job

Health insurance

401(k) plan

Job summary

Saama is seeking a Technical Lead for AWS PySpark to head data engineering initiatives for a Genentech project. You will design and optimize scalable data pipelines on AWS, lead a team of engineers, and drive data governance, quality, and lineage across enterprise workflows.

The role requires deep PySpark, SQL, and AWS ETL experience, with a strong architectural mindset and hands-on leadership in Agile environments. Hybrid onsite in South San Francisco / Remote work options are available.

Qualifications

• 6+ years in Data Engineering, Big Data, or Cloud Architecture
• Experience processing unstructured data and metadata extraction
• Proven experience with LLMs to parse unstructured data
• Vector databases indexing and storage experience
• Active AWS data engineer/solutions architect background preferred
• Strong Python and Spark optimization skills
• Proficient in SQL tuning for distributed DBs
• Experience with Agile methodologies and Jira reporting

Responsibilities

• Design and implement large-scale data pipelines on AWS (EMR, PySpark)
• Architect unstructured data and GenAI pipelines including metadata extraction
• Manage embeddings across vector databases for AI workloads
• Lead data engineers with Agile delivery and Jira dashboards
• Provide daily/weekly/monthly reporting to stakeholders
• Ensure data governance, quality, and lineage across workflows
• Optimize PySpark performance and SQL workloads
• Deploy serverless data workflows with Lambda and Redshift for analytics

Skills

Data Engineering

AWS

PySpark

SQL

GenAI/LLM

Agile/JIRA

Data Governance

Cloud Architecture

Education

Bachelor's or Master's in IT/CS

Tools

AWS EMR

AWS Glue

AWS DataBrew

AWS Lambda

AWS Redshift

S3/IAM/Step Functions

Vector DB (Pinecone/Milvus/OpenSearch)

Job description

Saama is seeking a Technical Lead for AWS PySpark to head data engineering initiatives for a Genentech project. You will design and optimize scalable data pipelines on AWS, lead a team of engineers, and drive data governance, quality, and lineage across enterprise workflows.

The role requires deep PySpark, SQL, and AWS ETL experience, with a strong architectural mindset and hands-on leadership in Agile environments. Hybrid onsite in South San Francisco / Remote work options are available.