Software Engineer AI/ML Systems - USA Onsite (Santa Clara, CA)

Dover; Santa Clara, Northern (CA, KY); Hybrid; USD 150,000 - 170,000; Full time

Benefits offered by this job

Unlimited PTO

Generous parental leave

Stock Purchase Program (ESPP)

Annual bonus program

Medical, Dental and Vision coverage

Mental health support

Job summary

Dover is seeking a Software Engineer specialized in AI/ML applications to independently drive end-to-end lifecycle management of AI-powered systems, blending advanced AI development with robust software engineering and automation.

You will deploy and scale open-source models on distributed GPU infrastructure, design automated testing frameworks, and implement secure CI/CD pipelines that ensure quality on every merge.

Qualifications

• 6+ years of production-grade, asynchronous Python development with clean architecture.

Responsibilities

• Architect and scale AI infrastructure across multi-node clusters using Kubernetes, Ray, or Slurm.
• Design, evaluate, and deploy production-grade AI models and agents with benchmarks.
• Conduct comprehensive model benchmarks and dashboards to communicate performance to stakeholders.
• Develop extensive automated test suites for end-to-end applications and stochastic AI outputs.
• Automate DevSecOps with vulnerability scanning in Merge Requests and fast remediation pipelines.
• Own features end-to-end from ideation to production, including repository synchronization and OSS interactions.

Skills

Python programming

AI systems design

Data analysis (pandas/NumPy)

GitLab pipelines

PyTest testing

Advanced Git workflows

Education

Bachelor's or Master's in CS/Engineering

Tools

Kubernetes

Ray

Slurm

LangChain

Hugging Face

MLOps

vLLM

SGLang

GitLab

PyTest

Job description

The Role

We're seeking a Software Engineer specialized in AI/ML applications to independently drive the development, evaluation, deployment, and end-to-end lifecycle management of AI-powered systems. This role sits at the intersection of advanced AI application development, robust software engineering, and continuous automation: you'll build the systems, and you'll build the machinery that keeps them tested, secure, and running at scale.

This position goes deep on infrastructure and reliability: deploying and scaling open-source models across distributed GPU infrastructure, designing automated testing frameworks that cover both deterministic code and stochastic AI outputs, and implementing secure CI/CD pipelines that enforce quality on every merge.

This role is built for an engineer who takes high ownership: comfortable carrying a feature from ideation through architecture, build, evaluation, and production, and rigorous enough to prove with benchmarks and dashboards that the system actually works.

What You'll Do

• Architect and scale AI infrastructure: deploy and scale open-source models using distributed orchestration frameworks (Kubernetes, Ray, or Slurm) to run highly available, fault-tolerant AI workloads across multi-node clusters.

• Design and evaluate AI systems: run experiments, prompt-tune, evaluate, and deploy production-grade models and AI agents, with flexible mechanisms to benchmark performance and swap models quickly as use cases evolve.

• Drive error and gap analysis: run comprehensive model benchmarks, perform deep error and gap analysis on model outputs, and build analytics dashboards that communicate system performance clearly to stakeholders.

• Build automated testing at depth: develop extensive automated test suites that validate end-to-end application code, aggressively improving coverage across both standard software and stochastic AI outputs.

• Automate DevSecOps: keep systems clean and secure by automating vulnerability scanning on GitLab Merge Requests and implementing fast remediation pipelines for identified issues.

• Execute independently: own features from ideation to production, including architectural decisions, public/private repository synchronization, and open-source community interactions.

What We’re Looking For

• 6+ years of professional experience writing production-grade, asynchronous Python, with a strong focus on decoupled, clean system architecture and design patterns.

• Deployment & orchestration: hands-on experience deploying, monitoring, and scaling models in production using Kubernetes, Ray, or Slurm, including multi-node cluster configurations.

• Hardware & scaling optimization: strong understanding of GPU memory management and infrastructure-level tuning for high-throughput, low-latency AI inference workloads.

• AI evaluation & frameworks: deep experience building with LangChain, Hugging Face libraries, MLOps, vLLM, and SGLang, with proven expertise in prompt engineering, automated model benchmarking, and systematic LLM evaluations.

• Data analysis: proficient in Python-based analysis (pandas, NumPy, or similar), able to extract insights from evaluation results and communicate findings clearly to technical and non-technical audiences.

• CI/CD & security automation: advanced knowledge of GitLab pipelines, including automated test jobs and vulnerability scanners integrated directly into the MR workflow.

• Testing toolchains: expert familiarity with Python testing frameworks (PyTest), mocking libraries, and automated test generation approaches for AI workloads.

• Advanced version control: high proficiency in advanced Git workflows, including rebase strategies, cryptographic commit signing, and complex public/private repository mirroring.

• Bachelor's or Master's degree in Computer Science, Engineering, or a related field (or equivalent experience).

Salary Range: US East/West Coast: $150000 - $170000

Disclaimer: The base salary range is a guideline and may vary based on factors such as candidate experience, specialized skills, and geographical location. Actual compensation may include additional benefits and bonuses.

Perks and Benefits of Working With Us

• Unlimited PTO.

• Please ask us about our very generous parental leave, much above industry standards!

• Entrepreneurial culture where pushing limits and taking risks is everyday business.

• Open communication with management and company leadership.

• Small, dynamic teams = massive impact.

• Medical, Dental and Vision coverage for employees.

• Access to Disability & Life insurance.

• Mental health and wellbeing support.

• Annual bonus program.

• Employer Stock Purchase Program (ESPP).

• Yearly team building experiences.

• Mentorship and sponsorship opportunities.

• Manager resources and support.

Cogniify is an equal opportunity employer. We celebrate diversity and are committed to creating an inclusive environment for all employees. All qualified applicants will receive consideration for employment without regard to race, color, religion, sex, sexual orientation, gender identity, national origin, disability, or any other protected characteristic