Senior Software Engineer

InnoveIQ; New York (NY); Hybrid; USD 90,000 - 103,000; Full time

Job summary

InnoveIQ in New York City seeks a Senior–Principal backend engineer to own production systems at scale, running production-grade Python services and Azure-native architectures. You will lead on-call incident response, reliability improvements, and complex system ownership.

This hybrid role emphasizes AI/LLM orchestration, observability, CI/CD discipline, and cross-team collaboration to ensure robust production operations in a fast-paced environment.

Qualifications

• 8–10+ years of backend or distributed systems experience with hands-on ownership of production systems.
• Strong Python engineering depth including performance optimization, concurrency, debugging, and reliability.
• Real-world Azure production experience beyond certs or labs, including addressing cold-start latency and scaling bottlenecks.
• Ability to clearly explain a past production incident, root cause, remediation, and prevention measures.

Responsibilities

• Own and maintain production-grade Python services end-to-end for reliability and observability.
• Design, build, and operate Azure-native distributed systems (Functions, Durable Functions, Event Grid, Service Bus).
• Build and maintain production-ready AI systems, including RAG pipelines and LLM orchestration frameworks.
• Diagnose and resolve live production issues using distributed tracing, log analytics, and monitoring tools.

Skills

Python

Azure

Incident management

On-call ownership

DevOps

Job description

Location: New York, NY (Manhattan)

Work Policy: Hybrid (3 days a week in-office required)

Level: Senior (Minimum 8 years of relevant industry experience)

About the Role

We are looking for seasoned engineers who have operated real production systems at scale—not just built prototypes or proof-of-concepts.

You have been on-call. You have debugged live production incidents under pressure. You deeply understand the difference between code that looks clean in a pull request and code that remains stable during peak production traffic, downstream failures, and operational stress.

This is not a research-only role or a greenfield experimentation position. This is a senior-to-principal-level engineering role focused on building, operating, and evolving critical backend and AI-powered systems in production environments. You will be expected to take ownership, drive engineering quality, and make sound technical decisions when systems fail—because eventually, they do.

What You'll Do

• Production Ownership: Own and maintain production-grade Python services end-to-end, ensuring reliability, scalability, observability, and robust operational support.
• Cloud Architecture: Design, build, and operate Azure-native distributed systems using Azure Functions, Durable Functions, Event Grid, Service Bus, and supporting cloud infrastructure.
• AI & Orchestration: Build and maintain production-ready AI systems, including RAG pipelines, agentic workflows, and LLM orchestration frameworks (e.g., LangChain, AutoGen, Semantic Kernel).
• Observability & Triage: Diagnose and resolve live production issues using distributed tracing, log analytics, profilers, Azure Monitor, Application Insights, and related observability tooling.
• Engineering Standards: Establish and uphold strong engineering standards around code reviews, CI/CD, testing strategy, deployment safety, and incident post-mortem practices.
• Polyglot Flexibility: Contribute to secondary technology stacks, including C#, when needed. Deep expertise is not required, but practical working proficiency and transparency around strengths and limitations are expected.

What We're Looking For

Non-Negotiable Qualifications

• Experience: Minimum 8–10+ years of professional backend or distributed systems engineering experience, with a proven track record of hands‑on ownership of systems running in production.
• Python Expertise: Strong Python engineering depth, including performance optimization, concurrency, debugging, and production reliability considerations.
• Azure Production Experience: Real-world Azure production experience—beyond certifications or lab environments. You have worked through issues such as cold-start latency, distributed failure modes, event ordering, scaling bottlenecks, and operational recovery.
• Incident Mastery: Ability to clearly explain a past production incident you owned, including the root cause, investigation approach, remediation steps, validation strategy, and long‑term prevention measures.

Strong Additional Signals

• Production AI/RAG: Experience shipping and maintaining production LLM or RAG systems beyond initial launch, including handling prompt drift, latency optimization, evaluation degradation, and cost management.
• High-Concurrency Systems: Experience with distributed data systems, including queue-backed architectures, streaming systems, or large‑scale SQL/NoSQL workloads.
• DevOps Culture: Ownership of CI/CD systems and deployment pipelines end-to-end, including rollback strategies, deployment safety, and operational automation.

Interview & Work Logistics

• Interview Process: Candidates must be open to participating in initial in-person interviews at our designated office location in the nearby city area.
• Work Location: Day-to-day employment is hybrid (3 days a week in-office) out of the New York City (Manhattan) office location only.

Integrity & Validation

• Experience Verification: All candidates will undergo comprehensive background and experience verification.
• Authenticity Over Buzzwords: We strongly value candidates who are honest about both their strengths and limitations. We are looking for genuine engineering depth, operational judgment, and production experience—not keyword‑heavy résumés.
• Salary Range: $65 - $75 base salary per hr. * (Note: Final compensation will be determined based on chosen leveling, relevant skills, experience, and qualifications.)*
• Information regarding benefits, where applicable, will be provided during the interview process.

Additional Information

• Visa Sponsorship: No visa sponsorship is available for this position. Applicants must have valid, unrestricted authorization to work in the United States.
• Equal Opportunity Statement: Employer is an Equal Opportunity Employer and complies with all applicable federal, state, and local employment laws. We do not discriminate on the basis of any protected characteristic. We are committed to providing reasonable accommodations for qualified individuals with disabilities.
• Consent to Communication & Use of AI Technology: By submitting your application and providing your contact information, you consent to receive text (SMS), email, and/or voice communications—whether automated, AI‑assisted, pre‑recorded, or individually initiated—from company, its representatives, partners, or affiliates using the contact information provided. These communications may relate to job opportunities and related recruiting information. Message and data rates may apply. Frequency may vary. You represent and warrant that the contact information you provide belongs to you and that you are authorized to receive communications at those contacts. To opt out of communications or for related questions, please contact us directly.