Senior Data Pipeline Engineer

Candid; New York (NY); Remote; USD 100,000 - 130,000; Full time

Benefits offered by this job

Health insurance

Retirement match

Life insurance & AD&D

Paid time off

Disability insurance

PSLF eligible employer

Public transit benefits

Job summary

Candid is seeking a Senior Data Pipeline Engineer to own the day-to-day development and delivery of our data pipelines. The role blends deep SQL and ETL expertise with hands-on cloud data platform work to move data from source systems into our warehouse and lakehouse.

Reporting to the Data Operations Manager, you will architect and implement robust pipelines, mentor junior engineers, and collaborate across distributed teams while supporting a remote-first work model with periodic in-person

Qualifications

• 5+ years in data engineering or ETL with production pipelines.
• Expert-level Transact-SQL and hands-on C#/.NET development.
• Experience architecting ETL processes with SSIS; Airflow experience preferred.
• Experience with AWS and open table formats (Iceberg, Parquet).
• Strong data quality and root-cause troubleshooting skills.
• Familiar with source control, CI/CD, and release processes across Dev/QA/Prod environments.
• Ability to mentor and review the work of other engineers; good communication.
• Commitment to inclusive, respectful collaboration.

Responsibilities

• Architect, develop, and maintain ETL/ELT pipelines across source systems into the data warehouse/lakehouse.
• Support evolving cloud data platform including Iceberg ingestion and DynamoDB tuning.
• Monitor data quality end-to-end and troubleshoot production data incidents.
• Onboard new external and regulatory data sources; contribute to schema/architecture decisions.
• Maintain SOPs and technical documentation; mentor Associate engineers.
• Partner with Data Operations Manager on architecture and roadmap; provide escalation support.

Skills

Transact-SQL

C#/.NET

ETL engineering

Data pipelines

SSIS

Apache Airflow

AWS

Data quality

CI/CD

Mentorship

Tools

Apache Iceberg

Parquet

DynamoDB

Trino/Starburst

SSIS

Job description

Position Summary

Candid is a nonprofit that provides the most comprehensive data and insights about the social sector. We get you the information you need to do good. Candid currently has an opportunity for a Senior Data Pipeline.

The Senior Data Pipeline Engineer is a senior technical role responsible for the day-to-day development, and delivery of Candid's data pipelines. This role blends deep SQL and ETL engineering expertise with hands‑on experience across modern cloud data platform technologies to design, build, and tune pipelines that move data reliably from source systems into Candid's warehouse and lakehouse.

Position: Senior Data Pipeline Engineer

Reporting to: Data Operations Manager

Supervises: N/A

Schedule: 35-hour work week, Monday through Friday

Compensation: $100,000 - $130,000 (this range is for the NYC area and will be adjusted for other localities; additionally, factors like skills and experience will be considered).

Location: Remote. In-person attendance is expected twice per year during our annual, weeklong all-staff summits. Additional in-person meeting participation is expected at least once per quarter for senior leaders and at least once per month for the executive team. Staff not located in the NYC area are expected to travel for these meetings.

Benefits: Health insurance (medical, dental, vision), retirement contribution with additional option for a match, paid life insurance and AD&D, paid leave time (PTO, compassionate leave, volunteer, holiday, parental), short-term and long-term disability, pre-tax transit, flexible spending accounts, supplemental insurance, summer hours, and Public Service Loan Forgiveness (PSLF) program eligible employer.

Applicants for this position must be legally authorized to work in the United States without employer sponsorship, now or in the future.

Responsibilities

• Pipeline development: Architect, develop, and maintain ETL/ELT and C#/.NET pipeline services (SSIS, Airflow, AWS Lambda, enhancer and message-processing services) that move data reliably from source systems into Candid's warehouse and lakehouse.
• Cloud platform & performance: Support Candid's evolving cloud data platform, including Iceberg ingestion and transformation pipelines, DynamoDB, and query performance tuning via Trino/Starburst.
• Data quality & incident resolution: Monitor data quality end-to-end and investigate production pipeline issues at the root cause, documenting findings and driving fixes through to reprocessing.
• Source onboarding & schema coordination: Scope and onboard new external and regulatory data sources end-to-end, and contribute to schema and architecture decisions, coordinating changes across ingestion and transformation layers.
• Documentation, mentorship & process: Maintain SOP and technical documentation, mentor and review the work of Associate-level engineers, and participate in the team's release and deployment process.
• Partnership & support: Partner with the Data Operations Manager on architecture and roadmap decisions, and provide escalation-level support for mission-critical pipeline issues, including after-hours as needed.

Requirements

• 5+ years of experience in database engineering, data engineering, or ETL development, with demonstrated ownership of production data pipelines.
• Expert-level Transact-SQL skills (stored procedures, views, indexing) and hands‑on C#/.NET development experience.
• Proven experience architecting ETL processes with SSIS; experience with orchestration platforms such as Apache Airflow preferred.
• Experience with AWS and open table formats (Apache Iceberg, Parquet); AWS Lambda, DynamoDB, and Trino/Starburst a plus.
• Strong data quality and root-cause troubleshooting skills, with experience resolving production data incidents.
• Familiarity with source control, CI/CD, and release/deployment processes across Dev/QA/Prod environments.
• Ability to mentor and review the work of other engineers; excellent written and verbal communication.
• Good business judgment, independent prioritization, and the ability to work well in a distributed team.
• Willingness to perform other duties and special projects as needed/requested.
• Sensitivity and respect for racial, gender, sexual orientation, and cultural differences.
• Commitment to Candid's values: driven, direct, accessible, curious, and inclusive.

About Candid

Candid's mission is to get you the information you need to do good.

The world's problems are only growing, and change can't wait. Nonprofits are needed now more than ever, but all too often their work goes without adequate support.

Candid makes it easier and faster for nonprofits and funders to connect in pursuit of solutions to change the world. Candid is where nonprofits find grants, donors find nonprofits that inspire them, and all can gain insights about the work being done for good.

Candid is a qualifying nonprofit organization as defined by the Public Service Loan Forgiveness Program. As such, Candid employees may claim their employment time on their PSLF application. We offer a competitive salary and excellent benefits. Due to the high volume of applicants we typically receive, we regret that we can only contact candidates we would like to interview.

For more information on positions available at Candid, please visit our website: Work with us

Candid is an equal opportunity employer. Candid provides equal employment opportunities to all employees and applicants for employment and prohibits discrimination and harassment of any type without regard to race, color, religion, age, sex, national origin, disability status, genetics, protected veteran status, sexual orientation, gender identity or expression, or any other characteristic protected by federal, state, or local laws.

This policy applies to all terms and conditions of employment, including recruiting, hiring, placement, promotion, termination, layoff, recall, transfer, leaves of absence, compensation, and training.

Notice:

We are aware of emails/communications impersonating Candid's recruitment staff. All legitimate recruiting communications come from a @candid.org email address. If you've received any communications claiming to be Candid that does not come from a @candid.org email address, please forward to [e-mail]. We appreciate your interest in Candid.