Lead Data Engineer: PySpark & AI Pipelines

BlackLine; New York (NY); On-site; USD 193,000 - 242,000; Full time

Job summary

BlackLine is seeking a Lead Data Engineer to design, build, and optimize data pipelines powering AI-driven accounting agents. You will lead scalable data infrastructure, work with cross-functional teams, and drive data quality, reliability, and governance across diverse data sources.

The role emphasizes extracting, transforming, and loading data from multiple sources, implementing robust integrations, and collaborating with data scientists and analysts to turn data into actionable business

Qualifications

• 4+ years of experience as a data engineer or similar role
• Bachelor’s degree in Computer Science, Engineering or related field
• Expertise in PySpark and data transformation pipelines; familiarity with AI technologies and cloud platforms (AWS, Azure) is a plus

Responsibilities

• Lead data pipeline development: Build and maintain PySpark ETL pipelines with high data quality and performance
• Manage integrations: Establish robust connections to client data sources via APIs and tools like FiveTran, Plaid, and BlackLine’s internal connector ecosystem
• Ensure reliability: Monitor pipeline performance, automate testing, and validate data accuracy
• Optimize for scale: Implement performance improvements for large-scale datasets
• Collaborate with cross-functional teams to deliver valuable insights and design data security/governance protocols

Skills

PySpark

Data pipeline development

API integrations

Cloud platforms

Communication skills

Education

Bachelor’s degree in Computer Science or related field

Tools

FiveTran

Plaid

Internal connector ecosystem

Job description

BlackLine is seeking a Lead Data Engineer to design, build, and optimize data pipelines powering AI-driven accounting agents. You will lead scalable data infrastructure, work with cross-functional teams, and drive data quality, reliability, and governance across diverse data sources.

The role emphasizes extracting, transforming, and loading data from multiple sources, implementing robust integrations, and collaborating with data scientists and analysts to turn data into actionable business