Lead Data Engineer: AI-Driven Data Pipelines

BlackLine; New York (NY); On-site; USD 193,000 - 242,000; Full time

Job summary

BlackLine, a leading cloud software provider, seeks a Lead Data Engineer in New York to design, build, and optimize data pipelines powering AI-driven accounting agents. You will lead scalable data infrastructure and collaborate across teams to deliver high-quality data infrastructure.

The role emphasizes data reliability, performance, and close partnership with data scientists, analysts, and business stakeholders in a fast-paced, in-office-first startup environment.

Qualifications

• 4+ years of experience as a Data Engineer or similar role.
• Bachelor’s degree in Computer Science, Engineering or related field.
• Proficient in PySpark and data transformation pipelines.
• Experience with AI technologies and cloud platforms (AWS, Azure) is a plus.

Responsibilities

• Lead data pipeline development: Build and maintain PySpark ETL pipelines with high data quality and performance.
• Manage integrations: Connect to client data sources via APIs and tools like FiveTran, Plaid, and internal connectors.
• Ensure reliability: Monitor pipeline performance, automate testing, and validate data accuracy.
• Optimize for scale: Implement performance improvements for large datasets (CDC, indexing).
• Collaborate with stakeholders to refine data requirements and integrate AI and big data tech.

Skills

PySpark

Data Pipelines

Cloud (AWS/Azure)

AI Technologies

Communication

Education

Bachelor's degree

Job description

BlackLine, a leading cloud software provider, seeks a Lead Data Engineer in New York to design, build, and optimize data pipelines powering AI-driven accounting agents. You will lead scalable data infrastructure and collaborate across teams to deliver high-quality data infrastructure.

The role emphasizes data reliability, performance, and close partnership with data scientists, analysts, and business stakeholders in a fast-paced, in-office-first startup environment.