Senior Data Engineer — Scalable ETL/ELT Pipelines

itD; Menlo Park (CA); On-site; USD 140,000 - 170,000; Full time

Benefits offered by this job

Medical benefits

401k plan

Paid holidays

Job summary

itD is seeking a Data Engineer III to design, build, and optimize scalable data pipelines and analytics-ready datasets powering critical operational decisions across the organization. You will contribute to workforce measurement, reporting, pipeline automation, and historical backfills, leveraging SQL, Python, and modern orchestration tools.

The role emphasizes reliability, data quality, and collaboration with data scientists and analysts to translate business needs into robust data solutions

Qualifications

• Strong proficiency in advanced SQL with Presto/Trino/Spark SQL.
• Python programming for production ETL/ELT development.
• Experience building large-scale data pipelines in Hive/Spark environments.
• Solid understanding of data warehousing, including partitioning and incremental loads.
• Experience with pipeline orchestration and monitoring.

Responsibilities

• Design, develop, and maintain scalable ETL/ELT data pipelines using advanced SQL and Python to support large-scale analytics and reporting requirements.
• Build, optimize, and maintain data workflows across Hive/Spark-based data warehouse environments, ensuring performance, accuracy, and reliability.
• Develop and manage pipeline orchestration workflows using Dataswarm, Airflow, or equivalent tools, including scheduling, backfills, and failure troubleshooting.
• Create and maintain analytics-ready datasets and data models supporting workforce measurement, staffing, capacity, and productivity reporting.
• Monitor pipeline health, implement data quality checks, and establish monitoring practices to ensure data accuracy and adherence to service-level agreements (SLAs).
• Maintain and improve legacy data pipelines, automate manual workflows, and develop self-service datasets to improve operational efficiency.

Skills

Advanced SQL (Presto/Trino/Spark SQL)

Python

Large-scale data pipelines

Data warehousing fundamentals

Pipeline orchestration (Airflow/Datasw

Data quality checks

Education

Bachelor's degree in Computer Science, Computer Engineering, or related field

Tools

Dataswarm

Airflow

Hive/Spark

Job description

itD is seeking a Data Engineer III to design, build, and optimize scalable data pipelines and analytics-ready datasets powering critical operational decisions across the organization. You will contribute to workforce measurement, reporting, pipeline automation, and historical backfills, leveraging SQL, Python, and modern orchestration tools.

The role emphasizes reliability, data quality, and collaboration with data scientists and analysts to translate business needs into robust data solutions