Data Engineer III - Scalable ETL Pipelines (6mo)

ITD Technology Co.; Menlo Park (CA); On-site; USD 140,000 - 180,000; Full time

Job summary

itD is seeking a Data Engineer III to build, optimize, and maintain scalable data pipelines and analytics-ready datasets powering critical operational decisions. You will support workforce measurement, reporting, and backfills, with emphasis on reliable ETL/ELT workflows and high-quality data infrastructure.

The role requires strong SQL/Python expertise, experience with Hive/Spark environments, and workflow orchestration using Dataswarm or Airflow.

Qualifications

• Strong proficiency in advanced SQL (Presto, Trino, or Spark SQL).
• Basic to intermediate Python programming for production ETL/ELT development.
• Experience building and maintaining large-scale data pipelines across Hive/Spark environments.
• Strong understanding of data warehousing fundamentals and query performance optimization.
• Experience with pipeline orchestration tools such as Dataswarm or Airflow.
• Experience troubleshooting pipeline failures and managing historical backfills.
• Understanding of data pipeline monitoring and SLA requirements.

Responsibilities

• Design, develop, and maintain scalable ETL/ELT data pipelines using SQL and Python for analytics and reporting.
• Build, optimize, and maintain data workflows across Hive/Spark-based data warehouses.
• Develop and manage pipeline orchestration workflows with Dataswarm, Airflow, or equivalent tools.
• Create analytics-ready datasets and data models supporting workforce measurement and reporting.
• Monitor pipeline health, implement data quality checks, and ensure SLAs are met.
• Maintain and improve legacy pipelines and automate manual workflows to boost efficiency.
• Collaborate with data scientists, analysts, and stakeholders to translate requirements into reliable data solutions.

Skills

Advanced SQL

Python programming

Hive/Spark DW

Data warehousing

Workflow orchestration

Data quality checks

Backfills

Tools

Dataswarm

Airflow

Job description

itD is seeking a Data Engineer III to build, optimize, and maintain scalable data pipelines and analytics-ready datasets powering critical operational decisions. You will support workforce measurement, reporting, and backfills, with emphasis on reliable ETL/ELT workflows and high-quality data infrastructure.

The role requires strong SQL/Python expertise, experience with Hive/Spark environments, and workflow orchestration using Dataswarm or Airflow.