Senior Data Pipelines Engineer & Automation Lead

Wpromote; El Segundo (CA); On-site; USD 150,000 - 210,000; Full time

Job summary

Wpromote is seeking a senior data engineer to act as a technical leader in data pipeline automation and workflow orchestration. You will design and maintain complex, automated data solutions using Airflow, dbt, and Google Cloud data products.

You will manage a large portfolio of dbt models, advocate for test-driven development, and collaborate across teams to ensure scalable, secure data practices. Strong Python/SQL skills and cloud experience are essential.

Qualifications

• Minimum 7+ years of experience in software development, with extensive experience in Python, SQL, and data pipelines.
• Expertise in building and automating ETL/ELT pipelines using Airflow and DAG-based workflow management software. Airbyte for data ingestion is a plus, but not required.
• Deep experience managing dbt models using macros and dbt tests. Mastery of Python or comparable scripting language, API integrations, and software architecture.
• Deep experience with databases like PostgreSQL and BigQuery, including query optimization for performance and cost.
• Good understanding of business intelligence tools like Looker or comparable alternatives. Experience with Google Cloud Platform, Kubernetes, and managing infrastructure as code using Terraform.
• Proficiency with advanced data formats (Parquet, Avro, Hive, JSONL) and data integration techniques.
• Experience with monitoring and logging tools (e.g., Prometheus, Grafana) is a plus.
• Familiarity with version control systems and CI/CD tools like GitHub Actions.
• Strong command of agile methodologies, continuous integration, and test-driven development.
• Exceptional problem-solving skills and technical leadership.
• Ability to influence and guide cross-functional teams and projects.

Responsibilities

• Acting as a technical leader and SME in data pipeline automation and workflow orchestration.
• Designing, implementing, and maintaining complex, reliable, data solutions with a focus on automation using Airflow, dbt, and Google Cloud data products.
• Managing a large portfolio of dbt models, leveraging macros and DRY patterns.
• Advocating for test-driven development and assisting QA in developing a robust and reliable process for continuous integration and delivery.
• Monitoring, troubleshooting, and optimizing the performance of data pipelines and workflows.
• Architecting solutions and reusable patterns that scale with business needs.
• Providing implementation, configuration, and deployment documentation.
• Proactively addressing issues and problems, generating and implementing innovative solutions.
• Participating in all agile ceremonies, including daily standups and regular sprint planning.
• Mentoring other engineers and fostering a culture of technical excellence.
• Staying up-to-date with the latest industry trends and technologies to drive continuous improvement and innovation in data engineering practices.
• Ensuring data security, governance, and compliance with relevant standards and privacy restrictions.

Skills

Python

SQL

Data pipelines

ETL/ELT automation

Airflow

dbt

APIs

Looker

GCP

Kubernetes

Terraform

CI/CD

GitHub Actions

Prometheus/Grafana

Data security

BigQuery

PostgreSQL

Parquet/Avro/JSONL

Tools

Airflow

Airbyte

dbt

Looker

Terraform

Kubernetes

Prometheus

Grafana

GitHub Actions

BigQuery

PostgreSQL

Parquet

Avro

Hive

JSONL

Job description

Wpromote is seeking a senior data engineer to act as a technical leader in data pipeline automation and workflow orchestration. You will design and maintain complex, automated data solutions using Airflow, dbt, and Google Cloud data products.

You will manage a large portfolio of dbt models, advocate for test-driven development, and collaborate across teams to ensure scalable, secure data practices. Strong Python/SQL skills and cloud experience are essential.