Senior Software Engineer I (Data Pipelines)

Wpromote; El Segundo (CA); On-site; USD 150,000 - 210,000; Full time

Job summary

Wpromote is seeking a senior data engineer to act as a technical leader in data pipeline automation and workflow orchestration. You will design and maintain complex, automated data solutions using Airflow, dbt, and Google Cloud data products.

You will manage a large portfolio of dbt models, advocate for test-driven development, and collaborate across teams to ensure scalable, secure data practices. Strong Python/SQL skills and cloud experience are essential.

Qualifications

• Minimum 7+ years of experience in software development, with extensive experience in Python, SQL, and data pipelines.
• Expertise in building and automating ETL/ELT pipelines using Airflow and DAG-based workflow management software. Airbyte for data ingestion is a plus, but not required.
• Deep experience managing dbt models using macros and dbt tests. Mastery of Python or comparable scripting language, API integrations, and software architecture.
• Deep experience with databases like PostgreSQL and BigQuery, including query optimization for performance and cost.
• Good understanding of business intelligence tools like Looker or comparable alternatives. Experience with Google Cloud Platform, Kubernetes, and managing infrastructure as code using Terraform.
• Proficiency with advanced data formats (Parquet, Avro, Hive, JSONL) and data integration techniques.
• Experience with monitoring and logging tools (e.g., Prometheus, Grafana) is a plus.
• Familiarity with version control systems and CI/CD tools like GitHub Actions.
• Strong command of agile methodologies, continuous integration, and test-driven development.
• Exceptional problem-solving skills and technical leadership.
• Ability to influence and guide cross-functional teams and projects.

Responsibilities

• Acting as a technical leader and SME in data pipeline automation and workflow orchestration.
• Designing, implementing, and maintaining complex, reliable, data solutions with a focus on automation using Airflow, dbt, and Google Cloud data products.
• Managing a large portfolio of dbt models, leveraging macros and DRY patterns.
• Advocating for test-driven development and assisting QA in developing a robust and reliable process for continuous integration and delivery.
• Monitoring, troubleshooting, and optimizing the performance of data pipelines and workflows.
• Architecting solutions and reusable patterns that scale with business needs.
• Providing implementation, configuration, and deployment documentation.
• Proactively addressing issues and problems, generating and implementing innovative solutions.
• Participating in all agile ceremonies, including daily standups and regular sprint planning.
• Mentoring other engineers and fostering a culture of technical excellence.
• Staying up-to-date with the latest industry trends and technologies to drive continuous improvement and innovation in data engineering practices.
• Ensuring data security, governance, and compliance with relevant standards and privacy restrictions.

Skills

Python

SQL

Data pipelines

ETL/ELT automation

Airflow

dbt

APIs

Looker

GCP

Kubernetes

Terraform

CI/CD

GitHub Actions

Prometheus/Grafana

Data security

BigQuery

PostgreSQL

Parquet/Avro/JSONL

Tools

Airflow

Airbyte

dbt

Looker

Terraform

Kubernetes

Prometheus

Grafana

GitHub Actions

BigQuery

PostgreSQL

Parquet

Avro

Hive

JSONL

Job description

You Will Be

• Acting as a technical leader and Subject Matter Expert (SME) in data pipeline automation and workflow orchestration.

• Designing, implementing, and maintaining complex, reliable, data solutions with a focus on automation using Airflow, dbt, and Google Cloud data products.

• Managing a large portfolio of dbt models, leveraging macros and DRY patterns.

• Advocating for test-driven development and assisting QA in developing a robust and reliable process for continuous integration and delivery.

• Monitoring, troubleshooting, and optimizing the performance of data pipelines and workflows.

• Architecting solutions and reusable patterns that scale with business needs.

• Providing implementation, configuration, and deployment documentation.

• Proactively addressing issues and problems, generating and implementing innovative solutions.

• Participating in all agile ceremonies, including daily standups and regular sprint planning.

• Mentoring other engineers and fostering a culture of technical excellence.

• Staying up-to-date with the latest industry trends and technologies to drive continuous improvement and innovation in data engineering practices.

• Ensuring data security, governance, and compliance with relevant standards and privacy restrictions.

You Must Have

Minimum 7+ years of experience in software development, with extensive experience in Python, SQL, and data pipelines.

Expertise in building and automating ETL/ELT pipelines using Airflow and DAG-based workflow management software. Airbyte for data ingestion is a plus, but not required.

Deep experience managing dbt models using macros and dbt tests.Mastery of Python or comparable scripting language, API integrations, and software architecture.

Deep experience with databases like PostgreSQL and BigQuery, including query optimization for performance and cost.

Good understanding of business intelligence tools like Looker or comparable alternatives.Experience with Google Cloud Platform, Kubernetes, and managing infrastructure as code using Terraform.

Proficiency with advanced data formats (Parquet, Avro, Hive, JSONL) and data integration techniques.

Experience with monitoring and logging tools (e.g., Prometheus, Grafana) is a plus.

Familiarity with version control systems and CI/CD tools like GitHub Actions.

Strong command of agile methodologies, continuous integration, and test-driven development.

Exceptional problem-solving skills and technical leadership.

Ability to influence and guide cross-functional teams and projects.