Sr. Data Engineer

TechDigital Group; El Segundo (CA); On-site; USD 80,000 - 120,000; Full time

Job summary

An innovative firm is seeking a skilled data engineer proficient in Python and Google BigQuery. This role involves creating data flows, managing integrations, and contributing to technical designs. You will collaborate with cross-functional teams, guide team members, and ensure the quality of deliverables through code reviews and user acceptance testing. With a focus on the retail and manufacturing domains, this position offers the chance to work in a dynamic environment where your expertise will drive impactful data solutions. If you are passionate about data engineering and thrive in a collaborative setting, this opportunity is perfect for you.

Qualifications

• Expertise in Python programming and Google BigQuery is essential.
• Experience in Google Cloud Platform and containerized execution.

Responsibilities

• Create data flows and manage connections for various data sources.
• Guide team members in development and conduct code reviews.

Skills

Google BigQuery

ETL

Python

Google Cloud Platform

Airflow

Dev-Sec-Ops

Ascend tool

Error logging

Containerization

Tools

Kubernetes

Composer

DBT

Job description

Mandatory skills - GBQ (Google BigQuery), ETL, Python

• Experience in Google Cloud Platform (GBQ, GCS, Kubernetes POD operator and Image creation, Composer, DAG creation)
• Python Programming Expert
• Experience in Ascend tool
• Create Data flow, Data service, Read and Write connection for the various sources.
• Failure Notification settings (Ascend Webhook notification, Scheduling in Ascend)
• Big query merge to perform merge
• Custom Read and Write connector using python framework
• Good understanding in Error logging
• Experience in Airflow (Create DAG, Configure the variables in Airflow, Scheduling)
• Experience in DBT to create the lineage in GCP – Optional
• Worked in Dev-Sec-Ops (CI/CD) environment.
• Experienced with containerized execution/code containerization
• Interact with the platform leads and 3rd party application leads to understand and analyze the various interfaces and data integration needs to create technical user stories
• Contribute to the creation of technical design, estimation as applicable for the user stories and review with the EAC team and Product Owner
• Conduct and participate in code review sessions for the team deliverables
• Guide the team members and involve in development and unit testing
• Work with business users for UAT and assist QA team accordingly for defects fixing
• Help the team troubleshoot critical issues in development, staging or production.
• Retail & Manufacturing Domain expertise.