Data Engineer — Build Scalable Pipelines (Hybrid)

Talentify; San Francisco (CA); Hybrid; USD 140,000 - 190,000; Full time

Job summary

Talentify in San Francisco, CA is seeking a Data Engineer to build core data systems powering batch and streaming data and ML applications. You will design pipelines, work with Hadoop, Spark, Databricks, Snowflake and AWS, and collaborate with cross-functional teams to deliver data with high quality.

You will implement data quality, observability and governance, write tests, and help automate processes to reduce cloud costs while scaling infrastructure for growth.

Qualifications

• 4+ years of experience and a bachelor's degree in computer science or a related field; or equivalent work experience.
• Working experience of distributed systems Hadoop, Spark, Hive, Kafka, DBT and Airflow/Dagster.
• At least 2 year of production coding experience in data pipeline implementation in Python.
• Experience working with public cloud platforms, preferably AWS.
• Experience working with Databricks and/or Snowflake.
• Experience in Git, JIRA, Jenkins, shell scripting.
• Familiarity with Agile methodology, test-driven development, source control management and test automation.
• Experience supporting and working with cross-functional teams in a dynamic environment.

Responsibilities

• You will build systems, core libraries and frameworks that power our batch and streaming Data and ML applications.
• The services you build will integrate directly with LendingClub's products, opening the door to new features.
• You will work with modern data technologies such as Hadoop, Spark, DBT, Dagster/Airflow, Atlan, Trino, etc., modern data platforms such as Databricks and Snowflake and cloud technologies across AWS stack.
• Build data pipelines that transform raw data into canonical schema representing business entities and publish it into the Data Lake
• Implement internal process improvements: automating manual processes, optimizing data delivery, reducing cloud costs, redesigning infrastructure for greater scalability, etc.
• Work with stakeholders including the Business, Product, Program and Engineering teams to deliver required data in time with high quality at reasonable cost
• Implement processes and systems to monitor Data Quality, Observability, Governance and Lineage.
• Support operations to manage the production environment and help in resolving production issues with RCA
• Write unit/integration tests, adopt Test-driven development, contribute to engineering wiki, and document design/implementation etc.

Skills

Python

AWS

Hadoop

Spark

Airflow

Dagster

Databricks

Snowflake

Git

JIRA

Jenkins

Shell scripting

Agile

TDD

Education

Bachelor's degree in CS or related field

Tools

DBT

Job description

Talentify in San Francisco, CA is seeking a Data Engineer to build core data systems powering batch and streaming data and ML applications. You will design pipelines, work with Hadoop, Spark, Databricks, Snowflake and AWS, and collaborate with cross-functional teams to deliver data with high quality.

You will implement data quality, observability and governance, write tests, and help automate processes to reduce cloud costs while scaling infrastructure for growth.