Data Engineer: Build Secure, Scalable Data Pipelines

Compunnel, Inc.; San Francisco (CA); On-site; USD 100,000 - 130,000; Full time

Job summary

A technology firm in San Francisco is seeking a Data Engineer to manage, analyze, and visualize large datasets. The role involves designing data pipelines and collaborating with product managers, data scientists, and analysts. Candidates should have a Bachelor’s degree in a related field and at least 2 years of experience with tools like Databricks and Python. This position offers a dynamic work environment where efficiency and security in data processes are prioritized.

Qualifications

• Bachelor’s degree in computer science, Information Systems, or related field, or equivalent experience.
• 2+ years’ experience with Databricks, Collibra, and Starburst.
• 3+ years’ experience with Python and PySpark.
• Experience using Jupyter notebooks for coding and unit testing.
• Strong recent experience with relational and NoSQL data stores.
• 2+ years’ experience with a modern data stack.
• Experience with cloud data warehouses such as RedShift or Snowflake.
• Overall data engineering experience across traditional ETL and Big Data.

Responsibilities

• Design, develop, and maintain efficient data pipelines.
• Participate in Agile ceremonies and follow processes defined by the program team.
• Deliver high-quality data products and services.
• Identify and resolve issues related to data pipelines.
• Implement monitoring and alerting for uptime and reliability.
• Apply a security-first approach with testing and automation.
• Collaborate with product managers and stakeholders to address data needs.
• Stay updated with data engineering trends and evaluate new tools.

Skills

Python

PySpark

Databricks

Collibra

Starburst

Jupyter notebooks

ETL processes

Cloud data warehouses

S3

Airflow

Education

Bachelor’s degree in Computer Science

Tools

Spark

RedShift

Snowflake

Job description

A technology firm in San Francisco is seeking a Data Engineer to manage, analyze, and visualize large datasets. The role involves designing data pipelines and collaborating with product managers, data scientists, and analysts. Candidates should have a Bachelor’s degree in a related field and at least 2 years of experience with tools like Databricks and Python. This position offers a dynamic work environment where efficiency and security in data processes are prioritized.