Data Engineer

Compunnel, Inc.; San Francisco (CA); On-site; USD 100,000 - 130,000; Full time

Job summary

A technology firm in San Francisco is seeking a Data Engineer to manage, analyze, and visualize large datasets. The role involves designing data pipelines and collaborating with product managers, data scientists, and analysts. Candidates should have a Bachelor’s degree in a related field and at least 2 years of experience with tools like Databricks and Python. This position offers a dynamic work environment where efficiency and security in data processes are prioritized.

Qualifications

• Bachelor’s degree in computer science, Information Systems, or related field, or equivalent experience.
• 2+ years’ experience with Databricks, Collibra, and Starburst.
• 3+ years’ experience with Python and PySpark.
• Experience using Jupyter notebooks for coding and unit testing.
• Strong recent experience with relational and NoSQL data stores.
• 2+ years’ experience with a modern data stack.
• Experience with cloud data warehouses such as RedShift or Snowflake.
• Overall data engineering experience across traditional ETL and Big Data.

Responsibilities

• Design, develop, and maintain efficient data pipelines.
• Participate in Agile ceremonies and follow processes defined by the program team.
• Deliver high-quality data products and services.
• Identify and resolve issues related to data pipelines.
• Implement monitoring and alerting for uptime and reliability.
• Apply a security-first approach with testing and automation.
• Collaborate with product managers and stakeholders to address data needs.
• Stay updated with data engineering trends and evaluate new tools.

Skills

Python

PySpark

Databricks

Collibra

Starburst

Jupyter notebooks

ETL processes

Cloud data warehouses

S3

Airflow

Education

Bachelor’s degree in Computer Science

Tools

Spark

RedShift

Snowflake

Job description

The Data Engineer will be responsible for collecting, parsing, managing, analyzing, and visualizing large datasets to transform information into actionable insights. The role involves building scalable, repeatable, and secure data pipelines across various platforms. The engineer will ensure that data workflows support multiple users while maintaining high performance, security, and reliability.

Key Responsibilities

• Design, develop, and maintain robust and efficient data pipelines to ingest, transform, catalog, and deliver curated, trusted, and quality data into the Common Data Platform.
• Actively participate in Agile ceremonies and follow Scaled Agile processes defined by the CDP Program team.
• Deliver high-quality data products and services following SAFe Agile practices.
• Identify and resolve issues related to data pipelines and analytical data stores.
• Implement monitoring and alerting for pipelines and data stores, enabling auto-remediation to ensure uptime and reliability.
• Apply a security-first approach with strong testing and automation practices.
• Collaborate with product managers, data scientists, analysts, and business stakeholders to understand data needs and provide tools and infrastructure.
• Stay updated with modern data engineering trends and evaluate new tools, frameworks, and technologies to improve efficiency.

Required Skills

• Bachelor’s degree in computer science, Information Systems, or related field, or equivalent experience.
• 2+ years’ experience with Databricks, Collibra, and Starburst.
• 3+ years’ experience with Python and PySpark.
• Experience using Jupyter notebooks for coding and unit testing.
• Strong recent experience with relational and NoSQL data stores, STAR and Dimensional Modeling methods.
• 2+ years’ experience with a modern data stack (S3, Spark, Airflow, Lakehouse architecture, real‑time databases).
• Experience with cloud data warehouses such as RedShift or Snowflake.
• Overall data engineering experience across traditional ETL and Big Data (on‑prem or cloud).
• Experience building end‑to‑end pipelines for unstructured and semi‑structured data using Spark.
• Ability to work with confidential supervisory information (Must meet Protected Individual requirements).

Education

Bachelor’s degree in computer science, Information Systems, or related field (or equivalent experience).