Data Engineer - III

Compunnel, Inc.; San Francisco (CA); On-site; USD 120,000 - 150,000; Full time

Job summary

A technology solutions provider in San Francisco is seeking a Data Engineer – III to design and maintain data pipelines processing large datasets. Ideal candidates will have over 3 years of experience with Python and hands-on knowledge of cloud data warehouses such as Redshift or Snowflake. The position emphasizes collaboration with teams and adherence to Agile practices.

Qualifications

• 3+ years of experience with Python and PySpark.
• 2+ years of experience with Databricks and Collibra.
• Strong background in traditional ETL and Big Data engineering.

Responsibilities

• Design and maintain robust data pipelines for curated data delivery.
• Implement monitoring and alerting for data pipelines.
• Work with cross-functional teams to understand data needs.

Skills

Python

PySpark

Data Engineering

ETL

AWS

Education

Bachelor’s degree in Computer Science or equivalent

Tools

Databricks

Redshift

Snowflake

Jupyter notebooks

Job description

We are seeking a Data Engineer – III to design, build, and maintain scalable, secure, and repeatable data pipelines across multiple platforms.

This role involves working with large datasets to transform raw data into actionable insights and collaborating with cross-functional teams to support data-driven decision-making.

Key Responsibilities

• Design, develop, and maintain robust data pipelines to ingest, transform, catalog, and deliver curated, trusted data from diverse sources.
• Participate in Agile ceremonies and follow Scaled Agile practices as defined by the program team.
• Deliver high-quality data products and services aligned with Safe Agile principles.
• Proactively identify and resolve issues related to data pipelines and analytical data stores.
• Implement monitoring and alerting for data pipelines and stores, including auto-remediation strategies.
• Apply a security-first, test-driven, and automation-focused approach to data engineering.
• Collaborate with product managers, data scientists, analysts, and business stakeholders to understand data needs and deliver appropriate infrastructure and tools.
• Stay current with emerging technologies and recommend tools and frameworks to enhance data engineering processes.

Required Qualifications

• Bachelor’s degree in Computer Science, Information Systems, or a related field, or equivalent experience.
• 3+ years of experience with Python and PySpark.
• 2+ years of experience with Databricks, Collibra, and Starburst.
• Experience using Jupyter notebooks for coding and unit testing.
• Hands-on experience with relational and NoSQL databases, including STAR and dimensional modeling.
• 2+ years of experience with modern data stacks (e.g., S3, Spark, Airflow, Lakehouse architectures, real-time databases).
• Experience with cloud data warehouses such as Redshift or Snowflake.
• Strong background in traditional ETL and Big Data engineering, both on-premises and in the cloud.
• Experience building end-to-end data pipelines for unstructured and semi-structured data using Spark.
• Data engineering experience in AWS, including familiarity with relevant services and tools.

Preferred Qualifications

• Experience working in secure environments with access to confidential supervisory information.
• Familiarity with compliance and data governance requirements in regulated industries.