Data Engineer II

Talentify; Sunnyvale (CA); On-site; USD 83,000 - 152,000; Full time

Job summary

Talentify is seeking a Contract Data Engineer to build robust data pipelines, focusing on API integration and database management. The role involves Databricks, Snowflake, and relational databases to ensure clean, reliable data for analytics tools like Tableau.

The contractor will design, implement, and optimize ETL/ELT processes, collaborate with BI teams, and maintain data models and documentation in a fast-paced project environment.

Qualifications

• Experience building data pipelines from APIs and handling authentication and parsing.
• Strong SQL skills with database design and query optimization.
• Hands-on with Snowflake, Databricks, and relational databases.

Responsibilities

• Design, develop, and maintain scalable data ingestion pipelines from APIs.
• Implement ETL/ELT processes within the data platform.
• Manage and optimize data storage in Snowflake and Postgres.
• Utilize Databricks for processing and orchestration (PySpark).
• Ensure data quality and integrity across pipelines.
• Collaborate with BI users (Tableau) to shape data models for performance.
• Monitor pipelines for issues and implement fixes.
• Create technical documentation for pipelines and models.
• Troubleshoot data-related issues and provide timely resolutions.
• Follow best practices for data security and governance.

Skills

API integration

SQL/Postgres

Snowflake

Databricks

Python

ETL/ELT

Data quality

Cloud platforms

Tableau BI

Problem solving

Communication

Tools

Postgres

Snowflake

Databricks

Airflow

dbt

Docker

Job description

Job Title: Contract Data Engineer (API & Database Focus)

Role Summary:

We are seeking an experienced and highly skilled Contract Data Engineer. This role is focused on building robust data pipelines, primarily involving the integration of data from various APIs and managing critical data infrastructure. The successful candidate will work hands-on with Databricks, Snowflake, and/or relational databases, ensuring data is clean, reliable, and optimized for consumption by analytics tools like Tableau. This is a project-based contract position requiring someone who can quickly integrate and deliver results.

Key Responsibilities:

• Design, develop, and maintain scalable data ingestion pipelines to extract data from various third-party and internal APIs (REST, SOAP, etc.).
• Implement efficient data transformation and loading processes (ETL/ELT) within the data platform.
• Manage and optimize data storage and schemas in Snowflake and Postgres databases.
• Utilize Databricks for data processing, transformation, and orchestration tasks.
• Ensure data quality, accuracy, and integrity throughout the data pipelines.
• Collaborate with data analysts and BI developers (particularly Tableau users) to understand data requirements and optimize data models for performance.
• Monitor data pipelines and systems for performance issues, errors, and data discrepancies, implementing necessary fixes and improvements.
• Develop and maintain technical documentation for data pipelines, processes, and data models.
• Troubleshoot data-related issues and provide timely resolutions.
• Implement best practices for data security and governance within the data platform.

Required Skills and Qualifications:

• Proven experience as a Data Engineer, with a strong focus on API integration and database management.
• Experience building data pipelines for API data ingestion, including handling authentication, error handling, and data parsing.
• Strong proficiency in SQL and experience with database design, query optimization, and performance tuning in relational databases (e.g., Postgres).
• Hands-on experience with Snowflake as a cloud data warehouse, including data loading, querying, and performance optimization.
• Experience using Databricks for data processing, ETL/ELT, and pipeline orchestration (e.g., PySpark, notebooks).
• Proficiency in a programming language commonly used for data engineering (e.g., Python).
• Experience working with cloud platforms (e.g., AWS, Azure, GCP) where Snowflake and Databricks are deployed.
• Understanding of data warehousing concepts and best practices.
• Experience supporting BI tools (like Tableau) by providing clean, structured, and performant data sources.
• Excellent problem-solving skills and ability to work independently in a fast-paced environment.
• Strong communication and collaboration skills.

Desired Skills (Nice to Have):

• Experience with specific ETL/orchestration tools (e.g., Airflow, Fivetran, dbt).
• Familiarity with containerization technologies (e.g., Docker).
• Experience in a contract or consulting role.
• Knowledge of data governance and compliance standards.
• Experience with BI tools like Tableau, including dashboard creation or assisting BI developers with visualization best practices related to data structure.