Data Engineer

Tavily; New York (NY); On-site; USD 100,000 - 130,000; Full time

Job summary

Tavily is looking for a Data Engineer to join its team in New York, NY. As part of our innovative company, you will design, construct, and maintain scalable data management systems crucial for our Search API.

The ideal candidate has at least 3 years of experience in data engineering, with expertise in Python, SQL, and AWS. You will work collaboratively to ensure data integrity and performance, while also optimizing infrastructure.

This role offers a unique opportunity to shape the data systems that drive our product.

Qualifications

• 3+ years of professional experience as a Data Engineer or in a similar role focused on data infrastructure.
• Solid experience with relational and NoSQL databases.
• Experience with data governance and security best practices.

Responsibilities

• Design and maintain efficient ETL/ELT pipelines for data warehousing.
• Develop and improve data infrastructure.
• Ensure data quality, integrity, and security.

Skills

Python

Data pipeline optimization

SQL

NoSQL

AWS data services

Education

Degree in Computer Science, Statistics, Engineering, or related field

Tools

MongoDB

Snowflake

Redis

Airflow

Docker

Kubernetes

Job description

About Tavily - Search API

Tavily is a cutting-edge company focused on providing a powerful and efficient Search API. We empower developers and businesses by delivering high-quality, relevant search results quickly and reliably. Join our team to help build the data infrastructure that supports and scales our core product.

The Role

We are seeking a highly motivated and experienced Data Engineer to join our growing team. You will be responsible for designing, constructing, installing, testing, and maintaining highly scalable data management systems. You will work closely with our engineering and devops teams to build and optimize the data pipelines that are crucial to the performance and accuracy of our Search API.

Responsibilities

• Design, build, and maintain efficient and reliable ETL/ELT pipelines for data warehousing.
• Develop and improve our data infrastructure.
• Ensure data quality, integrity, and security across all data platforms.
• Optimize data systems for performance and scalability.
• Troubleshoot and resolve issues related to data pipelines and data infrastructure.
• Collaborate with cross-functional teams to understand data needs and deliver solutions.

Minimum Qualifications

• A degree in Computer Science, Statistics, Engineering, or a related quantitative field.
• 3+ years of professional experience as a Data Engineer or in a similar role focused on data infrastructure.
• Proficiency in Python.
• Solid experience with relational databases (SQL) and NoSQL databases.
• Experience with AWS and their data services.
• Proven experience in building and optimizing data pipelines and data architectures.

Preferred Qualifications

• Experience with Mongo/Snowflake/Redis/S3 General/S3 Express.
• Experience with big data technologies
• Experience with Airflow.
• Familiarity with containerization and orchestration tools (e.g., Docker, Kubernetes).
• Experience working in a company focused on API services or search technology.
• Knowledge of data governance and data security best practices.