Software Engineer, Data Infrastructure

DatologyAI

Redwood City (CA)

On-site

USD 180,000 - 250,000

Full time

14 days+

Get more replies from employers

Send a job-specific resume in minutes.

Benefits offered by this job

100% covered health benefits

401(k) with 4% company match

Unlimited PTO

Annual wellness stipend

Learning and development stipend

Daily lunches and snacks

Relocation assistance

Job summary

A data platform company in Redwood City, CA, is seeking an experienced Data Platform Engineer to lead the development of core products and drive technical decisions. This role offers competitive compensation ranging from $180,000 to $250,000, along with comprehensive benefits including health coverage and unlimited PTO.

Qualifications

Experience leading and building scalable data processing solutions.
Proficiency in at least one programming language used in Data Engineering.
Experience with ETL frameworks.

Responsibilities

Design and maintain scalable data processing solutions.
Architect back-end systems for data curation.
Collaborate with teams for research capabilities.

Skills

Building production data systems

Data processing solutions (Spark, Flink)

SQL

Data lakes and warehouses (Snowflake, Hive)

Distributed storage systems (HDFS, S3)

Workflow management (Airflow, Dagster)

Python, Scala, or Java proficiency

ETL schedulers expertise (Airflow, Dagster)

End-to-end ownership

Job description

About the Company

Companies want to train their own large models on their own data. The current industry standard is to train on a random sample of your data, which is inefficient at best and actively harmful to model quality at worst. There is compelling research showing that smarter data selection can train better models faster—we know because we did much of this research. Given the high costs of training, this presents a huge market opportunity. We founded DatologyAI to translate this research into tools that enable enterprise customers to identify the right data on which to train, resulting in better models for cheaper. Our team has pioneered deep learning data research, built startups, and created tools for enterprise ML. For more details, check out our recent blog posts sharing our high-level results for text models and image-text models.

We've raised over $57M in funding from top investors like Radical Ventures, Amplify Partners, Felicis, Microsoft, Amazon, and notable angels like Jeff Dean, Geoff Hinton, Yann LeCun and Elad Gil. We're rapidly scaling our team and computing resources to revolutionize data curation across modalities.

This role is based in Redwood City, CA. We are in office 4 days a week.

About the Role

We're looking for an experienced Data Platform Engineer to join as a member of our core Datology AI team. As one of our early senior hires, you will partner closely with our founders on the direction of our product and drive business-critical technical decisions. You will lead the development of our core product and data platform. These are key components of our stack that allow us to process customer data and apply state of the art research for identifying the most informative data points in large-scale datasets. You will have a broad impact over the technology, product, and our company's culture. We provide visa sponsorship for candidates selected for this role.

What You'll Work On

Design, build and maintain highly scalable data processing solutions, while ensuring scalability, reliability, and security.
Architect, build, and deploy the back-end systems and services that power our data curation platform.
Partner with researchers and engineers to bring new features and research capabilities to our customers.
Ensure that our systems are reliable, secure, and worthy of our customers' trust.

About You

Have meaningful experience with leading and building production data systems to deliver on major product initiatives.
- You have built and managed highly scalable data processing solutions (e.g. Spark, Flink), data lakes or warehouses (e.g. Snowflake, Hive), authored queries (SQL), distributed storage systems (e.g., HDFS, S3), used workflow management (e.g. Airflow, Dagster), and have experience maintaining the infra that supports these.
Proficiency in at least one programming language commonly used within Data Engineering, such as Python, Scala, or Java.
Expertise with any of ETL schedulers such as Airflow, Dagster, or similar frameworks.
Experience maintaining a high quality bar for design, correctness, and testing.
Take pride in building and operating scalable, reliable, secure systems.
Have a humble attitude, an eagerness to help your colleagues, and a desire to do whatever it takes to make the team succeed.
Own problems end-to-end, and are willing to pick up whatever knowledge you're missing to get the job done.
You have experience being the technical lead of a Data Engineering / Platform / Infrastructure Team.
Experience building ML/DL systems and/or data infrastructure that feeds into training large ML models.

Don’t meet every single requirement? We still encourage you to apply. If you’re excited about our mission and eager to learn, we want to hear from you!

Compensation

At DatologyAI, we are dedicated to rewarding talent with highly competitive salary and significant equity. The base salary for this position ranges from $180,000 to $250,000.