Senior Data Engineer, AI Insights

Menlo Ventures; Toronto; On-site; CAD 113,000 - 187,000; Full time

Job summary

A leading data company in Toronto is looking for a Senior Data Engineer to design scalable data pipelines and collaborate with machine learning and product teams. The ideal candidate will have over 5 years of experience in data engineering, proficiency with SQL and knowledge of Spark. This role offers a competitive salary of $113,000 to $187,000 CAD depending on experience.

Qualifications

• 5+ years of experience as a Data Engineer or similar.
• Experience with cloud platforms like AWS.
• Strong proficiency in SQL and Spark for data processing.

Responsibilities

• Design scalable data pipelines for complex data.
• Define data roadmap and collaborate across teams.
• Ensure data quality and integrity in processes.

Skills

Experience in data modeling

Data warehousing

ETL pipeline development

SQL proficiency

Comfortable with large datasets

Knowledge of Spark

Familiarity with AWS

Tools

Spark

Databricks

AWS ecosystem

Job description

Overview

Affinity stitches together billions of data points from massive datasets to create a powerful, accurate representation of the world's professional relationship graph. Based on this data, we offer our users insights and visibility to nurture and tap into their team's network of opportunities.

This role is part of the AI Insights team, which owns the services that power Affinity's industry-leading relationship intelligence platform. Our team extracts and retrieves information from billions of structured and unstructured data points to deliver insights to our customers.

As a Senior Data Engineer, you will collaborate with machine learning engineers, software engineers, and product managers to shape the future of private capital's leading CRM platform. This involves designing and building scalable, efficient data extraction, load, and transform (ELT) solutions, monitoring and managing data quality, and ensuring data security and best practices.

Responsibilities

• Design scalable and reliable data pipelines to consume, integrate and analyze large volumes of complex data from different sources, supporting the evolving needs of our business.
• Help define our data roadmap. You'll collaborate with our team of machine learning engineers, software engineers, product, and business leaders to use data to shape product development.
• Build and maintain frameworks for measuring and monitoring data quality and integrity.
• Establish and optimize CI/CD processes, test frameworks, and infrastructure-as-code tooling.
• Build and implement robust data solutions using Spark, Python, Databricks, Kafka, and the AWS ecosystem (including S3, Redshift, EMR, Athena, Glue).
• Identify skill and process gaps within the team, and develop processes to drive team effectiveness and success.
• Articulate the trade-offs of different approaches to building ETL pipelines and storage solutions, providing clear recommendations aligned with product and business requirements.

Qualifications

Required:

• 5+ years of experience as a Data Engineer or Data Platform Engineer, working on complex, sometimes ambiguous engineering projects across team boundaries.
• Proficiency in data modeling, data warehousing, and ETL pipeline development is essential.
• Proven hands-on experience building scalable data platforms and reliable data pipelines using Spark and Databricks, and familiarity with Hadoop, AWS SQS, AWS Kinesis, Kafka, or similar technologies.
• Comfortable working with large datasets and high-scale data ingestion, transformation, and distributed processing tools such as Apache Spark (Scala or Python).
• Strong proficiency in SQL.
• Familiar with industry-standard databases and analytics technologies, including Data Warehousing and Data Lakes.
• Experience with cloud platforms such as AWS, Databricks, GCP, Azure or related technologies.
• Familiar with CI/CD processes and test frameworks.
• Comfortable partnering with product and machine learning teams on large, strategic data projects.

Nice to have:

• Hands-on experience with both relational and non-relational database/data stores, including vector databases (e.g. Weaviate, Milvus), graph databases, and text search engines (e.g. OpenSearch or Vespa clusters), with a focus on indexing and query optimization.
• Experience with Infrastructure as Code (IaC) tools, such as Terraform.
• Experience implementing data consistency measures using validation and monitoring tools.

Tech Stack

Our Data stack includes tools to build data pipelines between AWS RDS and DBX via scheduled batch jobs and streaming syncing. Spark SQL and MLlib for large-scale data processing in DBX. We also build data pipelines between RDS and other search-optimized engines, such as OpenSearch. In-house data quality tools and governance tools to ensure data quality, security and compliance.

How we work

Our culture is a key part of how we operate, as well as our hiring process:

• We iterate quickly. As such, you must be comfortable embracing ambiguity, be able to cut through it, and deliver value to our customers.
• We are candid, transparent, and speak our minds while simultaneously caring personally with each person we interact with.
• We make data-driven decisions and make the best decision for the moment based on the information available.

If you’d want to learn more about our values, please note that the role compensation details below reflect the base salary only and do not include any equity or benefits.

Compensation

A reasonable estimate of the current range is $113,000 to $187,000 CAD. Within the range, individual pay depends on various factors including geographical location and review of experience, knowledge, skills, abilities of the applicant.

About Affinity

With more than 3,000 customers worldwide and backed by some of Silicon Valley's best firms, Affinity has raised $120M to empower dealmakers to find, manage, and close more deals. Our Relationship Intelligence platform uses the wealth of data exhaust from trillions of interactions to deliver automated relationship insights that drive over 450,000 deals every month. We are proud to have received Inc. and Fortune Best Workplaces awards as well as to be Great Places to Work certified for the last 5 years running. Join us on our mission to enable anyone to cultivate and harness their network to succeed.

We use E-Verify. Our company uses E-Verify to confirm the employment eligibility of all newly hired employees. To learn more about E-Verify, please visit the following text: E-Verify program information is available at the DHS site.