Lead Data Engineer

Jobtailor; Massachusetts; Hybrid; USD 150,000 - 210,000; Full time

Job summary

Jobtailor seeks a data engineering leader to drive a team delivering scalable data solutions in a hybrid Boston/Madison role. You will own architecture oversight, data quality, and cross-functional collaboration, while hands-on coding loads and transforms data in the warehouse.

The role emphasizes experience with ETL workflows, cloud platforms, and distributed processing, enabling data-driven decisions across the organization.

Qualifications

• Demonstrated customer-driven solutions, support or service.
• In-depth knowledge of SQL or NoSQL and diverse data stores.
• Extensive Python experience focusing on ETL workflows.
• Ability to generalize code with design patterns.
• Robust, reusable code and shared libraries contribution.
• Expertise in Hadoop or Spark for big data batch processing.
• Experience developing distributed data processing solutions.
• Experience with cloud platforms: AWS, GCP or Azure.
• Knowledge of ML toolkits like sklearn, SparkML, or H2O.
• Strong data understanding and business acumen in data-heavy industries.
• Data modeling principles including dimensional modeling and star schemas.
• Deep knowledge of indexes, binary logging, and transactions.
• Infra-as-code tools: Docker, CloudFormation, or Terraform.
• Experience with Jenkins, CI/CD, and Git workflows.
• Experience authoring and consuming web services.
• Willing to work in a hybrid setup in Madison, WI or Boston, MA.
• Up to 10% travel.
• Offer contingent on background checks.
• Must sign a non-disclosure agreement.

Responsibilities

• Lead an engineering team to meet project deadlines and priorities.
• Supervise data engineering team members and activities.
• Ensure data quality, completeness, security and integrity.
• Document critical workflows and operational support responsibilities.
• Develop understanding of data sources, granularity, availability, and limitations.
• Provide technical oversight to application architecture and development teams.
• Foster reusable, scalable design and operational efficiency of data solutions.
• Create maintainable, scalable code to load and manipulate data in the data warehouse.
• Facilitate communication across project teams, stakeholders, and leadership.
• Collect, store, process, and build applications within the big data platform.
• Integrate applications with the organization-wide architecture.
• Participate in New Employee Orientation during the first week.

Skills

Team Leadership

Communication

Customer-Driven Solutions

Operational Efficiency

Data Lifecycle Understanding

Tools

Docker

Terraform

Jenkins

Git

CloudFormation

Hadoop

Spark

AWS

GCP

Azure

SparkML

sklearn

H2O

Job description

• Lead an engineering team to meet project deadlines and priorities


• Supervise assigned data engineering team members and activities


• Ensure data quality, completeness, security, privacy, and integrity throughout the data lifecycle


• Document critical workflows and operational support responsibilities


• Develop a deep understanding of data sources, granularity, availability, and limitations


• Provide technical oversight and advice to application architecture and development teams


• Foster reuse, scalable design, stability, and operational efficiency of data and analytical solutions


• Create maintainable, scalable code to load and manipulate data in the data warehouse


• Facilitate communication across project teams, business stakeholders, and leadership


• Collect, store, process, and build applications within the big data platform


• Integrate applications with the organization-wide architecture


• Participate in New Employee Orientation during the first week

Requirements

• Demonstrated experience providing customer-driven solutions, support or service


• In-depth knowledge of SQL or NoSQL and experience using a variety of data stores, including RDBMS, analytic databases, and scalable document stores


• Extensive hands-on Python programming experience, emphasizing ETL workflows and data-driven solutions


• Ability to employ design patterns and generalize code for common use cases


• Ability to author robust, high-quality, reusable code and contribute to shared libraries


• Expertise in big data batch computing tools such as Hadoop or Spark


• Demonstrated experience developing distributed data processing solutions


• Applied knowledge of cloud computing, including AWS, GCP, or Azure


• Knowledge of open source machine learning toolkits such as sklearn, SparkML, or H2O


• Solid data understanding and business acumen in data-rich industries such as insurance or financial services


• Applied knowledge of data modeling principles, including dimensional modeling and star schemas


• Strong understanding of database internals, including indexes, binary logging, and transactions


• Experience with infrastructure-as-code tools such as Docker, CloudFormation, or Terraform


• Experience with software engineering tools and workflows, including Jenkins, CI/CD, and git


• Practical experience authoring and consuming web services


• Ability to work in a hybrid arrangement in Madison, Wisconsin or Boston, Massachusetts


• Up to 10% travel


• Offer contingent on applicable background checks


• Must sign a non-disclosure agreement covering proprietary information, trade secrets, and inventions

Core Competencies

Demonstrates expertise in data engineering, including SQL and NoSQL database management, Python programming for ETL workflows, and big data processing with tools like Hadoop and Spark. Proven ability to lead teams, ensure data quality, and integrate applications within cloud environments such as AWS, GCP, or Azure.

Highest-signal resume keywords

• Data Engineering Leadership


• Python Programming for ETL


• Big Data Processing with Hadoop or Spark


• SQL and NoSQL Database Management


• Cloud Computing Knowledge (AWS, GCP, Azure)

Hard Skills

• SQL


• NoSQL


• Python


• Hadoop


• Spark


• Data Modeling


• ETL Workflows


• Distributed Data Processing


• Infrastructure-as-Code (Docker, Terraform)

Soft Skills

• Team Leadership


• Communication


• Customer-Driven Solutions


• Operational Efficiency


• Collaboration

Industry Keywords

• Data Quality


• Data Lifecycle


• Data Warehouse


• Business Acumen


• Insurance


• Financial Services

Tools & Technologies

• AWS


• GCP


• Azure


• Jenkins


• CI/CD


• Git
  
• CloudFormation


• H2O


• Sklearn


• SparkML