Data Engineering Lead: Big Data, Cloud & Scalable Solutions

Jobtailor; Massachusetts; Hybrid; USD 150,000 - 210,000; Full time

Job summary

Jobtailor seeks a data engineering leader to drive a team delivering scalable data solutions in a hybrid Boston/Madison role. You will own architecture oversight, data quality, and cross-functional collaboration, while hands-on coding loads and transforms data in the warehouse.

The role emphasizes experience with ETL workflows, cloud platforms, and distributed processing, enabling data-driven decisions across the organization.

Qualifications

• Demonstrated customer-driven solutions, support or service.
• In-depth knowledge of SQL or NoSQL and diverse data stores.
• Extensive Python experience focusing on ETL workflows.
• Ability to generalize code with design patterns.
• Robust, reusable code and shared libraries contribution.
• Expertise in Hadoop or Spark for big data batch processing.
• Experience developing distributed data processing solutions.
• Experience with cloud platforms: AWS, GCP or Azure.
• Knowledge of ML toolkits like sklearn, SparkML, or H2O.
• Strong data understanding and business acumen in data-heavy industries.
• Data modeling principles including dimensional modeling and star schemas.
• Deep knowledge of indexes, binary logging, and transactions.
• Infra-as-code tools: Docker, CloudFormation, or Terraform.
• Experience with Jenkins, CI/CD, and Git workflows.
• Experience authoring and consuming web services.
• Willing to work in a hybrid setup in Madison, WI or Boston, MA.
• Up to 10% travel.
• Offer contingent on background checks.
• Must sign a non-disclosure agreement.

Responsibilities

• Lead an engineering team to meet project deadlines and priorities.
• Supervise data engineering team members and activities.
• Ensure data quality, completeness, security and integrity.
• Document critical workflows and operational support responsibilities.
• Develop understanding of data sources, granularity, availability, and limitations.
• Provide technical oversight to application architecture and development teams.
• Foster reusable, scalable design and operational efficiency of data solutions.
• Create maintainable, scalable code to load and manipulate data in the data warehouse.
• Facilitate communication across project teams, stakeholders, and leadership.
• Collect, store, process, and build applications within the big data platform.
• Integrate applications with the organization-wide architecture.
• Participate in New Employee Orientation during the first week.

Skills

Team Leadership

Communication

Customer-Driven Solutions

Operational Efficiency

Data Lifecycle Understanding

Tools

Docker

Terraform

Jenkins

Git

CloudFormation

Hadoop

Spark

AWS

GCP

Azure

SparkML

sklearn

H2O

Job description

Jobtailor seeks a data engineering leader to drive a team delivering scalable data solutions in a hybrid Boston/Madison role. You will own architecture oversight, data quality, and cross-functional collaboration, while hands-on coding loads and transforms data in the warehouse.

The role emphasizes experience with ETL workflows, cloud platforms, and distributed processing, enabling data-driven decisions across the organization.