Data Engineering Manager

Jobtailor; Town of Florida (NY); On-site; USD 180,000 - 240,000; Full time

Job summary

Jobtailor is seeking a Senior Data Engineering Leader to drive the Data Foundation initiative and modernize the enterprise data ecosystem with a scalable lakehouse and cloud-based platform capabilities. You will define the data engineering strategy, target architecture, and reusable pipelines to deliver governed, high-quality data products.

You will lead distributed teams, set standards for data modeling (Data Vault 2.0), data quality, and CI/CD testing, and collaborate with Product, Supply

Qualifications

• Minimum 8 years of experience in data engineering including leadership roles.
• Strong hands-on experience with Databricks, PySpark, Spark SQL.
• Proficiency in Python and SQL development.
• Cloud platforms (Azure preferred, AWS acceptable).
• Expertise in modern data architecture and platform design.
• Knowledge of Dimensional modeling and Data Vault 2.0.
• Experience with dbt, Airflow, Azure Data Factory (or equivalents).
• Experience with CI/CD pipelines, automation, and testing practices.
• Experience designing automated testing for data pipelines and end-to-end validation.
• Data quality and integrity testing; CI/CD integration.
• Experience leading distributed teams and collaborating with vendors/stakeholders.

Responsibilities

• Lead the Data Foundation initiative to modernize the data ecosystem with a scalable lakehouse and cloud-based platform capabilities.
• Define the data engineering strategy, target architecture, and reusable pipeline frameworks for governed, high-quality data products.
• Develop the Data Supermarket strategy to deliver business-ready data products across functions.
• Translate business requirements into scalable architecture decisions and delivery plans.
• Provide technical leadership for business separation activities, aligning with future-state operating models.
• Establish and enforce best practices for data modeling, pipeline design, and quality controls.
• Scale data governance, data quality, observability including monitoring and SLAs.
• Define automated testing strategies for data pipelines and CI/CD integration.
• Lead development using Databricks, Python, SQL, dbt, Airflow, and cloud-native tools.
• Partner with vendors and internal teams to deliver outcomes and drive standards.
• Collaborate with Product, Supply Chain, AI/ML, Data Governance, and IT to deliver measurable business impact.

Skills

Databricks

PySpark

Spark SQL

Python

SQL

Cloud Platforms

Data Vault 2.0

Dimensional Modeling

dbt

Airflow

Azure Data Factory

CI/CD

End-to-End Pipeline Validation

Data Governance

Tools

Databricks

dbt

Airflow

Azure Data Factory

Job description

• Lead the Data Foundation initiative to modernize the enterprise data ecosystem through a scalable lakehouse architecture and cloud-based data platform capabilities
• Define the data engineering strategy, target architecture, and reusable pipeline frameworks needed to deliver governed, high-quality data products
• Develop the strategy for a Data Supermarket that delivers business-ready data products for use across multiple functions.
• Translate complex business requirements and technical challenges into scalable architecture decisions and executable delivery plans
• Provide technical leadership for business separation activities, ensuring alignment to future-state operating models and platform continuity
• Establish and enforce best practices for: Data modeling (dimensional, Data Vault 2.0)
• Pipeline design, modularity, and reuse
• Engineering standards and quality controls
• Establish and scale data governance, data quality, and observability practices, including monitoring, lineage, reliability, and service-level expectations
• Define and implement automated testing strategies for data pipelines, including validation, data quality controls, and CI/CD integration
• Lead development and orchestration using Databricks, Python, SQL, dbt, Airflow, and cloud-native tools
• Partner with vendors and internal teams to manage delivery, enforce standards, and drive outcome-based execution
• Collaborate across Product, Supply Chain business, AI/ML, Data Governance, and IT teams to deliver measurable business impact

Requirements

• Minimum of 8 years of experience in data engineering, including 2 or more years in leadership or people management roles
• Strong hands-on technical experience with: Databricks, PySpark, Spark SQL
• Python and SQL development
• Cloud platforms (Azure preferred, AWS acceptable)
• Proven expertise in: Modern data architecture and platform design
• Dimensional modeling and Data Vault 2.0
• Experience with: dbt, Airflow, Azure Data Factory (or equivalent tools)
• CI/CD pipelines, automation frameworks, and testing practices
• Required experience designing and implementing automated testing strategies for data engineering pipelines, including: End-to-end pipeline validation
• Data quality and integrity testing
• Integration with CI/CD pipelines
• Experience leading distributed teams and partnering effectively with external vendors and cross-functional stakeholders

Core Competencies

Demonstrates expertise in modern data architecture, data engineering strategy, and scalable lakehouse architecture, with a strong focus on data quality, governance, and automated testing practices. Proven ability to lead cross-functional teams and deliver high-quality data products that drive business impact.

Highest-signal resume keywords

• Data Engineering Leadership
• Databricks Development
• Automated Testing Strategies
• Cloud Platform Expertise
• Data Governance Practices

ATS Optimization Keywords

Hard Skills

• Data Engineering
• Dimensional Modeling
• Data Vault 2.0
• CI/CD Pipelines
• End-to-End Pipeline Validation
• Data Quality Testing
• Python Development
• SQL Development
• PySpark
• Spark SQL

Soft Skills

• Technical Leadership
• Collaboration
• Stakeholder Management
• Problem Solving
• Communication

Industry Keywords

• Data Ecosystem
• Data Governance
• Data Quality
• Data Products
• Business Separation Activities

Tools & Technologies

• Databricks
• Dbt
• Airflow
• Azure Data Factory
• Cloud-Native Tools