BI Data Engineer II

Jobtailor; Boston (MA); On-site; USD 120,000 - 180,000; Full time

Job summary

Jobtailor is seeking a data engineer to design, build, test, and support Databricks Lakehouse pipelines using Spark, Delta Lake, Python, and SQL for reporting and downstream business use cases.

You will develop and maintain notebooks, workflows, and reusable components, and build analytical datasets across bronze, silver, and gold layers with data quality and lineage in mind.

Qualifications

• Bachelor’s degree or equivalent experience.
• Experience building data pipelines in Databricks using notebooks, jobs, workflows, PySpark or Spark SQL, and Delta Lake.
• Strong SQL and Python skills with production-grade transformation logic.
• Knowledge of Lakehouse architecture, Delta tables, and medallion layers.
• Familiarity with Databricks platform features such as Unity Catalog, Delta Live Tables, Databricks SQL, clusters, or orchestration.
• Ability to apply data validation, governance, and security practices for trusted enterprise data products.
• Experience with source control and automated deployment practices in CI/CD.
• Ability to collaborate with analysts and stakeholders to optimize pipelines.
• Exposure to AI/ML and analytics frameworks is a plus.

Responsibilities

• Design, build, test, and support Databricks Lakehouse data pipelines using Spark, Delta Lake, Python, and SQL for reporting and analytics.
• Develop and maintain Databricks notebooks, workflows, jobs, and reusable components.
• Build curated datasets and analytics-ready models across Lakehouse layers with data quality and lineage.
• Support data ingestion and integration between Databricks, On-Prem SQL Server, SaaS platforms, and other systems.
• Partner with analysts and stakeholders to translate requirements into scalable data solutions.
• Monitor and optimize Spark jobs and Databricks workloads for performance and cost.
• Implement data validation, error handling, and governance standards across Databricks.
• Maintain and evolve the Databricks platform with team members.

Skills

Databricks & Spark

Python

SQL

Lakehouse concepts

Data quality & governance

CI/CD & automation

Collaboration & problem-solving

AI & analytics exposure

Education

Bachelor's degree in CS or related field

Tools

Azure DevOps

Databricks Asset Bundles

Job description

• Design, build, test, and support Databricks Lakehouse data pipelines using Spark, Delta Lake, Python, and SQL for reporting, analytics, and downstream business use cases.
• Develop and maintain Databricks notebooks, workflows, jobs, and reusable pipeline components following team standards for version control, documentation, testing, and deployment.
• Build and maintain curated datasets and analytics-ready models across Lakehouse layers, including bronze, silver, and gold, with attention to data quality, lineage, and business usability.
• Support data ingestion, migration, and integration between Databricks, On-Premise SQL Server, SaaS platforms, and other enterprise systems as part of platform modernization.
• Partner with analysts, data scientists, business stakeholders, and teams across the Data Enterprise organization, including Data Operations and MDM, to translate requirements into scalable and maintainable data solutions.
• Monitor, troubleshoot, and optimize Spark jobs and Databricks workflows for performance, reliability, and cost efficiency under established engineering best practices.
• Implement data validation, error handling, data quality checks, security practices, and governance standards across Databricks.
• Maintain, administer, and support the evolution of our Databricks platform as a shared responsibility with other team members.
• Support CI/CD and automated deployment practices, including Azure DevOps and Databricks Asset Bundles where applicable, to improve repeatability and production readiness.

Requirements

• Bachelor’s degree in Computer Science, a closely related field, or equivalent experience.
• Databricks & Spark: Hands-on experience developing data pipelines in Databricks using notebooks, jobs, workflows, PySpark or Spark SQL, and Delta Lake.
• Programming & SQL: Strong SQL and Python skills, with the ability to write maintainable transformation logic, troubleshoot data issues, and support production pipelines.
• Lakehouse Concepts: Working knowledge of Lakehouse architecture, Delta tables, medallion-style layers, batch processing, and analytics-ready data modeling.
• Databricks Platform Exposure: Exposure to one or more Databricks platform capabilities such as Unity Catalog, Delta Live Tables, Databricks SQL, job clusters, workflow orchestration, performance tuning, cluster configuration, administration, or resource provisioning.
• Data Quality & Governance: Ability to apply data validation, reconciliation, access controls, documentation, and governance practices to support trusted enterprise data products.
• CI/CD & Automation: Exposure to source control and automated deployment practices using tools such as Azure DevOps and Databricks Asset Bundles for reliable, production-ready workflows.
• Collaboration & Problem-Solving: Ability to work effectively with analysts, stakeholders, and cross-functional teams to troubleshoot and optimize pipelines.
• AI & Analytics: Exposure to AI, machine learning, feature engineering, or analytics frameworks within a modern data platform is a plus.