Senior Data Engineer - Databricks (No Sponsorship Available)

Heritage Construction + Materials; Indianapolis (IN); On-site; USD 120,000 - 180,000; Full time

Job summary

Heritage Construction + Materials seeks a Senior Data Engineer to design and implement strategic data initiatives. You will develop data pipelines in a Cloud Data Platform (Databricks or Snowflake), promote governance, and serve as the cloud data platform expert.

Responsibilities include optimizing Databricks clusters, automating CI/CD with Azure DevOps, integrating Databricks with Azure services, and building monitoring dashboards to ensure data reliability and cost efficiency.

Qualifications

• Bachelor’s or Master’s degree in Computer Science, Engineering, Data Science, or a related field.
• 5+ years of experience in data engineering or a related field.
• Experience with programming languages such as Python/PySpark, SQL, or Scala.
• Experience with cloud environments (Azure preferred) and strong cloud data architecture knowledge.
• Hands-on experience with Databricks or Snowflake Cloud Data Platforms.

Responsibilities

• Develop and support data pipelines within a Cloud Data Platform (Databricks or Snowflake).
• Monitor and optimize Databricks clusters for cost-effective scaling and performance.
• Implement and maintain Delta Lake for reliable data storage with versioning.
• Automate CI/CD pipelines for data workflows using Azure DevOps.
• Integrate Databricks with Azure services like Azure Data Factory, Synapse, and ADLS.
• Design monitoring solutions and dashboards for cluster and job performance.
• Collaborate with cross-functional teams to strengthen data governance via Unity Catalog.

Skills

Python/PySpark

SQL

Scala/Java

Azure Cloud

Databricks/Snowflake

CI/CD

Data governance

BI tools

Education

Bachelor’s or Master’s in CS/Engineering/Data Science

Tools

Databricks

Snowflake

Azure Data Factory

Azure Synapse

ADLS

Job description

Build Your Career at Heritage Construction + Materials!The Senior Data Engineer will help develop and implement strategic data initiatives. They will be responsible for developing and maintaining data pipelines, promoting sustainable data governance, and serving as the organization’s cloud data platform expert.Essential FunctionsDevelop and support data pipelines within a Cloud Data Platform, such as Databricks or SnowflakeMonitor and optimize Databricks cluster performance, ensuring cost-effective scaling and resource utilizationImplement and maintain Delta Lake for optimized data storage, ensuring data reliability, performance, and versioningAutomate CI/CD pipelines for data workflows using Azure DevOpsIntegrate Databricks with other Azure services, such as Azure Data Factory, Azure Synapse, and ADLSDesign and implement monitoring solutions, leveraging Databricks monitoring tools, Azure Log Analytics, or custom dashboards for cluster and job performanceCollaborate with cross-functional teams to support data governance through the use of Databricks Unity CatalogAdditional duties and responsibilities as assigned, including but not limited to continuously growing in alignment with the Company’s core values, competencies, and skills.Education QualificationsBachelor’s or Master’s degree in Computer Science, Engineering, Data Science, or a related fieldExperience Qualifications5+ years of experience in data engineering or a related fieldExpertise with programming languages such as Python/PySpark, SQL, or ScalaExperience working in a cloud environment (Azure preferred) with strong understanding of cloud data architectureHands-on experience with Databricks or Snowflake Cloud Data PlatformsExperience with workflow orchestration (e.g., Databricks Jobs, or Azure Data Factory pipelines)Skills and AbilitiesStrong analytical and problem-solving skillsExperience with programming languages such as Python, Java, R, or SQLExperience with Databricks or Snowflake Cloud Data PlatformsKnowledge of statistical analysis and data visualization tools, such as PowerBI or TableauUnderstanding of business financials and NPV analysisProficient in Microsoft OfficeCreative thinker that is open to working collaboratively with a teamAbout Heritage Construction + MaterialsHeritage Construction + Materials (HC+M) is part of The Heritage Group, a privately held, family-owned business headquartered in Indianapolis. HC+M has core capabilities in infrastructure building. Its collection of companies provides innovative road construction and materials services across the Midwest. HC+M companies, including Asphalt Materials, Inc., Evergreen Roadworks, The Hoosier Company, Milestone Contractors and US Aggregates, proudly employ 3,000 people at 68 locations across seven states. Learn more at www.heritagebuilds.com.All qualified applicants will receive consideration for employment without regard to race, color, religion, sex, sexual orientation, gender identity, national origin, disability, or status as a protected veteran.#HeritageConstruction+Materials