Cloud Data Platform Lead for Databricks Migration

Insight Global; Cleveland (OH); On-site; USD 170,000 - 230,000; Full time

Job summary

Insight Global is seeking a Lead Software Engineer to drive large-scale data platform modernization and migration initiatives. You will provide technical leadership for migrating legacy data and workloads to a Databricks-based cloud architecture, collaborating with product owners, architects, and engineers to design scalable solutions.

The role requires deep expertise in data engineering, cloud platforms, and modern software practices, with responsibility across the full migration lifecycle from

Qualifications

• 8+ years of software engineering experience with progressive technical leadership responsibilities.
• Experience leading large-scale data, application, or cloud migration projects.
• Strong experience developing solutions using Apache Spark (PySpark or Scala Spark).
• Experience building and optimizing ETL/ELT pipelines.
• Strong SQL development and database experience.
• Experience with Azure Databricks in a large enterprise environment.
• Experience migrating from traditional data warehouses to Databricks.
• Knowledge of Delta Lake, Unity Catalog, and Medallion architecture.
• Experience with Azure Data Factory, Synapse, or similar tools.
• Power BI knowledge is a plus.
• Experience leading Agile development teams.

Responsibilities

• Lead large-scale data platform modernization and migration initiatives.
• Provide technical leadership for migration of legacy data and workloads to cloud architecture on Databricks.
• Partner with Product Owners, Architects, Data Engineers, and stakeholders to design scalable solutions and set engineering standards.
• Guides team through the full migration lifecycle from assessment and planning to production deployment.

Skills

Leadership

Databricks

Spark PySpark/Scala

Data migration

CI/CD

SQL & Databases

Cloud platforms (Azure/AWS/GCP)

Distributed systems

Stakeholder communication

Agile development

Education

Bachelor's degree in CS/Engineering/IS

Tools

Azure Data Factory

Synapse

Delta Lake

Unity Catalog

Job description

Insight Global is seeking a Lead Software Engineer to drive large-scale data platform modernization and migration initiatives. You will provide technical leadership for migrating legacy data and workloads to a Databricks-based cloud architecture, collaborating with product owners, architects, and engineers to design scalable solutions.

The role requires deep expertise in data engineering, cloud platforms, and modern software practices, with responsibility across the full migration lifecycle from