Databricks Engineer

Insight Global; Waltham (MA); On-site; USD 140,000 - 190,000; Full time

Job summary

Insight Global seeks a Databricks Engineer to lead a Databricks implementation for a global pharmaceutical organization. You will stand up and configure workspaces, enforce governance with Unity Catalog, and translate designs into scalable pipelines using Python and PySpark.

You will re-platform report logic, integrate CI/CD, and re-point PowerBI reports to Databricks, leveraging your hands-on Databricks, SQL, and cloud experience within a global team.

Qualifications

• 3+ years of hands-on Databricks production experience and 5+ total YOE in data engineering.
• 1+ real-world example of end-to-end build from empty workspace to Databricks production pipelines.
• Expert-level SQL experience (Window functions, query performance tuning, complex transformation logic).
• Expert-level Python/PySpark experience for pipelines and APIs.
• Git, GitHub, or Databricks Asset Bundles.
• PowerBI exposure, ideally connected to Databricks.
• Experience working in a cloud environment.
• Platform foundation experience (Catalogs, schemas, external locations, grants, row-level security, workspace and compute setup, environment strategy).
• Bachelor's Degree in a related field; Databricks Certification preferred.
• ETL experience.
• Lakehouse experience.
• Agile framework.
• Global team experience.

Responsibilities

• Stand up and configure Databricks workspaces across dev, test, and production environments.
• Implement Unity Catalog governance, including catalogs and schemas.
• Build ingestion pipelines from SaaS, databases, and file sources using Python.
• Migrate and re-platform existing report-driven logic and CI/CD pipelines; re-point PowerBI reports to Databricks.
• Collaborate with a global team to design scalable lakehouse architectures.

Skills

Databricks

SQL

Python / PySpark

Git / GitHub

PowerBI

Cloud experience

Agile

Global team experience

Education

Bachelor's degree in a related field

Databricks Certified

Tools

Databricks Asset Bundles

Job description

Job Description

Insight Global is seeking a Databricks Engineer to join a global pharmaceutical organization to assist with a Databricks implementation. In this role, you will stand up and configure Databricks workspaces across dev, test, and production including compute, SQL warehouses, policies, tagging & cost monitoring. You will also implement Unity Catalog governance, and having this exposure will make you a strong fit for the role; This includes catalog & schema design. Additionally, you will be involved in data architecture/modeling by translating solution design into implementation with naming conventions, table design, and incremental load strategy. From a technical perspective, you will hands‑on build ingestion pipelines from a mix of Saas, database and file-based sources using Python. It’s crucial that this resource comes with hand‑on SQL & PySpark experience to re‑platform existing report‑embedded transformation logic into pipelines. You will be involved in implementing Git‑based CI/CD during this project, and repoint existing PowerBI reports onto Databricks. The perfect fit for this role will be someone who has completed a Databricks implementation on a large‑scale in the past.

We are a company committed to creating diverse and inclusive environments where people can bring their full, authentic selves to work every day. We are an equal opportunity/affirmative action employer that believes everyone matters. Qualified candidates will receive consideration for employment regardless of their race, color, ethnicity, religion, sex (including pregnancy), sexual orientation, gender identity and expression, marital status, national origin, ancestry, genetic factors, age, disability, protected veteran status, military or uniformed service member status, or any other status or characteristic protected by applicable laws, regulations, and ordinances. If you need assistance and/or a reasonable accommodation due to a disability during the application or recruiting process, please send a request to [e-mail] learn more about how we collect, keep, and process your private information, please review Insight Global's Workforce Privacy Policy: https://insightglobal.com/workforce-privacy-policy/.

Skills and Requirements

• 3+ years of hands‑on Databricks experience in production with 5+ total YOE in data engineering
• 1+ real‑world example of end‑to‑end build from empty workspace to Databricks production pipelines
• Expert‑level SQL experience (Window functions, query performance tuning, complex transformation logic)
• Expert‑level Python/PySpark experience for pipelines & APIs
• Git, GitHub, or Databricks Asset Bundles
• PowerBI exposure, ideally connected to Databricks
• Experience working in a cloud environment
• Platform foundation experience (Ex: Catalogs, schemas, external locations, grants, row‑level security, workspace and compute setup, environment strategy)
• Bachelors Degree in a related field -Databricks Certified
• ETL experience
• Lakehouse experience
• Agile framework
• Global team experience