Databricks Architect

Quantum Integrators; Raritan (NJ); On-site; USD 150,000 - 220,000; Full time

Job summary

Quantum Integrators is seeking an experienced Databricks Architect to design scalable data platforms using the Databricks Lakehouse architecture. The role focuses on enterprise data engineering within Life Sciences, collaborating with business and technical stakeholders to deliver secure, compliant data solutions.

The candidate will lead architecture reviews, mentor engineers, and optimize workloads for cost, performance, and reliability across Azure, AWS, or Google Cloud environments.

Qualifications

• 10+ years in data engineering and data architecture.
• 3-5+ years hands-on experience with Databricks.
• Experience in Life Sciences industry (pharma/biotech/medical devices/healthcare).
• Strong knowledge of Apache Spark (PySpark/Scala/Spark SQL).
• Proficient in SQL and Python.
• Experience with Delta Lake, Unity Catalog, and Databricks Workflows.
• Experience with Azure/AWS/GCP cloud platforms.

Responsibilities

• Design and implement enterprise-scale data platforms using Databricks.
• Architect end-to-end data solutions: ingestion, transformation, storage, analytics.
• Develop scalable ETL/ELT pipelines with Spark and Databricks.
• Design/optimize Delta Lake architectures for performance and reliability.
• Collaborate with stakeholders to translate data requirements into tech solutions.
• Define data governance, security, and best practices for Databricks.
• Lead architecture reviews, technical design sessions, and code reviews.
• Mentor data engineers and establish development standards.
• Integrate Databricks with enterprise data sources, APIs, and reporting tools.
• Support production deployments, troubleshooting, and performance tuning.
• Ensure data solutions align with Life Sciences regulatory requirements.

Skills

Leadership

Stakeholder management

Communication

Data modeling

Problem solving

Analytical thinking

Mentoring

Architecture design

Presentation

Education

Bachelor’s or Master’s in CS/Engineering or related field

Tools

Databricks

PySpark

Python

SQL

Unity Catalog

Databricks Workflows

Git

CI/CD

Job description

We are seeking an experienced Databricks Architect to design, architect, and implement scalable data platforms using the Databricks Lakehouse architecture. The ideal candidate will have extensive experience in modern data engineering, cloud platforms, and enterprise data architecture within the Life Sciences industry. This role requires close collaboration with business and technical stakeholders to deliver secure, scalable, and compliant data solutions.

Key Responsibilities

• Design and implement enterprise-scale data platforms using Databricks.
• Architect end-to-end data solutions, including data ingestion, transformation, storage, and analytics.
• Develop scalable ETL/ELT pipelines using Apache Spark and Databricks.
• Design and optimize Delta Lake architectures for performance and reliability.
• Collaborate with business stakeholders to understand data requirements and translate them into technical solutions.
• Define data governance, security, and best practices for the Databricks environment.
• Optimize workloads for cost, scalability, and performance.
• Lead architecture reviews, technical design sessions, and code reviews.
• Mentor data engineers and establish development standards.
• Integrate Databricks with enterprise data sources, APIs, and reporting platforms.
• Support production deployments, troubleshooting, and performance tuning.
• Ensure data solutions align with Life Sciences regulatory and compliance requirements, where applicable.

Required Qualifications

• Bachelor’s or Master’s degree in Computer Science, Information Systems, Engineering, or a related field.
• 10+ years of experience in data engineering and data architecture.
• 3-5+ years of hands-on experience with Databricks.
• Experience working in the Life Sciences industry (pharmaceutical, biotechnology, medical devices, or healthcare).
• Strong expertise in Apache Spark (PySpark/Scala/Spark SQL).
• Strong SQL and Python programming skills.
• Experience with Delta Lake, Unity Catalog, and Databricks Workflows.
• Experience with Azure, AWS, or Google Cloud.
• Strong understanding of data lakehouse architecture, data warehousing, and data modeling.
• Experience implementing CI/CD pipelines and DevOps practices.
• Excellent communication, leadership, and stakeholder management skills.

Preferred Qualifications

• Databricks Certified Professional Data Engineer or Databricks Certified Solutions Architect.
• Experience with Azure Data Factory, AWS Glue, or similar orchestration tools.
• Knowledge of streaming technologies such as Kafka or Spark Structured Streaming.
• Experience with Power BI, Tableau, or other BI platforms.
• Familiarity with Life Sciences data standards and regulatory environments (e.g., GxP, FDA 21 CFR Part 11, HIPAA, where applicable).
• Experience supporting AI/ML or advanced analytics workloads on Databricks.

Technical Skills

• Databricks
• PySpark
• Python
• SQL
• Unity Catalog
• Databricks Workflows
• Git
• CI/CD
• Data Lakehouse Architecture
• Data Modeling
• Performance Tuning

Soft Skills

• Strong leadership and architectural decision-making abilities.
• Excellent communication and presentation skills.
• Ability to engage effectively with business, IT, and executive stakeholders.
• Strong analytical and problem-solving skills.
• Ability to mentor and guide technical teams.

Work Model

• Full-time
• Onsite
• Experience working directly with business stakeholders in a Life Sciences environment is highly preferred.