Senior Data bricks Architect Security Data Platform

DCM Infotech Limited

United States

Remote

USD 180,000 - 280,000

Full time

10 days ago

Application generator

A complete application in a minute — tailored resume and cover letter, ready to send.

Get past ATS filters

Job summary

DCM Infotech Limited seeks an experienced Senior Databricks Architect to lead the Security Data Platform strategy for a Pharmaceutical/Life Sciences context. The role focuses on large-scale Databricks deployments processing hundreds of terabytes daily, with thousands of tables and users, and close collaboration with CISO and senior leadership.

Ideal candidates will drive architecture standards, RBAC, and secure data sharing, while guiding engineering teams through implementation and governance

Qualifications

8 years of experience in Data Engineering, Data Architecture, or Cloud Data Platforms.
Strong hands-on experience with Databricks and Apache Spark.
Proven experience architecting very large-scale Databricks environments.
Experience handling 300 TB of data per day or comparable large-scale data volumes.
Experience managing environments with thousands of users and thousands of tables.
Strong knowledge of Delta Lake, Lakehouse Architecture, Medallion Architecture, Spark, ETL/ELT, and distributed data processing.
Experience with Databricks performance tuning and large-scale workload optimization.
Strong understanding of cloud platforms such as AWS, Azure, or Google Cloud Platform.

Responsibilities

Lead the architecture and technical strategy for a large-scale Data bricks Security Data Platform.
Design and optimize Data bricks solutions capable of processing approximately 300 TB of data per day.
Architect highly scalable, secure, and governed data platforms supporting thousands of tables and users.
Develop architecture standards for security data ingestion, processing, storage, analytics, and consumption.
Collaborate with CISO, EVP, SVP, and VP-level security leaders to translate objectives into technical solutions.
Partner with engineering teams to implement and operationalize the target architecture.
Design secure data architectures with RBAC, governance, access controls, auditing, lineage, encryption, and compliance.
Establish best practices around Databricks security, governance, performance, scalability, and cost optimization.
Work with Observability teams to monitor data pipelines, platform health, security events, and performance.
Provide technical leadership across data engineering, security engineering, cloud, platform, and architecture teams.
Perform architecture reviews, design reviews, troubleshooting, and code reviews.
Identify performance bottlenecks and optimize Spark/Databricks workloads.
Define strategies for data lifecycle management, retention, access management, and secure data sharing.
Ensure platform meets enterprise security, regulatory, privacy, and compliance requirements for pharma.
Communicate complex technical concepts to executives and engineers.

Skills

Databricks

Apache Spark

Data Architecture

Security Data

RBAC

Data Governance

Observability

Executive Stakeholder Management

Large-Scale Data Platforms

3rd-Party Cloud Platforms

Tools

Unity Catalog

Delta Live Tables

Auto Loader

Workflows

Terraform

Databricks SQL

Job description

Senior Data bricks Architect Security Data Platform

Location: Remote Duration: Long-Term Contract Industry: Pharmaceutical / Life Sciences Role Type: Senior Architect / Technical Leadership

15 20 years in data engineering/data architecture, with 5 years of hands‑on Databricks experience and 3 years leading engineering teams. Has architected enterprise Databricks Lakehouse platforms using Spark, PySpark, Delta Lake, Unity Catalog, Python and SQL across Azure/AWS/Google Cloud Platform. Experienced in batch and streaming pipelines, Workflows, DLT, Auto Loader, performance/cost optimization, CI/CD, Terraform and enterprise governance. Has managed engineers/architects while remaining hands‑on with architecture reviews, code reviews, troubleshooting and client engagements.

Job Overview

We are seeking an experienced Senior Data bricks Architect with a strong background in Pharmaceutical / Life Sciences environments to lead the architecture and technical direction of a large-scale Security Data Platform.

The successful candidate will have hands‑on experience designing and architecting Data bricks platforms processing approximately 300 TB of data per day, with responsibility for highly complex enterprise data environments involving approximately 13,000 tables and 3,000 users.

This role will work closely with senior security and technology leadership, including CISO, EVP, SVP, and VP‑level security stakeholders, as well as engineering and platform teams.

Pharmaceutical / Life Sciences industry experience is mandatory for this position.

Key Responsibilities

Lead the architecture and technical strategy for a large-scale Data bricks Security Data Platform.
Design and optimize Data bricks solutions capable of processing approximately 300 TB of data per day.
Architect highly scalable, secure, and governed data platforms supporting approximately 13,000 tables and 3,000 users.
Develop architecture standards for security data ingestion, processing, storage, analytics, and consumption.
Work closely with CISO, EVP, SVP, and VP‑level Security executives to understand business and security objectives and translate them into technical solutions.
Partner with engineering teams to implement and operationalize the target architecture.
Design secure data architectures incorporating RBAC, data governance, access controls, auditing, lineage, encryption, and compliance.
Establish best practices around Databricks security, governance, performance, scalability, and cost optimization.
Work with Observability teams to develop solutions for monitoring data pipelines, platform health, security events, performance, and operational issues.
Provide technical leadership across data engineering, security engineering, cloud, platform, and architecture teams.
Perform architecture reviews, technical design reviews, troubleshooting, and code reviews.
Identify performance bottlenecks and optimize large-scale Spark/Databricks workloads.
Define strategies for data lifecycle management, retention, access management, and secure data sharing.
Ensure the platform meets enterprise security, regulatory, privacy, and compliance requirements applicable to the pharmaceutical industry.
Communicate complex technical concepts effectively to both executive stakeholders and engineering teams.

Required Qualifications

Databricks & Data Architecture

8 years of experience in Data Engineering, Data Architecture, or Cloud Data Platforms.
Strong hands‑on experience with Databricks and Apache Spark.
Proven experience architecting very large‑scale Data bricks environments.
Experience handling 300 TB of data per day or comparable large-scale data volumes.
Experience managing or architecting environments with thousands of users and thousands of tables.
Strong knowledge of Delta Lake, Lakehouse Architecture, Medallion Architecture, Spark, ETL/ELT, and distributed data processing.
Experience with Databricks performance tuning and large-scale workload optimization.
Strong understanding of cloud platforms such as AWS, Azure, or Google Cloud Platform.

Security & Governance

Strong experience with Data Security and Security Data Platforms.
Experience implementing enterprise RBAC, access controls, auditing, lineage, governance, and secure data sharing.
Strong understanding of Databricks Unity Catalog and enterprise data governance.
Experience working with security, observability, compliance, or cybersecurity data.
Understanding of security monitoring and observability platforms and their integration with data platforms.
Experience working with highly sensitive or regulated enterprise data.

Pharmaceutical / Life Sciences Mandatory

Mandatory: Candidates must have significant experience working within the Pharmaceutical, Life Sciences, Biotechnology, or closely related Healthcare industry.
Experience working with data platforms supporting pharmaceutical or life‑sciences organizations.
Understanding of the security, privacy, compliance, and regulatory requirements associated with pharmaceutical data environments.
Candidates without relevant Pharmaceutical / Life Sciences industry experience will not be considered.

Executive & Stakeholder Management

Experience working directly with CISO, CIO, CTO, EVP, SVP, and VP‑level executives.
Ability to communicate architecture strategy, security risks, platform capabilities, and business value to executive stakeholders.
Strong experience gathering requirements from senior leadership and translating them into actionable technical roadmaps.
Ability to work across executive sponsors, security teams, engineering teams, and enterprise architecture organizations.

Observability Experience

Experience designing or supporting data/platform observability solutions.
Knowledge of monitoring data pipelines, Spark jobs, Databricks workloads, infrastructure, data quality, and platform health.
Experience integrating observability and monitoring capabilities into enterprise data platforms.
Ability to establish operational metrics, alerting, logging, and performance monitoring.

Preferred Qualifications

Databricks certifications.
Experience with Unity Catalog, Delta Live Tables / Lakeflow Declarative Pipelines, Auto Loader, Structured Streaming, Workflows, MLflow, and Databricks SQL.
Experience with Terraform / Infrastructure as Code.
Experience with cloud security and identity platforms.
Experience with SIEM, security analytics, or cybersecurity data platforms.
Knowledge of HIPAA, GxP, FDA, SOX, GDPR, or other relevant regulatory/compliance frameworks.
Experience supporting global pharmaceutical enterprises.
Experience leading architecture initiatives across multiple engineering teams.