Senior Databricks Architect - Security Data Platform

DCM Infotech Limited; United States; Remote; USD 180,000 - 280,000; Full time

Job summary

DCM Infotech Limited seeks an experienced Senior Databricks Architect to lead the Security Data Platform strategy for a Pharmaceutical/Life Sciences context. The role focuses on large-scale Databricks deployments processing hundreds of terabytes daily, with thousands of tables and users, and close collaboration with CISO and senior leadership.

Ideal candidates will drive architecture standards, RBAC, and secure data sharing, while guiding engineering teams through implementation and governance

Qualifications

• 8 years of experience in Data Engineering, Data Architecture, or Cloud Data Platforms.
• Strong hands-on experience with Databricks and Apache Spark.
• Proven experience architecting very large-scale Databricks environments.
• Experience handling 300 TB of data per day or comparable large-scale data volumes.
• Experience managing environments with thousands of users and thousands of tables.
• Strong knowledge of Delta Lake, Lakehouse Architecture, Medallion Architecture, Spark, ETL/ELT, and distributed data processing.
• Experience with Databricks performance tuning and large-scale workload optimization.
• Strong understanding of cloud platforms such as AWS, Azure, or Google Cloud Platform.

Responsibilities

• Lead the architecture and technical strategy for a large-scale Data bricks Security Data Platform.
• Design and optimize Data bricks solutions capable of processing approximately 300 TB of data per day.
• Architect highly scalable, secure, and governed data platforms supporting thousands of tables and users.
• Develop architecture standards for security data ingestion, processing, storage, analytics, and consumption.
• Collaborate with CISO, EVP, SVP, and VP-level security leaders to translate objectives into technical solutions.
• Partner with engineering teams to implement and operationalize the target architecture.
• Design secure data architectures with RBAC, governance, access controls, auditing, lineage, encryption, and compliance.
• Establish best practices around Databricks security, governance, performance, scalability, and cost optimization.
• Work with Observability teams to monitor data pipelines, platform health, security events, and performance.
• Provide technical leadership across data engineering, security engineering, cloud, platform, and architecture teams.
• Perform architecture reviews, design reviews, troubleshooting, and code reviews.
• Identify performance bottlenecks and optimize Spark/Databricks workloads.
• Define strategies for data lifecycle management, retention, access management, and secure data sharing.
• Ensure platform meets enterprise security, regulatory, privacy, and compliance requirements for pharma.
• Communicate complex technical concepts to executives and engineers.

Skills

Databricks

Apache Spark

Data Architecture

Security Data

RBAC

Data Governance

Observability

Executive Stakeholder Management

Large-Scale Data Platforms

3rd-Party Cloud Platforms

Tools

Unity Catalog

Delta Live Tables

Auto Loader

Workflows

Terraform

Databricks SQL

Job description

DCM Infotech Limited seeks an experienced Senior Databricks Architect to lead the Security Data Platform strategy for a Pharmaceutical/Life Sciences context. The role focuses on large-scale Databricks deployments processing hundreds of terabytes daily, with thousands of tables and users, and close collaboration with CISO and senior leadership.

Ideal candidates will drive architecture standards, RBAC, and secure data sharing, while guiding engineering teams through implementation and governance