EPIC Data Platform Lead

Nityo Infotech; Santa Clara (CA); On-site; USD 180,000 - 240,000; Full time

Job summary

Nityo Infotech is seeking an EPIC Data Platform Lead to own the technical direction of a secure, governed Databricks-based platform. The role blends hands-on data engineering with architecture leadership across tenancy, data protection, and production readiness, working with stakeholders across data, cloud, security, and governance teams.

You will translate requirements into scalable platform capabilities for internal and customer-dedicated multi-tenant use cases, delivering reliable data

Qualifications

• Bachelor's degree in Computer Science, Engineering, Information Systems, Data Science, or a related field, or equivalent practical experience.
• Strong experience leading the design and implementation of enterprise cloud data platforms, with substantial hands-on Databricks experience.
• Strong working knowledge of Apache Spark, Delta Lake, Databricks Workflows, Unity Catalog, SQL, and Python. Scala experience is beneficial.
• Demonstrated ability to perform complex data analysis, profiling, reconciliation, debugging, performance analysis, and root-cause investigation using large datasets.
• Experience implementing production-grade batch, streaming, micro-batch, event, and file-based ingestion patterns, including schema evolution, replay, backfill, and idempotent processing.
• Experience with cloud-native data services, object storage, IAM, private networking, key management, logging, monitoring, infrastructure as code, and CI/CD. AWS experience is preferred; Azure or GCP experience is also relevant.
• Strong knowledge of data security and governance concepts, including least privilege, identity federation, service principals, classification, lineage, retention, masking, audit logging, DLP, controlled data sharing, and regulated or customer-sensitive data handling.
• Experience designing or operating dedicated-tenant, multi-tenant, or customer-isolated platforms, including tenant lifecycle, logical and physical isolation, resource governance, and cross-tenant security testing.
• Experience implementing encryption in transit and at rest, cloud KMS or HSM integrations, customer-managed keys, BYOK, key rotation, separation of duties, and cryptographic control evidence.
• Strong architecture, technical documentation, stakeholder management, and engineering leadership skills, with the ability to convert ambiguous requirements into executable designs and delivery plans.

Responsibilities

• Define target architecture and engineering standards for EPIC Databricks environments.
• Lead design reviews and make trade-offs across performance, security, operability, scalability, and cost.
• Lead workspace, Unity Catalog, Delta Lake, pipeline, workflow, SQL warehouse, compute policy, external location, storage credential, and deployment-pattern implementation.
• Design and review batch, streaming, and event-driven ingestion; transformation and source-to-target logic; reconciliation; data profiling; exploratory analysis; root-cause analysis; and analytical data products.
• Partner with cybersecurity, IAM, cloud, network, and application teams to implement least privilege, SSO/federation, service principals, secrets management, private connectivity, controlled egress, hardening, vulnerability remediation, and auditable access.
• Implement data classification, taxonomy, ownership, metadata, lineage, retention, access reviews, fine-grained permissions, row filters, column masks, controlled sharing, DLP-aligned controls, and evidence-driven compliance.
• Design and implement encryption in transit and at rest, customer-managed keys and BYOK patterns where required, KMS/HSM integration, key scope and separation, rotation, revocation, monitoring, recovery, and control validation.
• Define tenant onboarding, registry, provisioning, configuration, isolation, routing, metering, offboarding, and migration patterns. Prevent unauthorized cross-tenant access and validate isolation through automated negative testing and periodic control reviews.
• Implement end-to-end logging, auditability, lineage, data-quality monitoring, health dashboards, alerting, SIEM integration, incident response, runbooks, service-level measures, capacity planning, and cost showback.
• Own backlog quality, milestones, dependencies, risk mitigation, release readiness, production cutover, operational handoff, and stakeholder communication. Mentor engineers and coordinate delivery across data, cloud, security, governance, QA, infrastructure, and application teams.

Skills

Databricks

Apache Spark

Delta Lake

Unity Catalog

SQL

Python

Scala

Cloud Architecture

Data Governance

IAM

KMS

CI/CD

Leadership

Security

Education

Bachelor's degree in Computer Science

Equivalent practical experience

Tools

Terraform

Git

Azure

AWS

GCP

Databricks

Unity Catalog

Job description

The EPIC Data Platform Lead will own the technical direction and implementation leadership for a secure, governed, scalable cloud data platform built on Databricks. The role combines hands-on data engineering and analytical problem solving with architecture leadership across tenant isolation, data governance, identity and access, encryption, observability, production readiness, and platform operations. The lead will translate business and engineering requirements into implementable platform capabilities for internal, customer-dedicated, and controlled multi-tenant use cases.

Key Responsibilities

Platform architecture and technical leadership: Define target architecture, engineering standards, roadmaps, decision records, reusable patterns, and non-functional requirements for EPIC Databricks environments. Lead design reviews and make trade-offs across performance, security, operability, scalability, and cost.

Databricks implementation: Lead workspace, Unity Catalog, Delta Lake, pipeline, workflow, SQL warehouse, compute policy, external location, storage credential, and deployment-pattern implementation. Establish maintainable medallion-layer processing and production engineering practices.

Data engineering and analysis: Design and review batch, streaming, and event-driven ingestion; transformation and source-to-target logic; reconciliation; data profiling; exploratory analysis; root-cause analysis; and analytical data products. Use data to validate latency, completeness, linking, accuracy, and business-rule outcomes.

Security by design: Partner with cybersecurity, IAM, cloud, network, and application teams to implement least privilege, SSO/federation, service principals, secrets management, private connectivity, controlled egress, hardening, vulnerability remediation, and auditable access.

Governance and data protection: Implement data classification, taxonomy, ownership, metadata, lineage, retention, access reviews, fine-grained permissions, row filters, column masks, controlled sharing, DLP-aligned controls, and evidence-driven compliance.

Encryption and key management: Design and implement encryption in transit and at rest, customer-managed keys and BYOK patterns where required, KMS/HSM integration, key scope and separation, rotation, revocation, monitoring, recovery, and control validation.

Dedicated and multi-tenant delivery: Define tenant onboarding, registry, provisioning, configuration, isolation, routing, metering, offboarding, and migration patterns. Prevent unauthorized cross-tenant access and validate isolation through automated negative testing and periodic control reviews.

Observability and operations: Implement end-to-end logging, auditability, lineage, data-quality monitoring, health dashboards, alerting, SIEM integration, incident response, runbooks, service-level measures, capacity planning, and cost showback.

Delivery leadership: Own backlog quality, milestones, dependencies, risk mitigation, release readiness, production cutover, operational handoff, and stakeholder communication. Mentor engineers and coordinate delivery across data, cloud, security, governance, QA, infrastructure, and application teams.

Required Qualifications

• Bachelor's degree in Computer Science, Engineering, Information Systems, Data Science, or a related field, or equivalent practical experience.
• Strong experience leading the design and implementation of enterprise cloud data platforms, with substantial hands-on Databricks experience.
• Strong working knowledge of Apache Spark, Delta Lake, Databricks Workflows, Unity Catalog, SQL, and Python. Scala experience is beneficial.
• Demonstrated ability to perform complex data analysis, profiling, reconciliation, debugging, performance analysis, and root-cause investigation using large datasets.
• Experience implementing production-grade batch, streaming, micro-batch, event, and file-based ingestion patterns, including schema evolution, replay, backfill, and idempotent processing.
• Experience with cloud-native data services, object storage, IAM, private networking, key management, logging, monitoring, infrastructure as code, and CI/CD. AWS experience is preferred; Azure or GCP experience is also relevant.
• Strong knowledge of data security and governance concepts, including least privilege, identity federation, service principals, classification, lineage, retention, masking, audit logging, DLP, controlled data sharing, and regulated or customer-sensitive data handling.
• Experience designing or operating dedicated-tenant, multi-tenant, or customer-isolated platforms, including tenant lifecycle, logical and physical isolation, resource governance, and cross-tenant security testing.
• Experience implementing encryption in transit and at rest, cloud KMS or HSM integrations, customer-managed keys, BYOK, key rotation, separation of duties, and cryptographic control evidence.
• Strong architecture, technical documentation, stakeholder management, and engineering leadership skills, with the ability to convert ambiguous requirements into executable designs and delivery plans.

Preferred Qualifications

• Experience with Databricks on AWS, including S3, KMS, PrivateLink, VPC endpoints, IAM roles, CloudTrail, CloudWatch, and enterprise network controls.
• Experience with Kafka or equivalent event-streaming platforms and cloud edge or integration services.
• Experience with Databricks Asset Bundles, Terraform, Git-based workflows, automated testing, release pipelines, policy as code, and environment promotion.
• Experience building data-quality frameworks, lineage, observability, operational dashboards, and cost or usage reporting by tenant.
• Experience with Delta Sharing, APIs, BI tools, MLflow, AI/ML workloads, or governed data-product consumption patterns.
• Knowledge of Zero Trust, NIST CSF, CIS Controls, security architecture reviews, threat modeling, penetration testing, exception management, and audit evidence practices.
• Experience in semiconductor manufacturing, R&D, lab, metrology, equipment telemetry, or OT-integrated data environments is an advantage.
• Relevant Databricks, cloud architecture, data engineering, security, or governance certifications are desirable.