Senior Data & Analytics Platform Architect – AWS & Databricks Lakehouse

Cogniify, Inc.; San Francisco (CA); On-site; USD 180,000 - 240,000; Full time

Benefits offered by this job

Unlimited PTO

Medical, dental and vision coverage

Disability and life insurance

Mental health support

Employer Stock Purchase Program (ESPP)

Yearly team-building experiences

Mentorship opportunities

Open communication with leadership

Job summary

Cogniify, Inc. is seeking a Senior Data & Analytics Platform Architect to design and build a cloud-based data platform on AWS and Databricks.

You will define the lakehouse architecture, automate cloud provisioning, and lead major initiatives with a hands-on, IC mindset. The role requires deep expertise in AWS, Databricks, Terraform, and modern data stack tooling, with a focus on scalable, secure, production-grade platforms in a fast-paced environment in San Francisco Bay Area (remote options not

Qualifications

• 11+ years of related experience (9+years with a Master’s degree).
• At least 10 years of experience in cloud infrastructure and/or enterprise data platforms in a highly available production environment.
• Bachelor’s degree in a relevant field, or an equivalent combination of education and experience. A relevant technical certification may be required.
• Advanced knowledge of AWS, especially data storage, data processing, analytics, security and compute services.
• Advanced knowledge of infrastructure as code, especially Terraform.
• Strong knowledge of Databricks, lakehouse architecture, governance, metadata and data quality.
• Strong working knowledge of Python, SQL, Spark and dbt.
• Intermediate knowledge of cloud and infrastructure automation, and of CI/CD practices.

Responsibilities

• Translate overall direction into plans for the data and analytics platform, working toward long-range goals with a clear understanding of key business processes.
• Define and design the cloud platform, including cloud automation tools and standards, CI/CD pipelines, DevOps tooling and AWS account provisioning automation.
• Lead large, complex projects and major initiatives, setting direction for others and taking many assignments on your own initiative.
• Refine end-to-end development processes, including CI/CD, security scanning and production monitoring.
• Solve unusually complex technical problems and act as the go-to technical expert on new technology direction.
• Communicate across all organizational levels, including influencing executives when needed, and serve as the spokesperson for advanced platform projects.
• Advise management and business stakeholders on technical research, and coordinate with senior external contacts on significant technical matters.
• Oversee documentation of infrastructure and work instructions for systems and applications.
• Develop approaches to prevent future incidents, make sure work delivers long-term value, and make decisions on access, permissions and security controls.
• Serve as the subject matter expert for general engineering and platform strategy, mentor engineers and track industry trends to plan for future platform needs.

Skills

Databricks Lakehouse

Python

SQL

Spark

dbt

Terraform

CI/CD practices

AWS services

Education

Bachelor’s degree in a relevant field

Master’s degree (advantage)

Tools

Databricks

AWS

Terraform

Job description

Solution Architect · Data Science / Data Engineering

Senior Data & Analytics Platform Architect – AWS & Databricks Lakehouse

TheRole

Cogniify is hiring a Senior Data & Analytics Platform Architectto help design and build our cloud-based data and analytics platform on AWS andDatabricks. This role defines and designs the platform, including lakehousearchitecture, cloud automation tools and standards, CI/CD pipelines, DevOpstooling and AWS account provisioning automation, and works with applicationsupport teams on how platform products interface with application installationand configuration.

You will lead large, complex initiatives, refine end-to-enddevelopment processes across CI/CD, security scanning and productionmonitoring, and act as the go-to technical expert on new technology direction.You will advise management on technical research, mentor engineers and trackindustry trends to plan for future platform needs. This is an individualcontributor role.

WhatYou Will Do

• Translate overall direction into plansfor the data and analytics platform, working toward long-range goals with aclear understanding of key business processes.
• Define and design the cloud platform,including cloud automation tools and standards, CI/CD pipelines, DevOps toolingand AWS account provisioning automation.
• Lead large, complex projects and majorinitiatives, setting direction for others and taking many assignments on yourown initiative.
• Refine end-to-end development processes,including CI/CD, security scanning and production monitoring.
• Solve unusually complex technicalproblems and act as the go-to technical expert on new technology direction.
• Communicate across all organizationallevels, including influencing executives when needed, and serve as thespokesperson for advanced platform projects.
• Advise management and businessstakeholders on technical research, and coordinate with senior externalcontacts on significant technical matters.
• Oversee documentation of infrastructureand work instructions for systems and applications.
• Develop approaches to prevent futureincidents, make sure work delivers long-term value, and make decisions onaccess, permissions and security controls.
• Serve as the subject matter expert forgeneral engineering and platform strategy, mentor engineers and track industrytrends to plan for future platform needs.

WhatWe Are Looking For

• 11+ years of related experience (9+years with a Master’s degree).
• At least 10 years of experience in cloudinfrastructure and/or enterprise data platforms in a highly availableproduction environment.
• Bachelor’s degree in a relevant field,or an equivalent combination of education and experience. A relevant technicalcertification may be required.
• Advanced knowledge of AWS, especiallydata storage, data processing, analytics, security and compute services.
• Advanced knowledge of infrastructure ascode, especially Terraform.
• Strong knowledge of Databricks,lakehouse architecture, governance, metadata and data quality.
• Strong working knowledge of Python, SQL,Spark and dbt.
• Intermediate knowledge of cloud andinfrastructure automation, and of CI/CD practices.

PreferredExperience

• Proven experience with lakehousearchitectures on AWS data services (Glue, Lake Formation, Redshift, Athena) andDatabricks.
• Expertise in dbt, Python, Spark SQL andSQL.
• Knowledge of AWS services, includingIAM.
• Experience with Apache Spark, AWS Glue,Databricks, Apache Iceberg and Delta tables.
• Professional AWS and Databrickscertifications.
• Experience in regulated environments,including SOX- and FDA-regulated settings.

Perksand Benefits of Working With Us

• Unlimited PTO and generous parentalleave.
• Medical, dental and vision coverage.
• Access to disability and life insurance.
• Mental health and wellbeing support.
• Employer Stock Purchase Program (ESPP).
• Yearly team-building experiences.
• Mentorship and sponsorshipopportunities.
• Manager resources and support.
• Small, dynamic teams with opportunitiesto make an impact.
• Open communication with management andcompany leadership.
• An entrepreneurial culture thatencourages initiative and new ideas.

We are an equal opportunity employer.We celebrate diversity and are committed to creating an inclusive environmentfor all employees. All qualified applicants will receive consideration withoutregard to race, color, religion, sex, sexual orientation, gender identity,national origin, disability, veteran status or any other protectedcharacteristic.