Databricks Solutions Architect

Cogniify; United States; Remote; USD 180,000 - 250,000; Full time

Benefits offered by this job

Unlimited PTO

Generous parental leave

Medical, dental, and vision coverage

Disability and life insurance

Mental health support

Employee Stock Purchase Plan (ESPP)

Yearly team-building experiences

Mentorship opportunities

Management resources and support

Small teams where your work makes a差

Job summary

Cogniify is seeking a seasoned Databricks Solutions Architect to design and lead data platforms across AWS, Azure, and Google Cloud. You will guide data storage, cleaning, protection, and usage, while steering teams from legacy warehouses to Databricks.

You will collaborate with data engineers, product managers, and business teams from design to live deployment, mentoring engineers and shaping governance, security, and CI/CD standards. Occasional travel for workshops is expected.

Qualifications

• Bachelor's/Master's degree in a related field or equivalent practical experience.
• 12+ years in data engineering or data architecture.
• 4+ years hands-on Databricks and Apache Spark experience using PySpark, Python and SQL.
• 3+ production Databricks implementations from design to go-live.
• Strong experience on at least one major cloud (AWS/Azure/GCP).
• Knowledge of Delta Lake, Unity Catalog, performance tuning and cluster sizing.
• Leadership and communication skills with stakeholders in large organizations.
• Experience setting CI/CD and DataOps standards (Git, automated testing).
• Willingness to travel for workshops and milestones.

Responsibilities

• Design lakehouse architectures on AWS, Azure, or GCP with medallion layers, Delta Lake and Unity Catalog.
• Set data pipelines using Lakeflow, Spark, PySpark and Databricks SQL.
• Define data governance, security, and FinOps controls.
• Lead migration from legacy warehouses to Databricks and ensure data parity with old systems.
• Establish CI/CD and DataOps standards including Git and automated testing.
• Mentor engineers and present designs to business leaders in clear terms.
• Travel for workshops and milestones as needed.

Skills

Databricks

Apache Spark

PySpark

Python

SQL

Education

BS/MS in CS/Engineering or related field

Tools

Lakeflow

Unity Catalog

Delta Lake

Terraform

Git

Databricks Asset Bundles

Job description

Cogniify is hiring a Databricks Solutions Architect (also called a Databricks Architect or Lakehouse Architect) to design and lead Databricks data platforms on AWS, Azure and Google Cloud. You will decide how our data is stored, cleaned, protected and used, and guide the team that builds it.

You will work with data engineers, product managers and business teams, from the first design to a live system. You will also lead our move from older data warehouses to Databricks.

Much of our data work is in energy and utilities, such as smart meter, grid and outage data. Experience with this kind of data helps.

This is a remote role based in the United States, with some travel for workshops and key milestones.

What You Will Do

• Design lakehouse architectures on AWS, Azure or Google Cloud, using medallion layers (bronze, silver and gold), Delta Lake and Unity Catalog.
• Set how data pipelines are built with Lakeflow, Apache Spark, PySpark and Databricks SQL, so every engineer on the team builds in the same way.
• Define data governance, security and access control, along with cloud cost controls (FinOps), so data stays safe and spending stays in check.
• Lead the move from older data warehouses and ETL tools to Databricks, and make sure the new numbers match the old ones.
• Set CI/CD and DataOps standards, including Git, Databricks Asset Bundles and automated testing, so releases are safe and repeatable.
• Advise on machine learning and generative AI use cases, using tools such as MLflow and Agent Bricks where they make sense.
• Mentor engineers, run architecture reviews, and explain designs and trade-offs to business leaders and other stakeholders in simple words.
• Travel for workshops, planning sessions and key milestones when needed.

Skills and Tools You Will Use

• Databricks, Databricks SQL, Lakeflow, Unity Catalog and Delta Lake
• Apache Spark, PySpark, Python and SQL
• Lakehouse and medallion architecture, data modeling and data warehousing
• ETL and ELT pipelines, both batch and streaming
• AWS, Microsoft Azure and Google Cloud Platform (GCP)
• Data migration from legacy warehouses such as Teradata, Oracle and SQL Server
• Data governance, data security, data lineage and access control
• CI/CD, Git, DataOps and Databricks Asset Bundles
• Performance tuning, cluster sizing and cloud cost control (FinOps)
• MLflow, machine learning, generative AI and Agent Bricks
• Energy and utility data, including smart meter (AMI), grid and outage data

What We Are Looking For

• A BS, MS or higher in Computer Science, Engineering, Information Systems or a related field, or equivalent practical experience.
• At least 12 years of experience in data engineering or data architecture, with at least 4 years of hands‑on work on Databricks and Apache Spark using PySpark, Python and SQL.
• At least 3 production Databricks implementations that you took from design to go-live.
• Strong experience on at least one major cloud, AWS, Azure or Google Cloud (GCP).
• Deep knowledge of Delta Lake, Unity Catalog, performance tuning and cluster sizing.
• Strong leadership and communication skills, with a record of leading technical teams and working with business stakeholders in large, complex organizations.
• Experience setting CI/CD and DataOps standards, including Git and automated testing.
• Comfort working remotely and traveling when the work needs it.
• Databricks certifications such as Data Engineer Professional, Machine Learning or Generative AI are a plus.
• Databricks architect level or Champion level standing, or equivalent proven experience, is a plus.
• Hands‑on work with Lakebridge migrations, Lakebase or Genie is a plus.
• Experience with streaming and real‑time pipelines (such as Kafka), data modeling and infrastructure as code (such as Terraform) is a plus.
• Experience with energy and utility data, such as smart meter, grid or outage data, is a plus.
• Experience on large enterprise data programs is a plus.

Perks and Benefits of Working With Us

• Unlimited PTO
• Generous parental leave, well above industry standards
• Medical, dental, and vision coverage for employees
• Disability and life insurance
• Mental health and wellbeing support
• Employee Stock Purchase Plan (ESPP)
• Yearly team-building experiences
• Mentorship and sponsorship opportunities
• Manager resources and support
• Small teams where your work makes a real difference
• Open communication with management and company leadership
• An entrepreneurial culture where taking initiative is part of everyday work

Cogniify is an equal opportunity employer. We celebrate diversity and are committed to creating an inclusive environment for all employees. All qualified applicants will receive consideration for employment without regard to race, color, religion, sex, sexual orientation, gender identity, national origin, disability, veteran status, or any other protected characteristic.