Databricks & Lakehouse Architect | Spark, Python, Cloud

True Zero Technologies LLC.; United States; Remote; USD 110,000 - 165,000; Full time

Job summary

True Zero Technologies LLC. is seeking a data engineering professional to optimize data pipelines, ensure governance and security, and provide technical leadership across cloud-based Lakehouse architectures.

You will mentor staff and collaborate with engineers to deliver scalable data solutions. The role requires strong Databricks, Spark, Python/PySpark, and SQL skills, plus hands-on experience with AWS or Azure and CI/CD practices.

Qualifications

• 3-5 years of experience in data engineering, data warehousing, or data architecture roles.
• US citizenship and ability to obtain and maintain a federal security clearance may be required.
• Bachelor's degree is highly preferred.

Responsibilities

• Optimize data pipelines, configure clusters, and implement best practices for cost and performance.
• Governance & security: implement security protocols, such as Unity Catalog, IAM, and data access controls.
• Technical leadership: act as a trusted advisor to engineering teams, mentor staff, and define technical roadmaps.
• Collaboration: work with data engineers and stakeholders to align technical solutions with business requirements.
• Technical expertise: strong proficiency in Databricks, Apache Spark, Python/PySpark, and SQL.
• Cloud proficiency: experience with AWS and Azure.
• Data modeling: expertise in data modeling, data profiling, and ETL/ELT pipeline design
• Architecture & development: design and implement scalable data pipelines, data models, and Lakehouse architectures.
• CI/CD: implement CI/CD pipelines, use Azure DevOps or Git for version control, and automate routine tasks.
• Integrate data from various cloud sources (AWS, Azure, GCP) and databases.
• Monitor Spark jobs and system metrics to troubleshoot issues in the Databricks environment.

Skills

Databricks

Apache Spark

Python/PySpark

SQL

AWS

Azure

GCP

Data modeling

ETL/ELT

Lakehouse

CI/CD

Git

Education

Bachelor's degree

Tools

Azure DevOps

Git

Job description

True Zero Technologies LLC. is seeking a data engineering professional to optimize data pipelines, ensure governance and security, and provide technical leadership across cloud-based Lakehouse architectures.

You will mentor staff and collaborate with engineers to deliver scalable data solutions. The role requires strong Databricks, Spark, Python/PySpark, and SQL skills, plus hands-on experience with AWS or Azure and CI/CD practices.