Databricks Technical Lead

Anblicks; Dallas (TX); On-site; USD 120,000 - 160,000; Full time

Job summary

Anblicks seeks a senior data architect to design and implement end-to-end data architectures on the Databricks Lakehouse platform in Dallas, TX. You will define best practices, build scalable ETL/ELT pipelines, and architect Spark-based solutions to meet business needs.

You will collaborate with stakeholders to translate requirements into robust data solutions, optimize performance and costs, and mentor engineers while ensuring governance and security across cloud integrations (Azure/AWS/GCP).

Qualifications

• Strong hands-on experience with Databricks Platform.
• Expertise in Apache Spark (PySpark / Scala / SQL).
• Experience with Delta Lake, Unity Catalog, and Lakehouse architecture.
• Proficiency in data pipelines, ETL/ELT frameworks.
• Strong experience in SQL and data modeling.
• Hands-on experience with cloud platforms: Azure (ADF, ADLS, Synapse) OR AWS (S3, Glue, EMR) OR GCP.

Responsibilities

• Design and implement end-to-end data architecture using Databricks Lakehouse platform.
• Define data engineering best practices and build scalable ETL/ELT pipelines.
• Architect solutions leveraging Apache Spark (PySpark/Scala) and Databricks workflows.
• Implement Delta Lake for data reliability, performance, and governance.
• Collaborate with stakeholders to understand business requirements and translate them into technical solutions.
• Optimize performance, cost, and scalability of data pipelines.
• Provide technical leadership, mentoring data engineers and developers.
• Ensure data security, governance, and compliance standards are met.
• Integrate Databricks with cloud services (Azure, AWS, or GCP) and other enterprise systems.
• Support real-time and batch data processing use cases.

Skills

Databricks Platform

Apache Spark (PySpark/Scala/SQL)

Delta Lake

Unity Catalog

Lakehouse architecture

SQL & data modeling

Tools

Azure

AWS

GCP

Job description

• Design and implement end-to-end data architecture using Databricks Lakehouse platform
• Define data engineering best practices and build scalable ETL/ELT pipelines
• Architect solutions leveraging Apache Spark (PySpark/Scala) and Databricks workflows
• Implement Delta Lake for data reliability, performance, and governance
• Collaborate with stakeholders to understand business requirements and translate them into technical solutions
• Optimize performance, cost, and scalability of data pipelines
• Provide technical leadership, mentoring data engineers and developers
• Ensure data security, governance, and compliance standards are met
• Integrate Databricks with cloud services (Azure, AWS, or GCP) and other enterprise systems
• Support real-time and batch data processing use cases

Key Responsibilities

• Design and implement end-to-end data architecture using Databricks Lakehouse platform
• Define data engineering best practices and build scalable ETL/ELT pipelines
• Architect solutions leveraging Apache Spark (PySpark/Scala) and Databricks workflows
• Implement Delta Lake for data reliability, performance, and governance
• Collaborate with stakeholders to understand business requirements and translate them into technical solutions
• Optimize performance, cost, and scalability of data pipelines
• Provide technical leadership, mentoring data engineers and developers
• Ensure data security, governance, and compliance standards are met
• Integrate Databricks with cloud services (Azure, AWS, or GCP) and other enterprise systems
• Support real-time and batch data processing use cases

Technical Skills

Required Skills & Qualifications

• Strong hands-on experience with Databricks Platform
• Expertise in Apache Spark (PySpark / Scala / SQL)
• Experience with Delta Lake, Unity Catalog, and Lakehouse architecture
• Proficiency in data pipelines, ETL/ELT frameworks
• Strong experience in SQL and data modeling
• Hands-on experience with cloud platforms:
  • Azure (ADF, ADLS, Synapse) OR
  • AWS (S3, Glue, EMR) OR