Databricks Technical Lead

Anblicks Inc.; Dallas, Northern (TX, KY); Hybrid; USD 120,000 - 190,000; Full time

Job summary

Anblicks Inc. in Dallas, TX seeks a senior data engineer to design end-to-end data architecture on the Databricks Lakehouse platform.

You will define data engineering best practices, build scalable ETL/ELT pipelines, and collaborate with stakeholders to translate business requirements into technical solutions. You will mentor junior engineers, optimize performance and costs, and ensure data security and governance across cloud environments.

Qualifications

• Experience with Databricks Platform and Lakehouse architectures.
• Expertise in Apache Spark (PySpark/Scala/SQL) and Delta Lake.
• Proficiency in building scalable ETL/ELT pipelines and data modeling.
• Strong SQL skills and cloud platform experience (Azure/AWS).
• Knowledge of Unity Catalog and real-time vs batch processing.

Responsibilities

• Design end-to-end data architecture using the Databricks Lakehouse platform.
• Define data engineering best practices and scalable ETL/ELT pipelines.
• Architect solutions leveraging Spark and Databricks workflows.
• Implement Delta Lake for reliability, performance, and governance.
• Collaborate with stakeholders to translate business requirements into technical solutions.
• Optimize performance, cost, and scalability of data pipelines.
• Provide technical leadership and mentoring to engineers.
• Ensure data security, governance, and compliance standards are met.
• Integrate Databricks with cloud services (Azure, AWS) and other systems.
• Support real-time and batch data processing use cases.

Skills

Databricks Platform

Apache Spark

Delta Lake

Unity Catalog

Lakehouse architecture

Data pipelines

SQL

Cloud platforms

Azure

AWS

Job description

• Design and implement end-to-end data architecture using Databricks Lakehouse platform
• Define data engineering best practices and build scalable ETL/ELT pipelines
• Architect solutions leveraging Apache Spark (PySpark/Scala) and Databricks workflows
• Implement Delta Lake for data reliability, performance, and governance
• Collaborate with stakeholders to understand business requirements and translate them into technical solutions
• Optimize performance, cost, and scalability of data pipelines
• Provide technical leadership , mentoring data engineers and developers
• Ensure data security, governance, and compliance standards are met
• Integrate Databricks with cloud services (Azure, AWS, or GCP) and other enterprise systems
• Support real-time and batch data processing use cases

Required Skills & Qualifications

Technical Skills

• Strong hands-on experience with Databricks Platform
• Expertise in Apache Spark (PySpark / Scala / SQL)
• Experience with Delta Lake, Unity Catalog, and Lakehouse architecture
• Proficiency in data pipelines, ETL/ELT frameworks
• Strong experience in SQL and data modeling
• Hands-on experience with cloud platforms:
  • Azure (ADF, ADLS, Synapse) OR
  • AWS (S3, Glue, EMR) OR