Databricks Engineer

Novalink Solutions; Austin (TX); On-site; USD 140,000 - 190,000; Full time

Job summary

Novalink Solutions is seeking an experienced Databricks Engineer to design, develop, and optimize scalable data solutions in AWS/Azure environments. You will build ingestion pipelines, implement data governance, and create analytics dashboards to drive business insights.

The role requires 8+ years in IT, strong SQL/Python skills, and hands-on experience with Databricks, Delta Lake, and DLt. Excellent communication and collaboration with stakeholders are essential.

Qualifications

• 8+ years in IT delivering technology solutions.
• Experience with Databricks and building ETL/ELT pipelines using Apache Spark.
• Experience in data warehousing and dimensional modeling (star/snowflake).
• Proficiency in SQL and Python (or Scala) for large-scale processing.
• Experience designing dashboards and Databricks-native apps.
• Experience with data governance, data quality, and data security.
• Experience implementing Lakeflow Declarative Pipelines for production pipelines.
• Experience with Delta Lake and medallion architecture; scheduling offline jobs using Lakeflow or Airflow.
• Excellent communication skills.

Responsibilities

• Design, develop, and optimize scalable data solutions on Databricks.
• Build ingestion pipelines, DLt, and medallion architecture to support analytics.
• Develop data models and implement data quality and governance frameworks.
• Create dynamic dashboards and Databricks Apps to deliver insights.
• Optimize workloads, monitoring, and operations for scalability and cost efficiency.

Skills

SQL

Python

Communication

Tools

Databricks

Delta Lake

Airflow

Lakeflow Declarative Pipelines

Databricks SQL

Databricks Apps

Spark / ETL

Job description

8 or more years of experience, relies on experience and judgment to plan and accomplish goals, independently performs a variety of complicated tasks, a wide degree of creativity and latitude is expected.

Understands business objectives and problems, identifies alternative solutions, performs studies and cost/benefit analysis of alternatives. Analyzes user requirements, procedures, and problems to automate processing or to improve existing computer system: Confers with personnel of organizational units involved to analyze current operational procedures, identify problems, and learn specific input and output requirements, such as forms of data input, how data is to be; summarized, and formats for reports. Writes detailed description of user needs, program functions, and steps required to develop or modify computer program. Reviews computer system capabilities, specifications, and scheduling limitations to determine if requested program or program change is possible within existing system.

?The Databricks Engineer should design, develop, and optimize scalable data solutions on Databricks, leveraging PySpark or Scala for large-scale data processing. Build and maintain ingestion pipelines, Declarative Pipelines (DLT), and Medallion Architecture (Bronze, Silver, Gold) to support enterprise analytics and reporting. Develop robust data models and implement data quality, validation, and governance frameworks. Create dynamic dashboards, Databricks Apps, and analytical solutions to deliver actionable business insights. Optimize workloads, monitoring, and operational processes to ensure scalability, security, and cost efficiency.

II. CANDIDATE SKILLS AND QUALIFICATIONS

• |----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------|--------------------|-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------|
• | **Minimum Requirements:** Candidates that do not meet or exceed the **minimum** stated requirements (skills/experience) will be displayed to customers but may not be chosen for this opportunity. |
• | Years | Required/Preferred | Experience|
• | 8 | Required | Experience in IT, supporting the design, development, deployment, or delivery of technology solutions. |
• | 8 | Required | Experience with Databricks, including building and optimizing ETL/ELT data pipelines using Apache Spark. |
• | 8 | Required | Experience in data warehousing and dimensional data modeling (star/snowflake schemas). |
• | 8 | Required | Proficiency in SQL and Python (or Scala) for large-scale data processing. |
• | 8 | Required | Experience designing and developing dashboards and applications natively within Databricks (e.g., Databricks SQL dashboards, Databricks Apps).|
• | 8 | Required | Experience implementing data governance, data quality, and data security practices. |
• | 8 | Required | Experience implementing Lakeflow Declarative Pipelines (formerly Delta Live Tables/DLT) for building and managing production data pipelines. |
• | 8 | Required | Experience with Delta Lake, medallion architecture (bronze/silver/gold layers), data lakehouse design, and creating and scheduling offline jobs using Lakeflow Jobs (formerly Databricks Workflows) or similar orchestration tools (e.g., Airflow). |
• | 8 | Required | Excellent communication skills, both verbal and written, including presenting insights to technical and business stakeholders. |
• | 1 | Preferred | Experience working in public sector or state government environments. |
• | 1 | Preferred | Databricks certification (e.g., Databricks Certified Data Engineer Associate/Professional).|
• | 1 | Preferred | Experience with CI/CD practices for data pipelines (DevOps, Git-based workflows). |