Lead Data Engineer - Databricks

Nava Software Solutions LLC; Houston (TX); On-site; USD 150,000 - 210,000; Full time

Job summary

NAVA Software in Houston, TX is seeking a Lead Data Engineer to define and implement enterprise data architecture strategies, leveraging AWS, data lakes, lakehouse, and data mesh to support business goals.

You will lead modernization efforts, design scalable pipelines (batch and streaming) using AWS Glue, SageMaker, Databricks, and govern data with metadata and lineage. Mentorship and technical leadership across analytics teams are required.

Qualifications

• Bachelor's or master's degree in CS, engineering, or related field.
• 7–10 years in cloud data architecture, with at least 5 years in AWS.
• Proficiency with AWS services listed (Glue, Redshift, Athena, Lake Formation, SageMaker, Bedrock, Step Functions).
• Extensive experience with Databricks.
• Experience with the Alation data governance tool.
• Strong SQL (T-SQL), Python (PySpark/Pandas), DAX, Power Query (M), PL/SQL.
• Experience with SQL Server, Oracle, PostgreSQL, Azure SQL, Teradata.
• Power BI, Git, VS Code, SSMS.
• Knowledge of Medallion data architecture, data mesh, Lakehouse, SOA data layers.
• AWS and Databricks certifications preferred.
• Excellent communication, collaboration, and leadership abilities.

Responsibilities

• Develop and execute long-term data architecture vision aligned with business goals.
• Design scalable data systems, including Data Lakes, Lakehouse, and Data Mesh.
• Lead cloud migration strategies and data modernization projects.
• Collaborate with stakeholders to ensure architectural alignment.
• Architect and optimize batch and streaming pipelines using AWS Glue.
• Implement data governance, metadata management, lineage tracking, and compliance frameworks.
• Integrate data platforms to support AI/ML workflows (SageMaker, Bedrock, Databricks).
• Provide technical leadership and mentorship for data analysts and engineers.
• Conduct technical assessments and facilitate stakeholder workshops.
• Ensure best practices in version control, automation, and documentation.

Skills

Cloud data architecture

AWS

Leadership

SQL (T-SQL)

Python (PySpark)

Data modeling

Communication

Mentorship

ETL/ELT pipelines

Streaming data

Education

Bachelor's or Master's in Computer Science/Engineering or related field

Tools

AWS Glue

Redshift

Athena

Lake Formation

SageMaker

Bedrock

Databricks

Alation

Git

Visual Studio Code

SSMS

Job description

NAVA Software is looking for a Lead Data Engineer

Details:

Lead Data Engineer

Location: Houston TX Monday to Thursday onsite

Duration: 6-12 months

Position Overview

The Lead Data Engineer is responsible for defining, implementing, and overseeing enterprise-wide data architecture strategies that align with business objectives. This role requires deep expertise in cloud platforms (especially AWS), advanced data modeling, and the ability to lead modernization initiatives across the organization.

Key Responsibilities

• Develop and execute long-term data architecture vision in alignment with business goals.
• Design and implement scalable data systems, including Data Lakes, Lakehouse, and Data Mesh architecture.
• Lead cloud migration strategies and data modernization projects.
• Collaborate with business and technology stakeholders to ensure architectural alignment.
• Architect and optimize batch and streaming data pipelines using appropriate tools such as AWS Glue.
• Implement data governance, metadata management, lineage tracking, and compliance frameworks.
• Integrate data platforms to support AI/ML workflows (SageMaker, Bedrock, Databricks).
• Provide technical leadership, mentorship, and oversight for data analysts and engineers.
• Conduct technical assessments and facilitate stakeholder workshops.
• Ensure best practices in version control, automation, and documentation.

Required Skills & Qualifications

• Bachelor's or Master's degree in Computer Science, Engineering, or related field.
• 7 10 years of implementation experience in cloud data architecture, with at least 5 years in AWS environments.
• Proficiency in AWS services: Glue, Redshift, Athena, Lake Formation, SageMaker, Bedrock, Step Functions.
• Extensive experience working with Databricks.
• Working knowledge of the Alation data governance tool.
• Strong skills in SQL (T-SQL), Python (PySpark/Pandas), DAX, Power Query (M), PL/SQL.
• Experience with databases: SQL Server, Oracle, PostgreSQL, Azure SQL, Teradata.
• Skilled in Power BI, Git, Visual Studio Code, SSMS.
• Knowledge of architectural patterns: Medallion architecture, data mesh, Lakehouse, SOA data layers.
• AWS and Databricks certifications are preferred.
• Excellent communication, collaboration, and leadership abilities.

Preferred Experience

• Leading cloud migration and data modernization initiatives.
• Designing and implementing scalable data architectures in AWS and Databricks.
• Managing data governance, security, and compliance in complex environments.
• Mentoring junior architects and engineers.
• Working with cross-functional teams to deliver business value through data.

Additional Information

• This role may require occasional travel for stakeholder workshops or technical assessments.
• The Senior Data Engineer will play a key role in shaping the organization's data strategy and future readiness for AI/ML initiatives.