Principal Software Data Engineer

ComplexCare Solutions; Nashville (TN); On-site; USD 140,000 - 190,000; Full time

Job summary

ComplexCare Solutions in Nashville seeks a Principal Software Data Engineer to lead the data platform, building lakehouse and warehouse solutions, and creating complex data pipelines using Azure Data Factory and Databricks notebooks. You will mentor teams and drive scalable analytics while partnering with stakeholders to forecast business needs.

This role emphasizes data governance, HIPAA compliance, and DevOps practices, with a focus on performance, security, and reliable production support

Qualifications

• 10+ years of related professional experience in data engineering.
• 5+ years on Databricks platform (Unity Catalog, DLTs, Spark SQL, PySpark).
• 7+ years on data platforms (Lakehouse, data warehouse, data marts).
• 5+ years in cloud environments (AWS/Azure) with 3+ years on Azure.

Responsibilities

• Architect and develop data pipelines and data marts on Databricks/Azure.
• Design and implement ELT processes and analytics in Power BI.
• Collaborate with stakeholders to understand data models and reporting needs.

Skills

Databricks

Azure

PowerBI

ETL pipelines

Python

HIPAA

Education

Bachelors in CS or related field

Databricks certification preferred

Tools

Unity Catalog

Delta Live Tables (DLTs)

DeltaShare

Azure Data Factory

Job description

Principal Software Data Engineer

We are more than just a healthcare company; we are a community of professionals driven by the knowledge that our success leads to a positive impact on the communities that we support. At ComplexCare Solutions, you’ll play a crucial part in innovating solutions that empower health plan members and healthcare providers alike. Our collaborative environment fosters creativity and initiative, ensuring that every idea and effort contributes to meaningful change in the industry. Join us and become a part of a team that turns vision into action, making a tangible impact on the future of healthcare.

Overview

The Principal Software Data Engineer is responsible for contributing to data platform (data Lakehouse and warehouse), data analytics, data operations, and software product development across the organization. They will be part of team building next generation data & reporting platform for internal business stakeholders and external customers to provide insights to understand current state of business and provide insights & forecasting to improve decision‑making strategies and shifts for future business needs. They will architect, design, and develop new data pipelines using Azure Data Factory (ADF) and Databricks notebooks to transform & create complex data marts in Databricks lakehouse. They will work across various teams and systems to understand their data model and business process to build best industry standards ELT process and build analytics by combining heterogeneous data sets. This position requires significant independent work and assisting others with work requests. The Principal Software Data Engineer will also be considered a thought leader who mentors others and provides operational/support leadership across multiple products.

Duties and Responsibilities

• Deal with 15+ data sources (application databases, flat files, csv/excel files etc.) with huge volume of 1TB daily to load & transform to 20TB+ lake house;
• Continuously utilize modern frameworks and tools to drive advanced Lakehouse capabilities and onboard AI/ML use cases;
• Advance use of Databricks key features like Unity Catalog, DLTs, DeltaShare, AI/BI Genie, Agentic Works, Databricks One/Apps etc.;
• Collaborate with business stakeholders, product owners, business analysts to understand various source systems, and reporting needs and create architecture & design for data & reporting platforms on Databricks on Azure Cloud;
• Designing best in class ETL frameworks, data warehouse (star, snowflake, traditional), data export framework, data analytics & visualization design on PowerBI;
• Design various components of data & reporting platform including data ingestion pipeline, validation & cleansing framework, data warehouse, data marts/sets, data export to generate files & populate data marts on Databricks;
• Understand and design data analytics and visualization on BI tools (like PowerBI) utilizing best practices and concepts like multi‑dimensional data models, dashboards with charts, graphs, drill downs, heat maps etc.;
• Reporting performance utilizing caching techniques, API integrations & SQL queries to load large amounts of data;
• Design & write high quality and detailed design documents, create effective & clear visual design diagrams (utilizing Draw.io, MS Visio etc.) of various levels targeting business stakeholders to engineers;
• Design security aspects at various technical components integrating with Azure AD for end‑to‑end platform including row level security in PowerBI reporting;
• Work in agile model to write technical features, groom user stories for developers and work with team to explain technical design and resolve their issues;
• Perform code reviews and performance aspects of all technical components;
• Research new patterns, frameworks, tools to make data platform capabilities configurable, dynamic, reliance and performant, adhere to SLAs & quick time to market;
• Product production support, incident responses, troubleshot data & jobs issues & anomalies. Work with Infrastructure (cloud, security) teams to ensure resiliency, security & compliance;
• Focus on DevOps to automate code build, deployments & various scans (static, dynamic etc). Lead UAT & production releases ;
• Adhere to all confidentiality and HIPAA requirements as outlined within CCS’s Operating Policies and Procedures in all ways and at all times with respect to any aspect of the data handled or services rendered in the undertaking of the position; and
• Fulfill those responsibilities and/or duties that may be reasonably provided by CCS for the purpose of achieving operational and financial success of the Employer.

Job Requirements

• Minimum of 10 years of related professional experience to include;
• 4+ Years of minimum experience working on Databricks platform utilizing Data Loaders, Unity Catalogue, DLTs, Spark SQL, PySpark, AI/BI Genie etc.
• 7+ years of experience working on data platforms (Lakehouse, data warehouse, data marts)
• 5+ years working in cloud environments (like AWS, Azure) with 3+ years on Azure Cloud
• 2+ years working (hands‑on or design or integrating) with Analytics BI tools (like PowerBI, Tableau)
• Experience working on AI/ML, Databricks Agent Bricks or AI/BI Genie or data science concepts will be a big Plus
• Experience working on data governance and utilizing tools like MS Perview, Collibra, DbtLabs will be a big plus
• Expert knowledge of healthcare data formats, HIPAA and PHI will be a huge Plus
• 2+ years of experience on DevOps (CI/CD), Cloud Infrastructure (like storage account, service principles, key vaults, app services etc.)
• Demonstrates a growth mind‑set, seeks feedback often and is effective in continuous personal and professional development;
• Ability to learn quickly and independently;
• Must be agile and flexible and meet changing priorities and deadlines;
• Ability to work independently and to carry out tasks to completion following standard accepted practices;
• Coaching and mentoring skills.

Education

• Bachelors’ degree in computer science, software engineering, or related field; and
• Databricks certified preferred

Physical Demands and Work Environment

• Sedentary work (i.e., sitting for long periods of time);
• Exerting up to 10 pounds of force occasionally and/or negligible amount of force;
• Frequently or constantly to lift, carry push, pull or otherwise move objects and repetitive motions;
• Subject to inside environmental conditions; and
• Travel for this position will include less than 10%, typically for training and collaboration purposes.

ComplexCare Solutions offers competitive compensation and benefits package.

The Company is proud to be an equal opportunity workplace and is an affirmative action employer. We are committed to equal employment opportunity regardless of race, color, ancestry, religion, sex, national origin, sexual orientation, age, citizenship, marital status, disability, gender identity, veteran status, or any other protected characteristic under federal, state, or other applicable laws.

This company utilizes E-Verify.