Data Engineer / Data Architect

Innova software Services Inc; Charlotte (NC); On-site; USD 120,000 - 180,000; Full time

Job summary

Innova software Services Inc in Charlotte, NC is seeking a Senior Data Engineer / Data Architect with deep Databricks hands-on expertise and Insurance domain experience to design, build, and implement scalable enterprise data solutions that support analytics at scale.

You will lead the development of data lakes and lakehouse platforms, optimize Spark jobs, govern data quality and security, and translate business requirements into robust architectures across policy, claims, underwriting, and

Qualifications

• 12+ years in Data Engineering or Data Architecture.
• Expertise with Databricks, Spark, Python, SQL.
• Experience building data lakes/lakehouses and data warehouses.
• Cloud platform experience (Azure preferred; AWS/GCP valued).
• Knowledge of data governance, security, and lineage.
• Strong collaboration with stakeholders and engineers.

Responsibilities

• Design and build enterprise-scale data architecture and pipelines using Databricks, Spark, Python, and SQL.
• Architect data lakes, lakehouses, and data warehouses with Delta Lake, DLT, Unity Catalog.
• Develop high-performance batch/streaming ETL/ELT pipelines and data ingestion.
• Optimize Spark jobs and queries for performance and cost efficiency.
• Implement governance, data quality, security, lineage, and access controls.
• Translate business requirements into scalable technical architectures and mentor peers.

Skills

Databricks

Apache Spark

Python

SQL

Data Architecture

Azure

AWS

GCP

Delta Lake

Delta Live Tables

Unity Catalog

Data Modeling

ETL/ELT

Lakehouse

CI/CD

Azure Data Factory

Airflow

Snowflake

Kafka

Education

Bachelor's degree in Computer Science, IT, Engineering, or related field

Tools

Azure Data Factory

ADLS

AWS S3

Glue

Snowflake

Kafka

Airflow

Git

CI/CD tooling

Job description

Job Title: Senior Data Engineer / Data Architect Databricks

Location: Charlotte, NC

Work Arrangement: Onsite

Job Type: Contract

Industry: Insurance

Experience Level: 12+ Years

Job Summary

We are seeking a highly experienced Senior Data Engineer / Data Architect with deep Databricks hands-on expertise and strong Insurance domain experience to design, build, and implement scalable enterprise data solutions.

The ideal candidate brings 12+ years of experience across Databricks, Apache Spark, Python, SQL, data architecture, and modern cloud platforms. You will collaborate with business stakeholders, cross-functional engineering teams, and analysts to architect next-generation data platforms and analytics solutions.

Key Responsibilities

• Design, build, and optimize enterprise-scale data architecture and pipelines using Databricks, Apache Spark, Python, and SQL.

• Architect robust data lakes, lakehouse platforms, and data warehouses utilizing Delta Lake, Delta Live Tables (DLT), Unity Catalog, and Databricks Workflows.

• Develop high-performance batch and streaming ETL/ELT data pipelines, including data ingestion, transformation, cleansing, and validation.

• Optimize Spark jobs, SQL queries, and complex workloads for high performance and cost efficiency.

• Implement enterprise data governance, data quality standards, security policies, lineage, and access controls.

• Translate complex business requirements into scalable technical data architectures and engineering solutions.

• Troubleshoot production data issues, provide root-cause analysis, and mentor engineering team members.

Insurance Domain Responsibilities

• Apply strong domain knowledge across Property & Casualty (P&C), Life, or Health Insurance data (e.g., Policy, Claims, Premium, Billing, Underwriting, Rating, Coverage, Loss, and Risk).

• Integrate data across core insurance applications such as policy administration, claims management, underwriting engines, and billing systems.

Technical Skills & Qualifications

• Total Experience: 12+ years of progressive experience in Data Engineering, Data Architecture, or Enterprise Data Solutions.

• Core Technologies: Databricks (Workspace, PySpark, Delta Lake, DLT, Unity Catalog, SQL, CI/CD), Apache Spark, Python, and advanced SQL.

• Data Architecture: Expertise in Data Lakehouse, Data Warehousing, ETL/ELT design, and dimensional/reusable data modeling.

• Cloud Infrastructure: Strong hands-on experience with major cloud platforms (Azure preferred; AWS or GCP highly valued) and integration tools (e.g., Azure Data Factory, ADLS, AWS S3, Glue, Snowflake, Kafka, Airflow).

• Software Practices: Proficiency with Git, CI/CD automated deployments, and Agile/Scrum delivery.

• Education: Bachelor's degree in Computer Science, Information Technology, Engineering, or a related field.

Preferred Qualifications

• Databricks Certified Data Engineer / Data Architect.

• Major Cloud Provider Certification (Azure / AWS / GCP).

• Experience with P&C insurance platforms, master data management (MDM), and real-time streaming architectures.

Key Skills / Keywords

Senior Data Engineer, Data Architect, Databricks, Azure Databricks, Apache Spark, PySpark, Python, SQL, Delta Lake, Delta Live Tables, Unity Catalog, Lakehouse, ETL, ELT, Data Modeling, Data Architecture, Azure, AWS, GCP, Snowflake, Kafka, Insurance Domain, P&C Insurance, Claims, Policy, Underwriting, Data Governance, Data Integration.