Principal Data Engineer

Avidxchange, Inc.; Dallas (TX); Remote; USD 150,000 - 210,000; Full time

Benefits offered by this job

PTO 18 days

Holidays

Volunteer Time Off

Life Insurance

Long-Term Disability

Short-Term Disability

401(k) Match

Parental Leave

Hybrid Workplace

Job summary

AvidXchange, based in Charlotte, NC, seeks a Principal Data Engineer to lead modern data platforms and cloud-native architectures on Databricks. You will drive migration from Azure SQL Server, design real-time pipelines with Kafka, and enable AI capabilities such as Databricks Genie across enterprise-scale data products.

Collaborating with software, product, architecture, and analytics teams, you will govern data models, ensure reliability, and mentor engineers while shaping long-term data

Qualifications

• Bachelor's degree in Computer Science, Engineering, or related field.
• Hands‑on Databricks expertise: Delta Lake, Unity Catalog, Databricks Workflows, Databricks SQL, and cluster/job optimization.
• Proven experience migrating from legacy Azure SQL Server to Databricks — schema translation, data validation, and cutover strategies.
• Strong proficiency in Apache Kafka for streaming pipelines — producers/consumers, topic design, partitioning strategy, and Kafka Connect.
• Expert‑level PySpark and/or Scala Spark skills, including performance tuning, broadcasting, partitioning, and caching.
• Deep understanding of data architecture patterns: medallion (bronze/silver/gold), Lambda/Kappa, event sourcing, and streaming‑first designs.
• Strong knowledge of infrastructure components — networking, cloud storage (Delta/Parquet), and cloud cost optimization.

Responsibilities

• Lead scalable cloud-native data architectures on Databricks.
• Migrate legacy Azure SQL Server to Databricks, including schema translation and cutover planning.
• Define data modeling standards (medallion, star/snowflake) across pipelines.
• Evaluate tools and platforms to support long-term data strategy.
• Architect and implement Kafka-based streaming pipelines for real-time data ingestion.
• Design event-driven architectures using Kafka Streams, ksqlDB, or Spark Structured Streaming on Databricks.
• Establish patterns for schema management and offset management.
• Ensure streaming SLAs for latency and throughput.
• Debug and optimize Spark jobs and Delta Lake tables for performance and cost.
• Lead code reviews and CI/CD delivery workflows; promote data governance.
• Mentor engineers and establish data engineering standards.
• Stay current on platform trends and contribute to architectural vision.
• Develop plans for data security, disaster recovery, and archiving.
• Design and enable AI capabilities on Databricks, including Databricks Genie.
• Collaborate with ML/AI teams on feature stores and RAG pipelines.
• Define governance and observability standards for AI data pipelines.

Skills

Databricks

Kafka

PySpark

Scala Spark

Delta Lake

Unity Catalog

Databricks Genie

Data architecture

Azure SQL migration

Kafka Streams

ksqlDB

Education

Bachelor's degree in Computer Science, Engineering, or related field

Tools

Confluent Schema Registry

Kafka Connect

dbt

Azure DevOps

GitHub Actions

Job description

Charlotte, North Carolina, United States; Virtual

Overview

The Principal Data Engineer is a senior technical leader within AvidXchange's Data Engineering organization responsible for architecting, building, and scaling modern data platforms. In this role, you will drive the migration of legacy Azure SQL Server workloads to Databricks, design real-time streaming pipelines with Apache Kafka, enable AI and agentic capabilities such as Databricks Genie, and define the long-term data architecture strategy. You will partner closely with Software Engineering, Product, Architecture, DevOps, and Analytics teams to deliver secure, high-performing, and reliable data solutions at enterprise scale.

What You’ll Do

• Lead the design and implementation of scalable, cloud-native data architectures on Databricks (Delta Lake, Unity Catalog, Lakehouse patterns).
• Own and execute the migration strategy from legacy Azure SQL Server to Databricks, including schema translation, ETL/ELT re-platforming, data validation, and cutover planning.
• Define data modeling standards (medallion architecture, star/snowflake schemas) and ensure consistency across all pipelines and domains.
• Evaluate and recommend tools, frameworks, and platforms to support long-term data strategy and organizational goals.
• Collaborate with Solution and Enterprise Architects to review and approve new data architecture designs.
• Architect and implement Kafka-based streaming pipelines for real-time data ingestion, transformation, and delivery.
• Design event-driven architectures and streaming topologies using Kafka Streams, ksqlDB, or Spark Structured Streaming on Databricks.
• Establish patterns for schema management (Confluent Schema Registry), consumer group strategy, offset management, and dead-letter queuing.
• Ensure streaming pipelines meet SLA requirements for latency, throughput, and fault tolerance.

Optimization, Quality & Standards

• Debug and optimize Spark jobs, Delta Lake tables, and SQL workloads for performance, cost efficiency, and maintainability.
• Lead code reviews focused on senior engineers to enforce standards, best practices, and technical quality.
• Manage pipeline quality, data models, and CI/CD delivery workflows; guide teams on continuous improvement.
• Promote strong data management practices — data quality, lineage, observability, and governance.
• Identify opportunities to improve service delivery methods, processes, and resource utilization.

Leadership, Mentorship & Strategy

• Mentor data engineers at all levels, with particular emphasis on developing senior talent.
• Establish and evolve data engineering standards, best practices, and management of technical debt.
• Stay current on data platform trends (Databricks releases, Kafka ecosystem, open table formats) and contribute to long-term architectural vision.
• Develop plans for data security, disaster recovery, backup, business continuity, and archiving across the Lakehouse.
• Design and enable AI and agentic capabilities on the Databricks platform, including Databricks Genie for natural language data exploration and self-service analytics.
• Architect data foundations — clean, governed, well-documented Delta tables — that power Genie spaces, AI/BI dashboards, and LLM-driven data agents.
• Collaborate with ML and AI teams to build and maintain feature stores, vector stores, and retrieval-augmented generation (RAG) pipelines on Databricks.
• Evaluate and integrate emerging agentic frameworks (LangChain, Mosaic AI Agent Framework) to automate data workflows and enable intelligent data products.
• Define governance and observability standards for AI-driven data pipelines, ensuring reliability, auditability, and responsible AI practices.

Cross-Functional Collaboration

• Partner with project managers and business leaders on initiatives involving enterprise data.
• Collaborate across teams to influence and strengthen data engineering practices organization-wide.
• Work with Analytics, ML, and product engineers to design and deliver end-to-end data solutions that meet business needs.

What We’re Looking For

Required

• Bachelor's degree in Computer Science, Engineering, or a related field with 10+ years of data engineering experience in a high-availability, business-critical environment.
• Hands‑on Databricks expertise: Delta Lake, Unity Catalog, Databricks Workflows, Databricks SQL, and cluster/job optimization.
• Proven experience migrating from legacy Azure SQL Server (or other relational RDBMS) to a Databricks Lakehouse — schema translation, data validation, and cutover strategies.
• Strong proficiency in Apache Kafka for streaming pipelines — producers/consumers, topic design, partitioning strategy, and Kafka Connect.
• Expert‑level PySpark and/or Scala Spark skills, including performance tuning, broadcasting, partitioning, and caching.
• Deep understanding of data architecture patterns: medallion (bronze/silver/gold), Lambda/Kappa, event sourcing, and streaming‑first designs.
• Strong knowledge of infrastructure components — networking, cloud storage (Delta/Parquet), and cloud cost optimization.

Preferred

• Databricks Certified Data Engineer Associate or Professional certification.
• Experience with ksqlDB, Kafka Streams, or Spark Structured Streaming for stateful stream processing.
• Confluent Platform experience: Schema Registry, Kafka Connect connectors, RBAC, and cluster management.
• Advanced Azure SQL Server expertise (2018+/Azure SQL MI) — stored procedures, indexing strategies, query plan analysis — valuable for migration contexts.
• Experience with dbt (data build tool) for SQL-based transformation layers on Databricks.
• Proficiency with Git and CI/CD pipelines for data engineering (Azure DevOps, GitHub Actions).
• Experience working in Agile environments (Scrum/Kanban).
• Familiarity with data governance frameworks, data cataloging (Unity Catalog, Microsoft Purview), and data quality tooling (Great Expectations, Monte Carlo).
• Familiarity with secure coding practices, including OWASP Top 10 and secrets management.
• Experience implementing DataOps practices: automated testing, data observability, pipeline CI/CD, data contracts, and SLA monitoring across the Lakehouse.
• Hands‑on MLOps experience: model versioning (MLflow), experiment tracking, model serving, and integrating ML pipelines with production data workflows on Databricks.
• Familiarity with Databricks Mosaic AI (formerly MLflow + Model Serving) for end-to‑end MLOps lifecycle management.
• Experience with real-time ML feature stores or Lakehouse-based ML pipelines is a plus.

About AvidXchange

AvidXchange is a leading provider of accounts payable (“AP”) automation software and payment solutions for middle‑market businesses and their suppliers. By trade, we are a technology company, but if you ask anyone who works here, they’ll tell you our people are at the core of who we are. At AvidXchange,mindset is everything. We are Connected as People, Growth Minded, and Customer Obsessed. Thesethree mindsets represent our culture – who weare, who we’ve always been, and they guide usto improve every day.Since our founding in 2000 in Charlotte, NC, we’ve created a company of over 1,500 teammates working across the U.S., or remotely. AvidXchange is proud to be Certified™ as a Great Place to Work ®. The prestigious recognition is based on anonymous data from our teammates and makes official what our teammates have known for years – that AvidXchange is a Great Place to Work®.

Who you are:

• A go-getter with an entrepreneurial mindset – that meansyou arenot afraid of taking risks,winning bigorfacing the unknown.
• Someone who understands that business ispeople centric. Connecting with others as humans first allows you to develop mutually beneficial working relationships.
• Focused onmaking a difference for our customers. AvidXchange exists to help solve complex problems for our customers so we can all realize our potential.

What you’ll get:

• 18 days PTO*
• 11 Holidays (8companyrecognized & 3floatingholidays)
• 16 hours per year ofpaid Volunteer Time Off (VTO)
• High Deductible Heath Plan Option that has $0 monthly premium for teammate-only coverage
• 100% AvidXchange paid Life Insurance
• 100% AvidXchange paid Long-Term Disability
• 100% AvidXchange paid Short-Term Disability
• Employee Assistance Program (EAP) - Providescounseling services, legal and financial consultations and health advocacy for Teammates and their eligible dependents
• Onsite Health Clinic with Atrium Health - available to Teammates and their eligible dependents
• 401(k) Match: 100% match on the first 3% of your salary, plus 50% match on the next 2%
• Parental Leave: 8 weeks 100% paid by AvidXchange**
• Discounts on Pet, Home, and Auto insurance
• Perks at Work:free discount program that provides teammates the opportunity to save on items fromelectronics, movie tickets, car buying, vacations,andmore
• Onsite gym fitness center, yoga studio, and basketball court
• Tuition Reimbursement up to the federal maximum of $5,250***
• Hybrid Workplace Flexibility
• Free parking

*Fully granted from beginning of year, pro-rated if hired mid-year

*Must be full-time for at least 3 months

*Must be full-time for at least one year

AvidXchange is an equal opportunity employer. AvidXchange is committed to equal employment opportunity in accordance with applicable federal, state, and local laws. AvidXchange will not discriminate against applicants for employment on any legally recognized basis. This includes, but is not limited to veteran status, race, color, religion, sex, sexual orientation, gender identity, gender expression, national origin, age and physical or mental disability.