Senior Data Engineer – Databricks & AI Data Platform

Brillio; United States; On-site; USD 140,000 - 180,000; Full time

Job summary

Brillio is seeking a Senior Data Engineer to design, build, and scale enterprise-grade data pipelines and AI-ready data foundations. You will work at the intersection of Databricks, Azure AI, and knowledge management to deliver trusted data products for analytics and AI initiatives.

The role requires strong Python, PySpark, and SQL skills, plus experience with Delta Lake and Unity Catalog within cloud data platforms.

Qualifications

• 7+ years in data engineering or big data environments.
• Hands-on with Databricks including Delta Lake and Unity Catalog.
• Experience in building large-scale ETL/ELT pipelines.
• Proficient in Python, PySpark, and SQL.
• Experience with Azure Data Services and cloud data platforms.

Responsibilities

• Design, develop, and maintain scalable data pipelines on Databricks.
• Create AI-ready data foundations including ontology and knowledge graphs.
• Support AI/ML and Generative AI use cases with data prep and pipelines.
• Implement data governance, lineage, metadata, and security controls.
• Optimize performance, cost, and reusable engineering templates.

Skills

Databricks

Python

SQL

Cloud platforms

ETL/ELT

Data Modeling

Governance

Education

Bachelor's or Master's in CS or related

Tools

Delta Lake

Unity Catalog

Spark

Workflows

Databricks

Job description

Brillio is a leading digital technology consulting and services company that helps enterprises accelerate their digital transformation journey through innovative engineering, cloud, data, AI and customer experience solutions through a team of 5,000+ experts technologically elevating 200 companies globally. Brillio partners with global enterprises to build modern, scalable, and business-driven technology platforms that deliver measurable outcomes and competitive advantage. With a strong culture centered around Customer Success, We Care, Entrepreneurial Mindset, and Excellence, Brillio empowers organizations to drive innovation, improve operational efficiency, and create exceptional customer experiences. Through its highly skilled technology workforce and deep domain expertise, Brillio delivers impactful solutions across Product Engineering, Data & AI, Digital Experience, Cloud, and Infrastructure services.

About the Role

We are looking for a highly skilled Senior Data Engineer to help build and scale our next-generation data and AI platform. In this role, you will design, develop, and optimize enterprise-grade data pipelines that power advanced analytics, semantic knowledge layers, knowledge graphs, and AI-driven solutions. Working at the intersection of Data Engineering, Databricks, Azure AI, and Knowledge Management, you will play a critical role in transforming diverse datasets into trusted, AI-ready data products that accelerate business innovation.

You will collaborate with data architects, AI engineers, data scientists, business stakeholders, and governance teams to establish modern data foundations that enable scalable and compliant AI adoption across the organization.

Key Responsibilities

Build and Scale Modern Data Pipelines

• Design, develop, and maintain highly scalable data pipelines on Databricks leveraging Delta Lake, Unity Catalog, and Databricks Workflows.
• Ingest, integrate, and process large-scale datasets from research, commercial, content, operational, and third-party sources.
• Implement batch and real-time data processing solutions to support enterprise analytics and AI initiatives.
• Ensure reliable, secure, and high-performance data movement across cloud platforms and business systems.

Develop AI-Ready Data Foundations

• Build the core data infrastructure supporting enterprise ontology, semantic layer, and knowledge graph initiatives.
• Design and implement robust data models that enable efficient data discovery, contextual understanding, and relationship mapping.
• Develop entity resolution frameworks to improve data consistency and create trusted views of business-critical information.
• Create curated, reusable, and AI-ready data products that can be leveraged across multiple business domains.

Enable AI and Generative AI Use Cases

• Prepare and deliver high-quality datasets for Azure AI Foundry and advanced AI/ML applications.
• Build and optimize chunking, embedding, vectorization, and retrieval pipelines for Generative AI and RAG-based solutions.
• Develop feature engineering pipelines that support machine learning and predictive analytics initiatives.
• Enable seamless data access for AI applications through real-time and batch processing frameworks.

Ensure Data Governance and Compliance

• Implement enterprise data governance standards using Unity Catalog.
• Establish robust data quality controls, lineage tracking, metadata management, and auditing capabilities.
• Define and maintain role-based access controls and security policies to ensure data privacy and regulatory compliance.
• Collaborate with governance and compliance teams to support enterprise data management standards.

Drive Performance, Optimization, and Reusability

• Monitor and optimize platform performance, scalability, and operational efficiency.
• Implement cost optimization strategies across data storage, processing, and compute layers.
• Develop reusable frameworks, templates, and engineering standards that accelerate onboarding of new data and AI use cases.
• Champion engineering best practices including automation, CI/CD, testing, monitoring, and observability.

Required Qualifications

• Bachelor's or Master's degree in Computer Science, Data Engineering, Information Systems, or a related field.
• 7+ years of experience in Data Engineering, Data Platform Development, or Big Data environments.
• Hands-on expertise with Databricks, including Delta Lake, Unity Catalog, Workflows, Spark, and Lakehouse architecture.
• Strong experience building large-scale ETL/ELT pipelines and modern cloud data platforms.
• Experience working with Azure Data Services and enterprise cloud environments.
• Strong proficiency in Python, PySpark, and SQL.
• Deep understanding of data modeling, metadata management, master data management, and data governance principles.
• Experience designing and implementing enterprise-scale data architectures.

Why should you apply for this role?

As Brillio continues to gain momentum as a trusted partner for our clients in their digital transformation journey, we strive to set new benchmarks for speed and value creation. The DI team at Brillio is at the forefront of leading this charge by reimagining and executing how we structure, sell and deliver our services to better serve our clients.

Know what it’s like to work and grow at Brillio: https://www.brillio.com/join-us/

Equal Employment Opportunity Declaration

Brillio is an equal opportunity employer to all, regardless of age, ancestry, colour, disability (mental and physical), exercising the right to family care and medical leave, gender, gender expression, gender identity, genetic information, marital status, medical condition, military or veteran status, national origin, political affiliation, race, religious creed, sex (includes pregnancy, childbirth, breastfeeding, and related medical conditions), and sexual orientation.