Senior Data Engineer

Hillwood Development Company LLC; Dallas (TX); On-site; USD 140,000 - 180,000; Full time

Benefits offered by this job

Medical Insurance

Dental Insurance

Vision Insurance

Flexible Spending Accounts (FSA)

Health Savings Accounts (HSA)

Vacation time

Sick time

Holidays

Paid family caregiver leave

Mental health and wellness resources

Life Insurance

Disability Insurance

401(k) with company match

Tuition assistance

Additional programs to supportAssocie

Job summary

Hillwood Development Company LLC's Corporate IT team in Dallas is seeking a Senior Data Engineer to build and evolve an enterprise Data Lakehouse on the Databricks platform. You will design scalable data pipelines across ERP systems (JD Edwards, OneStream, Yardi, Entrata, Cecilian, Quickbase, UKG, Medius), implement CDC, manage Unity Catalog, and enable AI-driven analytics.

This role requires 5+ years data engineering with Databricks, strong SQL/PySpark, and collaboration with business

Qualifications

• Bachelor’s degree in Computer Science or related field.
• Minimum five years of data engineering experience.
• At least two years on Databricks production solutions.
• Experience with ERP/enterprise systems integration.
• Strong CI/CD and Git-based tooling knowledge.

Responsibilities

• Design, develop, and maintain scalable Databricks data pipelines to integrate data from enterprise systems.
• Develop and maintain data models across Bronze, Silver, and Gold layers using medallion framework.
• Implement incremental processing and CDC to keep data current and cost-efficient.
• Administer Unity Catalog with schema management, governance, and lineage tracking.
• Design and support production-ready pipelines with automated scheduling, monitoring, and recovery.

Skills

SQL

Python

PySpark

Databricks

Spark SQL

Delta Lake

Databricks Workflows

Unity Catalog

Education

Bachelor’s degree in Computer Science, Data Engineering, Information Systems, Software Engineering, or a related field

Tools

Databricks

Unity Catalog

Delta Lake

Databricks Workflows

Azure DevOps

GitHub

Job description

Hillwood’s Corporate IT team in Dallas is hiring a Senior Data Engineer to build and evolve an enterprise Data Lakehouse on the Databricks platform.

Responsibilities

• Design, develop, and maintain scalable Databricks data pipelines to integrate data from enterprise systems, including JD Edwards, OneStream, Yardi, Entrata, Cecilian, Quickbase, UKG, Medius, and other applications
• Develop and maintain data models across Bronze, Silver, and Gold layers using Hillwood’s enterprise data architecture and medallion framework
• Implement incremental processing and change data capture (CDC) to keep data current, accurate, and cost-efficient
• Administer and maintain Unity Catalog, including schema management, governance, permissions, and lineage tracking
• Design and support production-ready pipelines with automated scheduling, monitoring, alerting, and recovery
• Optimize workflows, jobs, and compute resources for performance, scalability, and cost efficiency
• Establish engineering best practices including version control, code reviews, automated testing, and CI/CD
• Create and maintain technical documentation, standards, and operational procedures for long-term sustainability and knowledge transfer
• Develop and implement data quality frameworks (validation rules, reconciliation processes, monitoring) to ensure accuracy and reliability
• Investigate and resolve data quality issues by identifying and addressing root causes
• Partner with business stakeholders and data owners to define data standards, ownership, business definitions, and quality expectations
• Ensure certified and business-critical datasets align with the data governance framework and quality standards
• Leverage AI-assisted development tools (coding assistants and platform-native AI capabilities) to accelerate development and improve code quality
• Support standards for responsible use of AI-assisted software development
• Continuously evaluate emerging AI technologies to improve engineering processes, platform operations, and delivery
• Prepare and optimize lakehouse data assets for AI, machine learning, and natural language access use cases
• Build data foundations for advanced analytics and AI, including feature stores, vector search, retrieval architectures, and document-based data pipelines
• Work with AI engineers, developers, and stakeholders to integrate structured and unstructured data within the Databricks ecosystem
• Support AI workloads and platform capabilities, including model serving, agent frameworks, and emerging GenAI
• Identify and onboard new structured, unstructured, and third-party data sources to expand enterprise platform value
• Collaborate with finance and business unit stakeholders to understand processes, data requirements, and reporting needs
• Validate outputs and business metrics with data owners prior to dataset certification for business use
• Communicate risks, trade-offs, and technical concepts to both technical and non-technical audiences
• Partner closely with stakeholders to align data capabilities with operational objectives and business priorities
• Partner with external engineering resources, review solution quality, and assume long-term ownership of delivered pipelines and platform components
• Serve as an internal subject matter expert on architecture, operations, and ongoing evolution of Hillwood’s Enterprise Data Lakehouse

Requirements

• SQL, Python, PySpark, and modern data engineering development practices
• Strong understanding of lakehouse architectures, data modeling principles, and enterprise data management
• Ability to design scalable, reliable, maintainable solutions for reporting, analytics, and AI use cases
• Knowledge of data governance, data quality, metadata management, and data lineage
• Familiarity with source control, testing methodologies, code reviews, and CI/CD
• Strong analytical and problem-solving skills to identify root causes and implement solutions
• Ability to balance performance, scalability, maintainability, and cost
• Effective written and verbal communication skills, including explaining technical concepts to non-technical audiences
• Strong organizational skills and attention to detail; ability to manage multiple priorities in a fast-paced environment
• Ability to work independently and collaborate with business stakeholders, technical teams, and external partners
• Curiosity and adaptability for emerging technologies, AI capabilities, and evolving data engineering practices
• Customer-focused mindset with commitment to high-quality, business-driven solutions
• Bachelor’s degree in Computer Science, Data Engineering, Information Systems, Software Engineering, or a related field (required)
• Minimum five years of data engineering experience, including at least two years on Databricks production solutions using PySpark, Spark SQL, Delta Lake, and Databricks Workflows
• Experience designing, building, and maintaining enterprise data pipelines, integrations, and lakehouse architectures in a cloud environment
• Hands-on experience administering and utilizing Unity Catalog (metadata management, lineage, security, access controls)
• Experience operating production data platforms, including monitoring, alerting, incident response, troubleshooting, and performance optimization
• Experience implementing data quality controls, reconciliation, and governance standards
• Experience integrating data from ERP, financial, operational, or other enterprise systems
• Experience with Microsoft Azure and Git-based tools, including Azure DevOps and/or GitHub
• Experience leveraging AI-assisted development tools in a professional software engineering workflow
• Experience supporting AI/ML initiatives through data engineering, including feature engineering, vector search, retrieval-augmented generation (RAG), and model-serving pipelines or related technologies
• Demonstrated success partnering with business stakeholders to understand requirements, validate data, and deliver business-focused solutions

Technologies

• Databricks
• SQL, Python, PySpark, Spark SQL
• Delta Lake
• Databricks Workflows
• Unity Catalog
• Bronze, Silver, Gold, medallion framework
• Change data capture (CDC)
• CI/CD, Git
• Azure DevOps, GitHub
• Microsoft Azure
• JD Edwards, OneStream, Yardi, Entrata, Cecilian, Quickbase, UKG, Medius
• Feature stores, vector search, retrieval architectures, document-based data pipelines
• Model serving, agent frameworks, GenAI, retrieval-augmented generation (RAG)

Benefits

• Medical Insurance
• Dental Insurance
• Vision Insurance
• Flexible Spending Accounts (FSA)
• Health Savings Accounts (HSA)
• Vacation time
• Sick time
• Holidays
• Paid family caregiver leave
• Mental health and wellness resources
• Life Insurance
• Disability Insurance
• 401(k) with company match
• Tuition assistance
• Additional programs to support Associates and their families

Preferred Experience

• Databricks Data Engineer certification or equivalent Databricks credentials
• Experience supporting JD Edwards, OneStream, Yardi, Entrata, UKG, Medius, or similar enterprise platforms
• Experience working with data in real estate, construction, energy, financial services, or related industries
• Experience transitioning, stabilizing, and optimizing data solutions developed by third-party consultants or implementation partners
• Experience supporting large language model (LLM), generative AI, or agent-based solution development
• Experience designing solutions that combine structured and unstructured data for AI-enabled business applications

Location: Dallas, TX (onsite)