Director of Technology, Enterprise Data Engineering

Jobgether; United States; Remote; USD 120,000 - 186,000; Full time

Benefits offered by this job

Remote work within United States

Career development

Leadership opportunities

Job summary

Jobgether is seeking a senior data engineering leader to drive modernization and oversee a distributed data platform. You will mentor teams, establish engineering standards, and partner with enterprise architects to deliver scalable data pipelines.

The role emphasizes Databricks proficiency, governance, and AI-enabled development, with remote work within the United States.

Qualifications

• At least 10 years in data engineering, with 3+ years in leadership.
• Bachelor's degree in Computer Science, Information Systems, or related field, or equivalent experience.
• Proven experience migrating legacy data platforms to cloud environments.
• Strong SQL and PySpark skills with large governed datasets.
• Experience with cloud data platforms and data governance practices.

Responsibilities

• Lead, coach, and develop a data engineering team across platforms.
• Orchestrate Databricks migration and platform modernization initiatives.
• Define data architecture, modeling, and production-ready pipelines.
• Ensure data quality, security, and regulatory compliance.
• Collaborate with enterprise architects and analytics teams to enable self-service analytics.
• Implement AI-assisted engineering workflows and governance standards.
• Manage vendor relationships and report progress to senior stakeholders.

Skills

Data leadership

Databricks

SQL

PySpark

Data modeling

Data architecture

CI/CD for data

AI-assisted engineering

Stakeholder mgmt

Security & compliance

Education

Bachelor's degree in CS/IS

Tools

Databricks

Lakebase

Agent Bricks

MLflow

PostgreSQL

Oracle

Job description

Accountabilities

- Data Engineering Leadership: Lead, coach, and develop data engineering professionals, managing performance, staffing, workload distribution, and career development. Establish consistent standards for employees, contractors, vendors, and engineers working within cross-functional teams.

- Platform Modernization: Lead the migration of legacy data lakes and curated data stores to Databricks, coordinating domain-by-domain delivery and resolving differences across development, testing, and production environments.

- Data Architecture and Modeling: Partner with Enterprise Architecture to translate strategic direction into production-ready data pipelines, data models, platform designs, and consistent engineering practices.

- Pipeline Development and Reliability: Own data ingestion and medallion architecture standards across bronze, silver, and gold layers. Maintain the availability, accuracy, performance, and scalability of existing pipelines while delivering new data integrations and supporting evolving business products.

- Semantic Data and Analytics Enablement: Develop and maintain the semantic data layer, Unity Catalog objects, and curated datasets in collaboration with Enterprise Data and Analytics Operations. Ensure consistent business definitions and reliable data for analytics, governance, and AI-assisted querying.

- Self-Service Engineering Enablement: Establish reusable pipeline templates, scaffolding, reference architectures, and automated tooling that allow autonomous teams to develop and operate compliant data pipelines with minimal centralized support.

- AI-Driven Engineering: Integrate AI tools and agentic development practices into daily workflows, including code generation, testing, documentation, and automated reviews. Define standards for AI-assisted development and measure its impact on productivity, quality, cost, and delivery speed.

- Governance, Security, and Risk Management: Ensure engineering deliverables meet data quality, lineage, documentation, privacy, and regulatory requirements. Collaborate with Information Security to support audits, remediate vulnerabilities, manage sensitive data appropriately, and implement automated compliance guardrails.

- Vendor and Stakeholder Management: Manage relationships with system integration partners and platform vendors, taking ownership of architectural decisions and technical deliverables. Communicate progress, risks, dependencies, and recommendations clearly to senior leaders and business stakeholders.

- Performance and Continuous Improvement: Define and monitor service-level agreements, operational KPIs, platform consumption costs, and data quality metrics. Lead complex troubleshooting efforts, maintain technical documentation and runbooks, and continuously evaluate emerging technologies such as Lakebase, Agent Bricks, and MLflow.

Requirements

- Professional Experience: At least 10 years of experience in data engineering or a related specialized technology field, including at least three years of team leadership or management experience.

- Education: A bachelor's degree in Computer Science, Information Systems, or a related discipline, or equivalent professional experience.

- Platform Modernization: Demonstrated experience leading legacy-to-cloud data platform migrations, managing enterprise data integrations, and working with both vendor-provided SaaS solutions and internally developed systems.

- Databricks Expertise: Strong hands-on knowledge of Databricks, including Unity Catalog, Delta Lake, medallion architecture, and Databricks Asset Bundles or YAML-based configurations. Experience with Lakebase is an advantage.

- Data Engineering and Modeling: Advanced SQL and PySpark skills, experience working with large governed datasets, and a strong understanding of dimensional modeling, semantic layers, domain-driven design, and data lifecycle management.

- Data Quality and Analytics: Experience building certified datasets and semantic models for self-service analytics or AI-assisted natural-language querying. Familiarity with data quality frameworks, automated testing, reconciliation, and historical data management is essential.

- Engineering Standards and Automation: Working knowledge of CI/CD practices for data pipelines, version control, governed deployment environments, and automated testing. Experience developing internal platforms or self-service infrastructure for distributed engineering teams is highly desirable.

- AI and Automation Skills: Practical experience using tools such as Claude, Copilot, ChatGPT, or Gemini to improve engineering productivity and decision-making. Understanding of AI output validation, responsible AI principles, and AI governance is required, along with the ability to coach teams on effective adoption.

- Agentic Development Practices: Experience designing or governing AI-assisted software development workflows, including specification-driven development, autonomous coding agents, automated code review, and testing pipelines.

- Security and Compliance: Experience collaborating with Information Security and applying data privacy, access control, and regulatory requirements. Familiarity with insurance regulations, including NAIC guidelines and state insurance requirements, is beneficial.

- Project and Vendor Management: Experience planning and prioritizing technical initiatives, managing resources and budgets, coordinating external vendors, and taking ownership of technical decisions and delivery outcomes.

- Communication and Leadership: Strong analytical, organizational, and problem-solving skills, with the ability to translate complex technical concepts into business-friendly language. Demonstrated ability to influence senior stakeholders, hold teams accountable, challenge existing approaches constructively, and raise risks early.

- Preferred Technical Experience: Familiarity with AWS-based data platforms, IBM DataStage, PostgreSQL, Oracle source systems, Lakebase, Agent Bricks, or MLflow is an advantage.

- Additional Qualifications: Experience in insurance or another regulated industry, formal mentoring and coaching, and knowledge of agile methodologies, project management tools, Atlassian products, Lucidchart, and Microsoft Office are beneficial.

- Work Environment: Candidates must currently reside in the United States and be authorized to work without sponsorship. The position is fully remote and requires a suitable private, quiet workspace. Occasional travel may be necessary for meetings or other business activities.

Benefits

- Competitive Salary: Annual salary range of $120,000–$186,000 , depending on qualifications, experience, and other relevant factors.

- Comprehensive Benefits Package: Access to a range of employee benefits. Specific coverage and plan details are available through the employer's benefits information.

- Remote Flexibility: Work remotely from within the United States.

- Career Development: Lead a major enterprise data modernization initiative while expanding your expertise in cloud data platforms, AI-assisted engineering, and enterprise architecture.

- Leadership Opportunities: Build and develop a data engineering team, shape technical strategy, and influence engineering standards across distributed teams.

- Innovation and Impact: Work with modern data technologies and emerging AI capabilities to improve platform performance, scalability, governance, and delivery efficiency.

- Collaborative Environment: Partner with senior technology leaders, enterprise architects, analytics specialists, security teams, and external technology providers.

- Meaningful Business Contribution: Help strengthen data capabilities that support reliable operations and better decision-making in a regulated industry.