Principal Data Engineer

DataJobs; Houston (TX); On-site; USD 140,000 - 200,000; Full time

Job summary

Engineers and Constructors International Inc. in Houston, TX seeks a Principal Data Engineer for a 7-month, on-site contract to lead Azure-based data lake and governance initiatives.

You will architect scalable data pipelines and drive production-ready solutions using Databricks Delta Live Tables, Structured Streaming, and Unity Catalog. You will define capability-building roadmaps, guide technical execution, and ensure quality, security, and governance across high-volume data environments while

Qualifications

• Bachelor’s or Master’s degree in IT or related discipline.
• 15-17 years of IT industry experience delivering cost-effective cloud-based data analytics solutions.
• Proven experience building data lake architectures and optimizing performance in big data environments.
• Certification in Azure Data Engineering or Databricks (preferred).

Responsibilities

• Architect, design, and implement scalable Azure-based data lake solutions.
• Establish roadmaps for capability-building across Databricks Delta Live Tables, Unity Catalog, and related technologies.
• Provide technical leadership to ensure adherence to best practices and quality standards.
• Lead development of data transformation pipelines using Delta Live Tables and Structured Streaming.
• Oversee use of Databricks Unity Catalog for data governance, lineage, and metadata management.
• Develop and enforce robust data governance policies to support compliance, security, and data quality.

Skills

Azure

Databricks

Delta Live Tables

Structured Streaming

Unity Catalog

SQL

Python

Scala

Spark-SQL

Data governance

Education

Bachelor’s or Master’s degree in IT or related discipline

Tools

Qlik Replicate

Databricks

Unity Catalog

ADLS

ADF

Synapse Analytics

PowerBI

Hadoop

Job description

Engineers and Constructors International Inc. is hiring a Principal Data Engineer for a contract assignment in Houston, TX (onsite) with an anticipated 7-month duration. In this role, you will lead Azure-based data lake and data governance capabilities, building transformation and streaming pipelines for high-volume data environments.

Role overview

This Principal Data Engineer contract position focuses on designing and delivering an Azure data platform that leverages Databricks Delta Live Tables, Structured Streaming, and Unity Catalog. You will also help define capability-building roadmaps across additional data technologies and guide technical execution to ensure quality, governance, and production stability.

Responsibilities

• Architect, design, and implement scalable Azure-based data lake solutions.
• Establish roadmaps for capability-building across Qlik Replicate, Databricks Delta Live Tables, and Unity Catalog.
• Provide technical leadership to ensure adherence to best practices and quality standards.
• Lead development of data transformation pipelines using Databricks Delta Live Tables and Structured Streaming.
• Optimize Databricks resources for high-volume processing while staying within budget.
• Oversee use of Databricks Unity Catalog for data governance, lineage, and metadata management.
• Develop and enforce robust data governance policies to support compliance, security, and data quality.
• Monitor data pipeline performance and proactively resolve production issues.
• Collaborate with cross-functional teams to align project objectives with business goals.
• Translate technical challenges into actionable insights for executive teams.
• Mentor and develop a high-performing team skilled in Azure, Databricks, and data streaming technologies.
• Facilitate knowledge transfer sessions to help internal teams adopt cutting-edge methodologies.

Requirements

• Bachelor’s or Master’s degree in IT or a related discipline.
• 15-17 years of experience in the IT industry, with expertise in delivering cost-effective cloud-based data analytics solutions.
• Proven experience building data lake architectures and optimizing performance in big data environments.
• Certification in Azure Data Engineering or Databricks (preferred).
• Strong knowledge of Azure technologies including ADF, ADLS, Databricks, Stream Analytics, Delta Live Tables, and Synapse Analytics.
• Advanced proficiency in SQL and programming languages such as Python, Scala, or Spark-SQL.
• Experience designing and developing scalable ETL pipelines and optimizing data workflows.
• Deep understanding of data migration: on-premises to cloud and cloud to cloud.
• Proven ability to lead diverse teams and manage end-to-end analytics projects.
• Exceptional communication and collaboration skills for executives and stakeholders.
• Strong ownership and accountability, with adaptability and a commitment to results.
• Expertise in identifying risks and implementing effective mitigation strategies.

Technologies

Azure, Qlik Replicate, Databricks Delta Live Tables, Unity Catalog, Databricks, Structured Streaming, SQL, Python, Scala, Spark-SQL, ADF, ADLS, Stream Analytics, Synapse Analytics, Hadoop, Kafka, PowerBI, QlikSense, AI/ML, NLP, GenAI, AGILE, KANBAN, MuleSoft, Shell IDT/Downstream & Renewables IDT/Trading & Supply/TS - Crude & Products/TS - Products/TS - Products Investments, VIP (Voyage Management System), IMOS, ETRMs, MuleSoft

Additional notes

• Eligibility: You must be a US Citizen or Permanent US Resident to qualify.

Project context

The work includes building a new Freight Platform to replace RADAR and serve freight business needs and customers. The platform is expected to act as a hub for seamless integration of data to support an ecosystem of IT tools aligned with Shell’s business growth roadmap. The project will start a detailed design phase through Stage Gate 2, including a solution architecture that provides an industry standard Voyage Management System (VIP) alongside other IT components to close capability gaps identified in the study phase. You will also help optimize business processes, define master data maintenance processes, determine how the platform supports freight exposure management (including potential use of existing ETRMs), and identify relevant IMOS reporting capabilities to avoid unnecessary functionality and implement cost-efficient alternatives where applicable. Additional responsibilities within the project include identifying MI data use cases for competitive value (for example, QlikSense-based alternate solutions for exposure reporting, VaR, and Carbon Emissions reporting), defining a middleware integration approach to replace MuleSoft for connecting Shell systems to VESON VIP, and developing a data platform solution to ingest, host, and distribute data between multiple integration points.

Business applications (covered in scope)

• RADAR Bulk Upload, RADAR Mail, RADAR-SAFE, RadarMsgServer, RADAR_app
• VESON VIP T&S Pr| Data Modelling
• Data Engineering, Data Integration

Portfolio references

• Shell IDT/Downstream & Renewables IDT/Trading & Supply/TS - Crude & Products/TS - Products/TS - Products Investments

AI-assisted review notice

ECI Group uses AI-assisted tools in the early stages of application review, with all decisions guided by human oversight and fully compliant with General Data Protection Regulation (GDPR) and ECI Group’s applicable data protection policies and Privacy Notice.