Principal Data Engineer

engineersandconstructorsinternationalinc; Houston (TX); On-site; USD 150,000 - 195,000; Full time

Job summary

ECI Group seeks a Principal Data Engineer to lead Azure-based data lake solutions and governance in a high-volume data environment. This role drives strategy, architecture, and cross-functional collaboration with key stakeholders in Houston, TX.

You will mentor teams, oversee Databricks Unity Catalog, and optimize pipelines to deliver scalable analytics platforms aligned with Shell Freight Platform aspirations and business goals.

Qualifications

• Bachelor’s or Master’s degree in IT or related field.
• 15-17 years IT experience delivering cost-effective cloud-based data analytics solutions.
• Proven track record in building data lake architectures and optimizing performance in big data environments.
• Azure Data Engineering or Databricks certification preferred.

Responsibilities

• Architect, design, and implement scalable Azure-based data lake solutions.
• Establish roadmaps for capability-building across technologies like Qlik Replicate and Unity Catalog.
• Provide technical leadership and ensure best practices and quality standards.
• Lead data transformation pipelines using Databricks Delta Live Tables and Structured Streaming.
• Optimize Databricks resources for high-volume processing within budget.
• Oversee Databricks Unity Catalog for governance, lineage, and metadata management.
• Develop and enforce robust data governance policies for compliance and data quality.
• Monitor data pipeline performance and resolve production issues.
• Collaborate with cross-functional teams to align objectives with business goals.
• Translate technical challenges into actionable insights for executives.
• Mentor and develop a high-performing Azure, Databricks, and data streaming team.

Skills

Leadership

Data governance

Cross-functional collaboration

Strategic thinking

Education

Bachelor’s or Master’s degree in IT or related discipline

Azure Data Engineering or Databricks certification (preferred)

Tools

Azure

Databricks

Qlik Replicate

Unity Catalog

PowerBI

QlikSense

Synapse

ADLS

ADF

Job description

Join the ECI Group's Professional Technical Services Network as a contract Principal Data Engineer. This is a Houston TX based contract role with an anticipated 7-month duration with an estimated start date of October 5, 2026. Note: You must be a US Citizen or Permanent US Resident to qualify for this position.

About the Role

We are seeking a visionary Principal Data Engineer to lead the delivery of innovative data analytics solutions, with a focus on technical excellence, governance, and operational performance in a high-volume data environment. This role involves strategic and technical leadership in building Azure-based data lake solutions, driving maturity in data management, and engaging with cross-functional teams to leverage data for organizational success.

Key Responsibilities

• Architect, design, and implement scalable Azure-based data lake solutions.
• Establish roadmaps for capability-building across technologies such as Qlik Replicate, Databricks Delta Live Tables, and Unity Catalog.
• Provide technical leadership, ensuring adherence to best practices and quality standards.
• Lead the development of data transformation pipelines using Databricks Delta Live Tables and Structured Streaming.
• Optimize Databricks resources for high-volume data processing without exceeding budget.
• Oversee the use of Databricks Unity Catalog for data governance, lineage, and metadata management.
• Develop and enforce robust data governance policies, ensuring compliance, security, and data quality.
• Monitor data pipeline performance and proactively resolve production issues.
• Collaborate with cross-functional teams to align project objectives with business goals.
• Translate technical challenges into actionable insights for executive teams.
• Mentor and develop a high-performing team skilled in Azure, Databricks, and data streaming technologies.
• Facilitate knowledge transfer sessions to empower internal teams with cutting‑edge methodologies.

Education and Experience

• Bachelor’s or Master’s degree in IT or a related discipline.
• 15-17 years of experience in the IT industry with expertise in delivering cost-effective cloud-based data analytics solutions.
• Proven track record in building data lake architectures and optimizing performance in big data environments.
• Certification in Azure Data Engineering or Databricks (preferred).

Technical Skills

• Strong knowledge of Azure technologies, including ADF, ADLS, Databricks, Stream Analytics, Delta Live Tables, and Synapse Analytics.
• Advanced proficiency in SQL and programming languages such as Python, Scala, or Spark‑SQL.
• Experience in designing and developing scalable ETL pipelines and optimizing data workflows.
• Deep understanding of data migration from on‑premises to cloud, cloud to cloud.

Leadership and Soft Skills

• Proven ability to lead diverse teams and manage end‑to‑end analytics projects.
• Exceptional communication and collaboration skills for engaging executives and stakeholders.
• Strong sense of ownership, accountability, adaptability, and commitment to results.
• Expertise in identifying risks and implementing effective mitigation strategies.

Nice to Have

• Familiarity with big data tools like Hadoop, Kafka, and Spark.
• Experience with visualization tools such as PowerBI or QlikSense.
• Knowledge of AI/ML, NLP, and GenAI concepts.
• Familiarity with AGILE and KANBAN methodologies.

Portfolio: Shell IDT/Downstream & Renewables IDT/Trading & Supply/TS - Crude & Products/TS - Products/TS - Products Investments

Project: T&S Future Freight Platform - Jigsaw

Build a new Freight Platform (replacing RADAR) to serve the freight business and its customers. This includes a new industry standard Voyage Management System (VIP) alongside other components to close capability gaps. Optimize business processes and define master data maintenance processes.

The overall Freight Platform will serve as a hub to facilitate seamless integration of necessary data, supporting an ecosystem of IT tools that are aligned with Shell's business growth roadmap.

Freight Platform Project will begin a detailed design phase through Stage Gate 2 to accomplish the following:

• • An approved (Solution Architecture) that provides a solution for Industry standard Voyage Management System (VIP) alongside other IT components to close capability gaps - identified during the Study Phase
  • Optimize business processes and define master data maintenance processes
  • Establish how well the platform can meet the needs for capturing Freight exposure management across the house, with the potential to use existing ETRMs for this if required,
  • What IMOS reporting capability to utilise and avoid paying for functionality in IMOS we don't need and can do internally in a cost-efficient manner,
  • Identify MI data use‑cases that provide Shell a competitive edge and develop them in-house e.g. with Qliksense offering-up alternate solutions for exposure reporting, VaR, Carbon Emissions reporting.
  • Identify a middleware integration solution to replace Mulesoft to tie Shell systems to VESON VIP
  • Develop a data platform solution to ingest, host and distribute data between multiple integration points

Business Application

RADAR Bulk Upload, RADAR Mail, RADAR-SAFE, RadarMsgServer, RADAR_app, VESON VIP T&S Pr| Data Modelling, Data Engineering, Data Integration

Please Note: The ECI Group uses AI-assisted tools in the early stages of application review, with all decisions guided by human oversight and fully compliant with General Data Protection Regulation (GDPR) and ECI Group's applicable data protection policies and Privacy Notice.