Azure Data Lake Architect & Databricks Leader

engineersandconstructorsinternationalinc; Houston (TX); On-site; USD 150,000 - 195,000; Full time

Job summary

ECI Group seeks a Principal Data Engineer to lead Azure-based data lake solutions and governance in a high-volume data environment. This role drives strategy, architecture, and cross-functional collaboration with key stakeholders in Houston, TX.

You will mentor teams, oversee Databricks Unity Catalog, and optimize pipelines to deliver scalable analytics platforms aligned with Shell Freight Platform aspirations and business goals.

Qualifications

• Bachelor’s or Master’s degree in IT or related field.
• 15-17 years IT experience delivering cost-effective cloud-based data analytics solutions.
• Proven track record in building data lake architectures and optimizing performance in big data environments.
• Azure Data Engineering or Databricks certification preferred.

Responsibilities

• Architect, design, and implement scalable Azure-based data lake solutions.
• Establish roadmaps for capability-building across technologies like Qlik Replicate and Unity Catalog.
• Provide technical leadership and ensure best practices and quality standards.
• Lead data transformation pipelines using Databricks Delta Live Tables and Structured Streaming.
• Optimize Databricks resources for high-volume processing within budget.
• Oversee Databricks Unity Catalog for governance, lineage, and metadata management.
• Develop and enforce robust data governance policies for compliance and data quality.
• Monitor data pipeline performance and resolve production issues.
• Collaborate with cross-functional teams to align objectives with business goals.
• Translate technical challenges into actionable insights for executives.
• Mentor and develop a high-performing Azure, Databricks, and data streaming team.

Skills

Leadership

Data governance

Cross-functional collaboration

Strategic thinking

Education

Bachelor’s or Master’s degree in IT or related discipline

Azure Data Engineering or Databricks certification (preferred)

Tools

Azure

Databricks

Qlik Replicate

Unity Catalog

PowerBI

QlikSense

Synapse

ADLS

ADF

Job description

ECI Group seeks a Principal Data Engineer to lead Azure-based data lake solutions and governance in a high-volume data environment. This role drives strategy, architecture, and cross-functional collaboration with key stakeholders in Houston, TX.

You will mentor teams, oversee Databricks Unity Catalog, and optimize pipelines to deliver scalable analytics platforms aligned with Shell Freight Platform aspirations and business goals.