Senior Data Lake Architect & Platform Engineer

Pilot Company; Knoxville (TN); On-site; USD 120,000 - 170,000; Full time

Job summary

Pilot Company is seeking a seasoned Data Engineer to design, build, and optimize data pipelines and analytics workflows supporting big data infrastructure for retail and energy sectors. You will collaborate with Data Analytics, Data Scientists, and business stakeholders to meet data needs and improve data velocity and quality.

The role requires strong SQL, Spark, Python, and cloud platform experience, with a focus on ETL/ELT processes and data lake/lakehouse architectures.

Qualifications

• Bachelor’s degree in computer science, engineering, information technology or related field is required.
• Minimum eight years of technology operations experience.
• Strong SQL knowledge and skills required.
• Experience with Apache Spark or Spark-streaming and Python is required.
• Experience with EDW data models and Star Schema modeling is required.
• Experience with Databricks or Snowflake is preferred.
• Cloud data platforms (AWS/Azure/GC) knowledge preferred.

Responsibilities

• Research, develop, document, and modify data engineering processes and jobs per data architecture requirements.
• Collaborate with data strategists and data scientists; understand data velocity and access patterns.
• Provide technical expertise to implement data and analytics specifications.
• Join cross-functional project teams to deliver key data deliverables.
• Troubleshoot complex issues within SLAs; tune code and maintain standards.
• Review documentation for new development and modifications of data lake processes.
• Perform code reviews and oversee testing to ensure data integrity and security.
• Educate stakeholders on Data Lake/Lakehouse technologies; mentor team members.

Skills

SQL

Analytical skills

Communication

Mentoring

Self-motivated

Problem solving

Education

Bachelor’s degree in computer science/engineering/IT or related field

Tools

Spark

Python

Databricks

Snowflake

AWS

Azure

Google Cloud

Oracle

PostgreSQL

SQL Server

Message Queues

Job description

Pilot Company is seeking a seasoned Data Engineer to design, build, and optimize data pipelines and analytics workflows supporting big data infrastructure for retail and energy sectors. You will collaborate with Data Analytics, Data Scientists, and business stakeholders to meet data needs and improve data velocity and quality.

The role requires strong SQL, Spark, Python, and cloud platform experience, with a focus on ETL/ELT processes and data lake/lakehouse architectures.