Sr Engineer, Data Engineering

Pilot Company; Knoxville (TN); On-site; USD 110,000 - 160,000; Full time

Job summary

Pilot Company is seeking a seasoned Data Engineer to design, build, and optimize data pipelines for our retail and energy platforms, aligning data architecture with business needs. You will work with analytics teams to implement reliable data Lakehouse solutions and ensure data quality across datasets.

The role requires 8+ years of tech operations experience, strong SQL and EDW knowledge, and experience with Spark, Databricks, Snowflake, and cloud platforms (AWS/Azure/Google).

Qualifications

• Bachelor’s degree in computer science, engineering, information technology, or related field is required.
• Minimum eight (8) years of technology operations experience required.
• Experience in the commodities and energy space preferred.
• Strong SQL knowledge and skills required.
• Relational databases such as Oracle, PostgreSQL, or SQL Server knowledge required.
• Experience with Spark, Spark Streaming, message queue technologies and Python required.
• EDW data models with Star Schema focus required.
• Databricks or Snowflake experience preferred.
• Cloud data platforms like AWS, Azure or Google Cloud preferred.
• Excellent analytical skills and ability to solve complex data problems.
• Strong communication and customer service abilities.
• Willingness to learn new technologies and mentor others.
• Ability to manage multiple projects in a fast-paced environment.
• Strong organizational skills and attention to detail.

Responsibilities

• Research, develop, document, and modify Data Engineering processes and jobs per data architecture and modeling requirements.
• Collaborate with business stakeholders to understand data needs including data velocity, veracity, and access patterns.
• Provide technical expertise to implement Data and Analytics specifications.
• Serve on cross-functional project teams and provide perspective for the Data Analytics team.
• Troubleshoot complex issues including connections, failed jobs, and server alerts within SLAs.
• Maintain and tune code according to internal Data Engineering standards.
• Review documentation for new development and modifications of Data Lake processes and jobs.
• Perform code and process reviews and oversee testing to ensure data integrity and security.
• Educate stakeholders on Data Lake/Lakehouse technologies.
• Mentor and guide less experienced team members; provide feedback on project work.
• Ensure compliance with rules, regulations, policies, and procedures.
• Complete other duties as assigned.

Skills

Analytical thinking

Customer service

Communication

Mentoring

Learning agility

Education

Bachelor’s degree in computer science, engineering, information technology, or related field

Tools

SQL

Oracle

PostgreSQL

SQL Server

Apache Spark

Spark Streaming

Python

Databricks

Snowflake

AWS

Azure

Google Cloud

Job description

Company Description

Pilot Company is an industry-leading network of travel centers with more than 30,000 team members and over 750 retail and fueling locations in 44 states and six Canadian provinces. Our energy and logistics division serves as a top supplier of fuel, employing one of the largest tanker fleets and providing critical services to oil operations in our nation's busiest basins. Pilot Company supports a growing portfolio of brands with expertise in supply chain and retail operations, logistics and transportation, technology and digital innovation, construction, maintenance, human resources, finance, sales and marketing.

Company Description

Pilot Company is an industry-leading network of travel centers with more than 30,000 team members and over 750 retail and fueling locations in 44 states and six Canadian provinces. Our energy and logistics division serves as a top supplier of fuel, employing one of the largest tanker fleets and providing critical services to oil operations in our nation's busiest basins. Pilot Company supports a growing portfolio of brands with expertise in supply chain and retail operations, logistics and transportation, technology and digital innovation, construction, maintenance, human resources, finance, sales and marketing.

Founded in 1958 by Jim A. Haslam II and currently led by CEO Adam Wright, our founding values, people-first culture and commitment to giving back remains true to us today. Whether we are serving guests, a fellow team member, or a trucking company, we are dedicated to fueling people and keeping North America moving.

All qualified applicants will receive consideration for employment without regard to race, color, religion, sex, national origin, disability, protected veteran status or any other characteristic protected under applicable federal, state, or local law.

Military encouraged to apply.

Job Description

The purpose of this job is to provide technical expertise for research, development and modification of data engineering processes and workflows in support of big data infrastructure for Pilot Travel Centers in the retail and energy space.

• Research, develop, document, and modify Data Engineering processes and jobs per data architecture and modeling requirements; collaborate with Data Analytics team members such as Data Strategists and Data Scientists


• Collaborate with business stakeholders to understand data needs including data velocity, veracity, and access patterns


• Provide technical expertise to implement Data and Analytics specifications


• Serve on cross-functional project teams and provide perspective for the Data Analytics team on executing key deliverables


• Troubleshoot complex, escalated issues including connection, failed jobs, application errors, server alerts and space thresholds within predefined service level agreements (SLAs)


• Proactively maintain and tune all code according to internally documented Data Engineering standards and best practices


• Review and ensure appropriate documentation for all new development and modifications of the Data Lake processes and jobs


• Perform code and process reviews and oversee testing for solutions developed, and ensure integrity and security of institutional data


• Educate business stakeholders on the usage and benefits of the


• Data Lake/Lakehouse and related technologies


• Mentor and guide less experienced team members and provide feedback on project work


• Model behaviors that support the company’s common purpose; ensure guests and team members are supported at the highest level


• Ensure all activities are in compliance with rules, regulations, policies, and procedures


• Complete other duties as assigned

Qualifications

• Bachelor’s degree in computer science, engineering, information technology, or related field, required.


• Minimum eight (8) years of technology operations experience required.


• Experience in the commodities and energy space preferred.


• Strong SQL knowledge and skills required


• Strong knowledge of Relational Databases like Oracle, Postgres or SQL Server required


• Experience with Apache Spark or Spark-streaming, Message Queue technologies and Python required


• Strong knowledge of enterprise data warehouse (EDW) data models with a focus on Star Schema data modeling techniques required


• Experience with unified data platforms such as Databricks or Snowflake preferred


• Strong knowledge of Cloud Data platforms such as Amazon Web Services (AWS), Azure or Google Cloud preferred


• Excellent analytical skills and the ability to identify solutions to complex data problems


• Ability to provide excellent customer service


• Excellent written and verbal communication skills


• Willingness to learn and embrace new technologies


• Ability to mentor and motivate a diverse team; ensure team and individual accountability and performance standards are met


• Ability to prioritize, multitask and manage multiple projects successfully in a fast-paced and dynamic environment


• Strong organizational skills with attention to detail


• Ability to communicate and interact effectively with different levels of the organization to negotiate, problem solve, complete projects and influence decision making


• Self-motivated with ability to work both independently and within teams in order to establish and meet deadlines, goals, and objectives

Additional Information

• Travel required less than 10%


• General office work requiring sitting or standing for long periods of time


• Able to lift up to 30 lbs.


• Able to work evenings, weekends and odd hours as needed