Principal Data Engineer

Dollar General; Nashville (TN); On-site; USD 120,000 - 160,000; Full time

Job summary

Dollar General in Nashville, TN is seeking a Principal Data Engineer to lead data architecture, build scalable data pipelines, and optimize data delivery for multiple teams. The role emphasizes partnering with data scientists, engineers, and business analysts to enable reliable data infrastructure.

The ideal candidate will have extensive experience in data systems from the ground up, strong cloud experience (GCP/Snowflake/Azure), and a track record of delivering high-performance data platforms

Qualifications

• H hands-on programming with Python/SQL/GoLang.
• SQL DB design concepts & data models.
• Strong numerical and analytical skills.
• Degree in Computer Science or IT; Master’s preferred.
• Data engineering certification is a plus.
• Experience with Hadoop, Spark, Kafka.
• Experience with MongoDB, PostgreSQL, Cassandra.
• Experience with Airflow, Dataflow, Composer.
• Experience with GCP, Snowflake, Azure; BigQuery/Redshift.
• Experience with stream processing: Kafka, Spark Streaming.
• Knowledge of software engineering principles.
• Multitasking and shifting priorities.
• BI tools for data analysis and QA.

Responsibilities

• Develop, construct, test, and maintain data architectures and pipelines.
• Collaborate with data scientists and stakeholders.
• Assemble large, complex data sets to meet business requirements.
• Build analytics tools using the data pipeline for insights.
• Maintain data security and compliance.
• Identify and implement internal process improvements.
• Keep up-to-date with latest technology trends.

Skills

Python

SQL

GoLang

Big data tooling

Data modeling

Java

Spark

Kafka

Airflow

GCP

Snowflake

Azure

Databricks

NoSQL

BI Tools

Education

9+ years IT/CS experience

BS/MS in CS or related field

4+ years Python or Java dev

4+ years SQL experience

4+ years schema design and dimensional modeling

Cloud platforms (Snowflake/Azure/Databricks)

Tools

Hadoop

Spark

Kafka

Airflow

Dataflow

Composer

GCP

Snowflake

Azure

BigQuery

Redshift

MongoDB

PostgreSQL

Cassandra

Job description

A Principal Data Engineer is responsible for expanding and optimizing data and data pipeline architecture, as well as optimizing data flow and collection for cross-functional teams. This role involves building and optimizing data systems from the ground up, ensuring that data delivery architecture is consistent throughout ongoing projects.

The Principal Data Engineer supports software developers, database architects, data analysts, and data scientists on data initiatives, ensuring optimal data delivery and architecture. This position requires expertise in data pipeline building and data wrangling, with a focus on enhancing data systems for improved efficiency and performance.

Duties & Responsibilities: What major responsibilities does this position have and what percentage of time is spent on completing them? (Typically 5-7)

• Develop, construct, test, and maintain data architectures: responsible for creating and maintaining optimal data pipeline architectures necessary for the organization's operational efficiency. 65%
• Collaborate with data scientists and other stakeholders: expected to collaborate with data scientists, business analysts, IT professionals, and management to assist with data-related technical issues and support their data infrastructure needs. 10%
• Assemble and prepare large, complex data sets to meet functional business requirements: required to work with different teams to understand their data needs and ensure those needs are met. 10%
• Build analytics tools: This will involve developing tools that utilize the data pipeline to provide actionable insights into key business performance metrics. 5%
• Maintain data security and compliance: ensure systems meet industry practices and business requirements for data privacy and protection. 5%
• Identify, design, and implement internal process improvements: This includes optimizing data delivery, automating manual processes, and re-designing infrastructure for greater scalability. 3%
• Keep up-to-date with the latest technology trends: Continually learn and adapt to new technologies that could improve existing systems 2%

Knowledge, Skills and Abilities (KSAs): What KSAs are required to perform this job?

• Hands on experience with programming languages (e.g. Python, SQL, GoLang)
• Knowledge with SQL database design concepts and Data Models
• Great numerical and analytical skills
• Degree in Computer Science, IT, or similar field; a Master's is a plus
• Data engineering certification (e.g., Cloud Engineer, Data Engineer) is a plus
• Experience with big data tools: Hadoop, Spark, Kafka, etc.
• Experience with relational SQL and NoSQL databases, including MongoDG, Postgres and/or Cassandra.
• Experience with data pipeline and workflow management tools: Airflow, Dataflow, Composer, etc.
• Experience with GCP, Snowflake, Azure cloud services: BQ, Dataflow, Redshift
• Experience with stream-processing systems: Kafka, Spark-Streaming, etc.
• Good Knowledge of software engineering principles, methodologies and current best practices.
• Able to multitask and shift priorities as needed.
• Able to use BI Tools for data analysis and data QA.

Work Experience &/or Education: What are the minimum education and/or experience requirements necessary to perform this job?

• 9+ years experience and a degree in information technology or computer science with additional vendor-specific certification.
• BS or MS degree in Computer Science or a related technical field
• 4+ years of Python or Java development experience
• 4+ years of SQL experience (No-SQL experience is a plus)
• 4+ years of experience with schema design and dimensional data modeling
• Ability in managing and communicating data warehouse plans to internal clients
• Experience designing, building, and maintaining data processing systems
• Experience working with a cloud platform such as Snowflake / Azure or Databricks