Data Engineer (All Levels) - Shiloh, IL

Silverthorne Advisory Group; Illinois; On-site; USD 110,000 - 140,000; Full time

Benefits offered by this job

Health insurance

401(k)

401(k) matching

Paid time off

Bonus based on performance

Professional development

Company parties

Job summary

Silverthorne Advisory Group is seeking a Data Engineer to design, build, and maintain advanced data infrastructures in the Defense sector. The role emphasizes ETL, Databricks, Python, Spark, Scala, JavaScript/JSON, SQL, and Jupyter Notebooks for efficient data processing and analysis.

You will collaborate with data scientists and analysts to deliver timely data insights, while ensuring data quality, governance, and regulatory compliance across large federal datasets.

Qualifications

• Experience designing end-to-end data pipelines with ETL tooling.
• Proven ability to work with large datasets and big data tech.
• Knowledge of data governance and regulatory best practices.

Responsibilities

• Design, develop, and implement end-to-end data pipelines using ETL and Databricks.
• Create and optimize pipelines for scalability and high-performance processing.
• Perform data cleansing, integration, and quality assurance on large datasets.
• Leverage big data tech to process federal agency data efficiently.
• Troubleshoot data issues and implement innovative solutions.
• Enforce data governance and regulatory compliance.
• Collaborate with cross-functional teams to design data models and architectures.
• Provide data insights to stakeholders and support decision-making.
• Maintain documentation for applications and workflows.
• Stay updated with evolving data engineering trends and tools.

Skills

ETL processes

Big data processing

Data governance

Regulatory compliance

Tools

Databricks

Python

Spark

Scala

JavaScript/JSON

SQL

Jupyter Notebooks

AWS S3

Job description

Job Description

Silverthorne Advisory Group is seeking a skilled and highly motivated Data Engineer to join an exciting and growing opportunity within the Defense sector. As a Data Engineer, you will play a crucial role in designing, developing, and maintaining our Advana data infrastructure and systems. Your expertise in ETL, Databricks, Python, Spark, Scala, JavaScript/JSON, SQL, and Jupyter Notebooks will be essential in ensuring efficient data processing and analysis.

Responsibilities

• Design, develop, and implement end-to-end data pipelines, utilizing ETL processes and technologies such as Databricks, Python, Spark, Scala, JavaScript/JSON, SQL, and Jupyter Notebooks.
• Create and optimize data pipelines from scratch, ensuring scalability, reliability, and high-performance processing.
• Perform data cleansing, data integration, and data quality assurance activities to maintain the accuracy and integrity of large datasets.
• Leverage big data technologies to efficiently process and analyze large datasets, particularly those encountered in a federal agency.
• Troubleshoot data-related problems and provide innovative solutions to address complex data challenges.
• Implement and enforce data governance policies and procedures, ensuring compliance with regulatory requirements and industry best practices.
• Work closely with cross-functional teams to understand data requirements and design optimal data models and architectures.
• Collaborate with data scientists, analysts, and stakeholders to provide timely and accurate data insights and support decision-making processes.
• Maintain documentation for software applications, workflows, and processes.
• Stay updated with emerging trends and advancements in data engineering and recommend suitable tools and technologies for continuous improvement.

Requirements

• Secret clearance or higher required (can be sponsored).
• Strong SQL knowledge: Ability to write, troubleshoot, and modify complex SQL procedures and queries across a variety of datasets.
• Extract, Transform, Load (ETL) experience: Experience with establishing and maintaining data load processes as well as manipulating and combining various data sources.
• Business Intelligence experience: Creating and updating applications, reports, dashboards, etc. Cognos Analytics, Qlik Sense, and Power BI especially are relevant to the role.
• Python and Coding experience: Hands-on experience with languages/technologies such as Python, Spark, Scala, JavaScript/JSON, and SQL.
• Databricks experience: Familiarity with Databricks, Amazon AWS/S3, Jupyter Notebooks will be helpful in the role.
• Strong problem-solving skills and ability to solve complex data-related issues.
• Demonstrated experience working with large datasets and leveraging big data technologies to process and analyze data efficiently.
• Understanding of data modeling/visualization, database design principles, and data governance practices.
• Excellent communication and collaboration skills, with the ability to work effectively with cross-functional teams.
• Detail-oriented mindset with a commitment to delivering high-quality results.
• Teradata experience is a plus. Knowledge of Multiloads, BTEQs, and other assorted Teradata procedures will be helpful in the role.
• Accounting knowledge is a plus. Some background on DoD accounting system (DEAMS, TFMS, NERP etc.) and transportation policies will be helpful in the role.

Benefits

• Retirement Plan
• 401(k)
• 401(k) matching
• Bonus based on performance
• Company parties
• Competitive salary
• Dental insurance
• Health insurance
• Opportunity for advancement
• Paid time off
• Training & development
• Vision insurance
• Compensation - We carefully consider a wide range of compensation factors, including but not limited to prior experience, skills, expertise, location, and other considerations permitted by law.
• Healthcare - We offer Health, Vision, and Dental Plans for our employees and their families.
• Retirement Plan – We invest in your future with a competitive 401(k) plan, where we match 100% of your contributions up to your first 6% and give you access to Vanguard Admiral funds.
• Paid Time Off - Based on length of service, we offer a generous amount of paid leave.
• Bonus System – As you invest in us, we invest in you. We offer bonuses to all employees who meet and exceed goals throughout the year.
• Professional Development – Support for career growth through training programs and certifications.
• Company Retreats & Team Events – Sponsored trips, team-building activities, and annual conferences related to your skillset.