Data Engineer

Credence Management Solutions, LLC; McLean (VA); On-site; USD 80,000 - 120,000; Full time

Job summary

Credence Management Solutions, LLC is looking for a motivated Data Engineer in McLean, Virginia. The Data Engineer will design, develop, and maintain efficient data pipelines, ensuring data flows seamlessly from various sources into our data systems.

The ideal candidate will have a strong background in data engineering, programming skills in Python or Java, and knowledge of ETL processes, along with excellent analytical and problem-solving capabilities.

A Bachelor's degree in a relevant field is mandatory, and candidates must be clearable for DoD Public Trust or higher.

Qualifications

• Clearable for DoD Public Trust or higher.
• Understanding of data ingestion, standardization, and transformation.
• Knowledge of distributed computing and big data.

Responsibilities

• Design and maintain data pipelines for data warehouse.
• Collaborate with teams for data processing solutions.
• Monitor and troubleshoot data issues.

Skills

Data engineering concepts

Python

SQL

Data modeling

Apache Hadoop

Kafka

AWS

Analytical skills

Communication

Attention to detail

Education

Bachelor's degree in computer science or related field

Tools

Data Orchestration tools

ETL/ELT processes

SQL databases

Job description

We are seeking a motivated and skilled Data Engineer to join our team. The ideal candidate is passionate about data engineering, possesses a strong analytical mindset, and demonstrates a keen interest in working with large-scale data systems. In this role, you will assist in designing, developing, and maintaining our data pipelines, ensuring the efficient flow of data between various systems. You will collaborate closely with cross-functional teams, including data scientists, cloud engineers, software engineers, and business analysts, to enable data-driven decision-making and deliver high-quality solutions.

Responsibilities

• Assist in designing, developing, and maintaining data pipelines, ensuring smooth data flow from diverse sources into our data warehouse and lakes
• Familiarity with Data Orchestration tools and concepts such as Medallion Architecture
• Collaborate with data scientists and software engineers to implement efficient and scalable data processing solutions.
• Perform data cleansing, transformation, and aggregation to ensure data integrity and accuracy.
• Help optimize and tune data pipelines for performance and reliability.
• Assist in monitoring and troubleshooting data-related issues to ensure smooth operations.
• Collaborate with cross-functional teams to understand data requirements and translate them into technical solutions.
• Stay up-to-date with emerging data engineering technologies and best practices to contribute innovative ideas and improvements.
• Document data engineering processes, procedures, and standards for future reference.

Education, Requirements and Qualifications

• Must be clearable for DoD Public Trust or higher
• Bachelor's degree in computer science, information systems, economics or a related critical thinking field. Equivalent practical experience will also be considered.
• Strong understanding of data engineering concepts and principles (ingestion, standardization, transformation)
• Proficiency in programming languages such as Python, Java, or Scala.
• Proficiency with SQL and relational databases
• Familiarity with data modeling and ETL/ELT (Extract, Transform, Load) processes.
• Basic understanding of distributed computing and big data technologies such as Apache Hadoop, Spark, or Kafka.
• Knowledge of cloud-based data storage and processing platforms like AWS, Azure, or Google Cloud is a plus.
• Excellent problem-solving and analytical skills.
• Strong communication and teamwork abilities.
• Attention to detail and commitment to delivering high-quality work.
• Ability to work in a fast-paced environment and handle multiple tasks simultaneously

Nice to have

• Familiarity with DevOps scripting languages (Gitlab CI/CD)
• AWS Cloud Practitioner or AWS Associate Level Certification
• Streaming data experience: Kafka, Firehose, Kinesis, etc
• ElasticSearch or OpenSearch integration