Senior Big Data Engineer

Omniscius Consulting; Washington (District of Columbia); On-site; USD 120,000 - 180,000; Full time

Job summary

Omniscius Consulting is seeking an experienced Big Data Engineer to design, develop, and support large-scale data pipelines and transformations. The ideal candidate will bring strong SQL, Python, ETL/ELT skills and the ability to communicate complex concepts to clients and cross-functional teams.

Java experience is helpful but not central; responsibilities include building ETL pipelines, writing SQL, Python data transforms, XML parsing, Linux scripting, and collaborating with security,

Qualifications

• Bachelor’s degree or equivalent experience.
• 5+ years in data engineering, ETL/ELT, or related data integration.
• Strong hands-on SQL with complex queries, performance tuning, and data modeling.
• Experience designing, building, optimizing, and supporting large-scale data pipelines.
• Strong Python for data transformation, integration, automation, or pipeline operations.
• Experience parsing, transforming, and validating XML data.
• Experience in Linux and shell environments, including scripting and cron.
• Experience implementing data validation, error handling, monitoring, and production troubleshooting.
• Excellent written and verbal communication skills and ability to explain technical concepts to clients and stakeholders.
• U.S. citizenship and ability to undergo a background check.

Responsibilities

• Design, build, maintain, and optimize ETL/ELT pipelines for large-scale data.
• Write and optimize complex SQL queries, stored procedures, data models, and validation processes.
• Develop data transformation, integration, and automation using Python.
• Use Java as needed to support existing applications and integrations.
• Parse, transform, and validate XML-based data formats.
• Work in Linux and shell environments for scripting, cron jobs, and troubleshooting.
• Implement data validation, error handling, monitoring, and traceability controls.
• Troubleshoot pipeline failures and data discrepancies across environments.
• Collaborate with client, security, infrastructure, data engineering, and delivery teams.
• Communicate issues and risks clearly to technical and nontechnical stakeholders.
• Participate in Agile planning, status reporting, documentation, and cross-team coordination.

Skills

SQL

Python

ETL/ELT

Data pipelines

Communication

Linux

Shell scripting

Agile

Problem solving

Education

Bachelor’s degree or equivalent

Relevant professional experience

Tools

AWS

S3

Glue

Lambda

Redshift

Oracle

PostgreSQL

Power BI

MicroStrategy

Job description

Our client is seeking an experienced Big Data Engineer to design, develop, and support large-scale data pipelines and transformation processes. The ideal candidate will have strong SQL, Python, ETL/ELT, and data engineering experience, along with the ability to communicate technical concepts clearly to clients, stakeholders, and cross-functional teams.

Java experience is helpful for supporting existing systems and integrations but is not the primary focus of this position.

Responsibilities

• Design, build, maintain, and optimize ETL/ELT pipelines that move and transform large-scale data across source and target systems.
• Write and optimize complex SQL queries, stored procedures, data models, and validation processes.
• Develop data transformation, integration, and automation solutions primarily using Python.
• Use Java as needed to support existing applications, services, and system integrations.
• Parse, transform, and validate XML-based data formats.
• Work within Linux and shell environments to support scripting, scheduled jobs, cron processing, log reviews, and troubleshooting.
• Implement data validation, error handling, monitoring, and traceability controls.
• Troubleshoot pipeline failures, data discrepancies, performance issues, and integration defects.
• Collaborate with client, security, infrastructure, data engineering, and delivery teams.
• Communicate technical issues, risks, and recommendations clearly to technical and nontechnical stakeholders.
• Participate in Agile planning, status reporting, documentation, issue tracking, and cross-team coordination.

Required Qualifications

• Bachelor’s degree from an accredited college or university or equivalent professional experience.
• Five or more years of professional experience in data engineering, database development, ETL/ELT development, or a related data integration role.
• Strong hands-on SQL experience, including complex queries, performance tuning, stored procedures, and data modeling.
• Experience designing, building, optimizing, and supporting large-scale data pipelines.
• Strong Python programming experience supporting data transformation, integration, automation, or pipeline operations.
• Experience parsing, transforming, and validating XML-based data.
• Experience working in Linux and shell environments, including scripting, scheduled jobs, cron, and log troubleshooting.
• Experience implementing data validation, error handling, pipeline monitoring, and production troubleshooting.
• Strong written and verbal communication skills.
• Ability to explain technical concepts clearly to clients and stakeholders.
• Ability to collaborate across security, infrastructure, engineering, and delivery teams.
• Self-motivated and proactive, with a positive and collaborative approach.
• U.S. citizenship is required.
• Must be willing and able to undergo a background investigation for a Public Trust suitability determination.

Preferred Qualifications

• Experience using Python as the primary language for data pipeline automation, transformation, monitoring, or production troubleshooting.
• Experience supporting Java-based applications, services, or integrations.
• Working knowledge of AWS data services, including S3, Glue, Lambda, RDS, or Redshift.
• Exposure to XBRL or other structured financial, regulatory, or compliance-driven data formats.
• Experience supporting federal, government, or compliance-driven data environments.
• Experience with Oracle, PostgreSQL, Redshift, or other relational database platforms.
• Familiarity with reporting, analytics, or dashboarding tools such as Power BI or MicroStrategy