Data Engineer - Cloud Analytics & AI Pipelines

Talentify; Whitehall (OH); On-site; USD 131,000 - 171,000; Full time

Job summary

Talentify is seeking a data engineer to architect, build, and operate scalable data pipelines ingesting data from DLA, DoD, and commercial sources. The role focuses on batch and real-time processing using Python, Scala, or Java with Spark and Airflow to support strategic reporting and AI model serving.

Responsibilities include designing data warehouse/data lake structures, implementing data governance, and optimizing cloud warehouses like Snowflake, Redshift, or BigQuery while collaborating with

Qualifications

• U.S. citizen required.
• Experience building scalable data pipelines and cloud data warehouses.
• Proficiency with Python, Scala, or Java.
• Knowledge of CI/CD practices for pipelines.
• Experience with Apache Spark and Airflow.
• Ability to translate data requirements for AI/ML use cases.
• Strong communication to non-technical stakeholders.

Responsibilities

• Architect, develop, test, and maintain scalable data pipelines from diverse sources.
• Develop batch and real-time data solutions using Python/Scala/Java, Spark and Airflow.
• Implement CI/CD for automated testing and deployment of pipelines.
• Design and administer data warehouse and data lake structures (dimensional/Data Vault).
• Optimize cloud data warehouses (Snowflake/Redshift/BigQuery) with partitioning and clustering.
• Enforce data governance and security controls for CUI/Secret data.
• Advise AI/Analytics COE and translate data needs into technical specs.
• Mentor junior staff and conduct peer reviews of data engineering work.
• Explain complex data concepts to leadership to guide decisions.
• Research new technologies and prototype cloud services for modernization.

Skills

Python

Scala

Java

ETL/ELT

Apache Spark

Airflow

CI/CD

Data warehousing

Snowflake

Redshift

BigQuery

Data governance

Mentoring

Communication

Tools

Snowflake

Redshift

BigQuery

Airflow

Spark

Job description

Talentify is seeking a data engineer to architect, build, and operate scalable data pipelines ingesting data from DLA, DoD, and commercial sources. The role focuses on batch and real-time processing using Python, Scala, or Java with Spark and Airflow to support strategic reporting and AI model serving.

Responsibilities include designing data warehouse/data lake structures, implementing data governance, and optimizing cloud warehouses like Snowflake, Redshift, or BigQuery while collaborating with