Data Engineer

Jobtailor; Pittsburgh (Allegheny County); On-site; USD 120,000 - 180,000; Full time

Job summary

Jobtailor is seeking a senior data engineer leader to drive Palantir Foundry (Vantage) initiatives and build scalable data pipelines on Databricks.

You will mentor teams, design ontologies, and integrate AI/ML capabilities into data platforms, collaborating with scientists, designers, and developers in an Agile environment.

Qualifications

• Bachelor’s degree or equivalent work experience in a relevant field.
• 3–5 years of data engineering with hands-on Palantir Foundry (Vantage).
• Experience with Foundry pipeline development (PySpark/Pipeline Builder) and Workshop app building.
• Working experience with Databricks notebooks, Delta Lake, and Spark processing.
• Proficiency in Python and SQL for data manipulation and pipeline development.
• Familiarity with data visualization (Plotly/D3.js/Recharts) for surfacing outputs.
• Experience planning, designing, implementing, testing, and maintaining data pipelines for enterprise systems.
• Familiarity with Git, Agile, CI/CD, and automated deployments.
• Leadership roles, mentorship, or client relationship management.
• Active Secret Clearance or ability to obtain one.

Responsibilities

• Lead data engineering efforts within Foundry (Vantage), including ontology design and pipelines.
• Build and optimize Foundry and Databricks pipelines using Python and SQL.
• Develop data workflows in Databricks to complement Foundry environments.
• Create reusable, reliable code for Foundry Workshop or Slate apps.
• Translate analytics into governed data products surfaced through Foundry and Databricks.
• Lead AI/ML integration into Foundry and Databricks apps.
• Champion platform-first development practices and mentor the team.
• Provide technical leadership and grow team capabilities across Foundry and Databricks.
• Engage with Army stakeholders and expand development footprint.
• Collaborate with data scientists, UX designers, and developers using Git; follow Agile.

Skills

Leadership

Mentorship

Agile methodologies

Data-driven decision making

Communication

Education

Bachelor’s degree in Computer Science or related field

Tools

Palantir Foundry (Vantage)

Databricks

Python

SQL

PySpark

Delta Lake

Plotly/D3.js/Recharts

Git

CI/CD

Job description

• Lead data engineering efforts within Palantir Foundry (Vantage), including ontology design, pipeline development, and data integration for Army AI2C mission applications
• Build, maintain, and optimize Foundry pipelines using Python-based transforms (PySpark/Pipeline Builder), Code Repositories, and Foundry’s data connection framework
• Develop and maintain data engineering workflows in Databricks, including notebooks, Delta Lake tables, and Spark jobs, for workloads that complement or feed into the Foundry environment
• Write, design, and implement efficient, reusable, and reliable code using Python, PySpark, SQL, and TypeScript/JavaScript as needed for Foundry Workshop or Slate applications
• Translate complex data science and analytics outputs into governed, reusable data products surfaced through Foundry and Databricks
• Lead the integration of AI/ML capabilities into Foundry and Databricks-based applications, identifying opportunities to leverage emerging AI technologies (including Foundry AIP or Databricks ML/AI tooling)
• Champion AI-forward, platform-first development practices and guide the team in implementing AI/ML features and data-driven experiences
• Provide technical leadership and mentorship to grow the team’s capabilities across Foundry (Vantage) and Databricks
• Identify and pursue opportunities to expand development initiatives with Army stakeholders
• Collaborate effectively with data scientists, UX designers, and software developers using Git, participating in code reviews and adopting Agile methodologies
• Engage with Army managers and representatives in a consultative capacity to identify data platform needs, explore innovative data-driven solutions including AI applications, and build strategic relationships that expand our development footprint

Requirements

• Bachelor’s degree in Computer Science, Data Science, Data Engineering, Software Engineering, Information Systems, or a related field, or equivalent work experience
• 3-5 years of experience in data engineering, with hands‑on experience building and operating applications on the Palantir Foundry (Vantage) platform
• Demonstrated experience with Foundry pipeline development (PySpark/Pipeline Builder), Code Repositories, ontology/object modeling, and Foundry Workshop application building
• Working experience with Databricks, including notebooks, Delta Lake, and Spark-based data processing, as a secondary platform to Foundry
• Proficiency in Python and SQL for data manipulation and pipeline development (pandas, PySpark, numpy, or similar)
• Familiarity with data visualization approaches (D3.js, Plotly, Recharts, or Foundry-native Workshop widgets) for surfacing pipeline outputs
• Demonstrated experience in planning, analyzing, designing, implementing, testing, and maintaining data pipelines and applications for an enterprise-level system
• Familiarity with Git, Agile methodologies, CI/CD, and automated deployments
• Demonstrated experience in leadership roles, technical mentorship, or client relationship management
• Active Secret Clearance or ability to obtain one is required