Databricks Data Engineer / Business Analyst - Data Contracts & OCDS at Scicom Infrastructure Services Remote

Scicom Infrastructure Services; United States; Hybrid; USD 110,000 - 150,000; Full time

Job summary

Scicom Infrastructure Services seeks a Data Engineer / Business Analyst to drive a large Databricks implementation bridging business, governance, and engineering teams. You will define data contracts, map source data, and build reliable data pipelines for procurement and OCDS data.

Responsibilities include requirements gathering, data profiling, source-to-target mappings, and ensuring data quality and traceability across the contracting lifecycle using Delta Lake and Spark.

Qualifications

• Bachelor’s degree in computer science, information systems, data analytics, or related field.
• At least five years of combined data-engineering, data-analysis, business-analysis, or data-integration experience.
• Hands-on experience implementing Open Contracting Data Standard.
• Experience mapping procurement data to OCDS schemas.
• Strong knowledge of OCDS releases, records, schemas, and validation.
• Three+ years of hands-on Databricks experience.
• Proficiency in Spark, PySpark, Python, SQL, and Delta Lake.
• Experience gathering requirements and writing user stories and acceptance criteria.
• Agile delivery experience and effective stakeholder communication.
• Ability to document rules, data dictionaries, and process flows.

Responsibilities

• Lead requirements sessions with procurement, contracting, analytics, governance, and technical stakeholders.
• Define and maintain data contracts between data producers and consumers.
• Translate business requirements into user stories, mappings, and specs.
• Map source-system data to OCDS fields and structures.
• Develop Databricks data pipelines using Spark, PySpark, Python, and SQL.
• Implement Bronze-Silver-Gold data layers and Delta Lake schemas.
• Validate data against OCDS schemas and data-quality checks.
• Support data governance and access-control planning.
• Collaborate with agile teams on backlogs, sprints, and testing.

Skills

Data contracts

Databricks

Python

SQL

Apache Spark

REST APIs

Agile delivery

Stakeholder communication

Data modeling

Education

Bachelor’s degree in CS/IS/Data analytics

Tools

Delta Lake

JSON Schema

CI/CD tools

Job description

Position Summary Scicom Infrastructure Services is seeking a Data Engineer / Business Analyst to support a large-scale Databricks implementation. This hybrid role will bridge business, contracting, data-governance, and engineering teams to define business requirements, construct data contracts, map source data, and develop reliable Databricks data pipelines.

Key Responsibilities

Data Contracts and Business Analysis

• Lead requirements-gathering sessions with procurement, contracting, program, analytics, governance, and technical stakeholders.
• Define, construct, document, and maintain data contracts between data producers and consumers.
• Establish data-contract requirements covering:
  • Dataset purpose and ownership
  • Source and target systems
  • Schemas, fields, and data types
  • Required and optional attributes
  • Business definitions and transformation rules
  • Data-quality expectations
  • Validation and reconciliation rules
  • Refresh frequency and delivery schedules
  • Versioning and schema-evolution requirements
  • Security classifications and access controls
  • Service-level expectations
  • Issue ownership and change-management procedures
• Translate business requirements into user stories, acceptance criteria, process flows, data mappings, interface specifications, and technical requirements.
• Conduct source-system analysis, data profiling, gap assessments, and source-to-target mapping.
• Identify differences between existing procurement data and required OCDS structures.
• Facilitate agreement among data owners, producers, consumers, architects, and governance teams.
• Maintain traceability from business requirements through data models, engineering implementation, testing, and acceptance.
• Evaluate requested changes for downstream effects on data products, reports, integrations, and analytical use cases.
• Support backlog refinement, sprint planning, demonstrations, testing, and stakeholder acceptance.

OCDS Responsibilities

• Apply the Open Contracting Data Standard to procurement and public-contracting datasets.
• Map source-system data to appropriate OCDS fields and structures.
• Work with data across the contracting lifecycle, including:
  • Planning
  • Tender
  • Award
  • Contract
  • Implementation
• Develop and maintain mappings for OCDS releases, records, release packages, record packages, identifiers, organizations, parties, items, milestones, documents, transactions, amendments, and related contracting elements.
• Interpret and apply OCDS schemas, codelists, validation rules, and implementation guidance.
• Determine whether standard OCDS fields meet project requirements or whether documented extensions are necessary.
• Support the construction of complete contracting records from multiple transactional releases.
• Establish rules for handling amendments, updates, cancellations, corrections, and historical changes.
• Validate transformed data against applicable OCDS JSON schemas.
• Identify missing, invalid, inconsistent, or nonconforming procurement data and work with stakeholders to resolve deficiencies.
• Document assumptions, business rules, mappings, extensions, and exceptions.
• Support the publication, exchange, analysis, or internal use of standardized contracting data.

Databricks Data Engineering

• Design, develop, test, and maintain data pipelines using Databricks, Apache Spark, PySpark, Python, and SQL.
• Build ingestion and transformation pipelines for structured and semi-structured procurement data.
• Process JSON, CSV, XML, Parquet, relational database, API, and file-based data sources.
• Implement Bronze, Silver, and Gold data layers using medallion architecture.
• Build normalized, dimensional, analytical, and OCDS-aligned data models.
• Use Delta Lake capabilities for schema enforcement, schema evolution, versioning, auditability, and reliable processing.
• Develop reusable frameworks for mapping source procurement data into OCDS-compatible outputs.
• Implement batch and incremental ingestion patterns.
• Use Databricks Workflows, notebooks, jobs, Auto Loader, Delta Live Tables or Lakeflow capabilities, as appropriate.
• Develop REST API integrations for source ingestion and downstream data delivery.
• Implement automated data-quality, reconciliation, completeness, and conformity checks.
• Support Unity Catalog implementation for metadata, ownership, lineage, access control, and data discovery.
• Optimize Spark jobs, SQL queries, clusters, partitioning, file sizes, and data layouts.
• Participate in code reviews, automated testing, CI/CD, deployment, monitoring, and production support.
• Investigate pipeline failures, data discrepancies, and performance issues.

Data Quality and Governance

• Define measurable quality rules for accuracy, completeness, validity, timeliness, consistency, and uniqueness.
• Develop validation controls for required OCDS fields, identifiers, dates, amounts, currencies, organizations, classifications, and contracting relationships.
• Create dashboards or reports that show data-contract compliance and data-quality results.
• Establish processes for detecting and managing schema drift.
• Document data lineage from original procurement systems through Databricks transformations and downstream products.
• Work with governance teams to assign data owners, stewards, classifications, retention requirements, and access policies.
• Ensure sensitive procurement and supplier information is handled according to security and privacy requirements.
• Support auditability through documented rules, version-controlled mappings, validation results, and change histories.

Required Qualifications

• Bachelor’s degree in computer science, information systems, data analytics, business analysis, engineering, public administration, supply-chain management, or a related field.
• At least five years of combined data-engineering, data-analysis, business-analysis, or data-integration experience.
• Hands‑on experience implementing or working with the Open Contracting Data Standard.
• Demonstrated experience constructing, documenting, negotiating, or maintaining data contracts.
• Experience mapping procurement or contracting data to OCDS schemas.
• Strong knowledge of OCDS releases, records, schemas, codelists, identifiers, contracting stages, and validation practices.
• At least three years of hands‑on experience with Databricks.
• Strong experience with: Apache Spark, PySpark, Python, SQL, Delta Lake, ETL and ELT pipelines, Data modeling, JSON and JSON Schema, REST APIs, Data‑quality validation, Source-to‑target mapping.
• Experience gathering requirements and translating them into implementable engineering specifications.
• Ability to communicate effectively with technical and nontechnical stakeholders.
• Experience writing user stories, acceptance criteria, business rules, data dictionaries, interface specifications, and process documentation.
• Strong analytical, troubleshooting, facilitation, and documentation skills.
• Experience working within Agile delivery teams.

Preferred Qualifications

• Experience with government procurement, public‑sector contracting, grants, acquisition, supplier, or financial data.
• Experience implementing OCDS extensions or tailoring OCDS for specific organizational requirements.
• Familiarity with procurement classifications, organizational identifiers, tender processes, awards, amendments, milestones, transactions, and contract implementation.
• Databricks Certified Data Engineer Associate or Professional certification.
• Experience with Unity Catalog, Delta Live Tables, Lakeflow, Databricks Workflows, or Structured Streaming.
• Experience with Microsoft Azure, AWS, or Google Cloud.
• Experience with Azure Data Factory, ADLS Gen2, AWS Glue, Amazon S3, Snowflake, dbt, Kafka, Airflow, or Power BI.
• Experience using JSON Schema validation tools and automated data-quality frameworks.
• Knowledge of metadata management, master-data management, data lineage, and reference-data governance.
• Experience with CI/CD, Git, Azure DevOps, GitHub Actions, Jenkins, or Terraform.
• Experience supporting large federal, state, local-government, or regulated-enterprise data programs.
• Familiarity with federal acquisition, procurement, reporting, transparency, or open‑data requirements.

Core Competencies

• Ability to operate equally well in technical engineering and business‑analysis discussions.
• Strong understanding of how data contracts create accountability between data producers and consumers.
• Ability to convert complex procurement processes into clear data structures and transformation rules.
• Attention to detail when interpreting schemas, codelists, business definitions, and validation requirements.
• Strong stakeholder‑facilitation and conflict‑resolution skills.
• Ability to identify gaps and ambiguities before they become engineering defects.
• Commitment to documentation, traceability, governance, and data quality.
• Ability to work effectively within a large, multidisciplinary Databricks team.