Remote Data Engineer: Azure Databricks & Lakehouse

TEKsystems; New York (NY); Remote; USD 62,000 - 76,000; Full time

Job summary

TEKsystems is seeking a Data Engineer to join its Enterprise Services Infrastructure team on a fully remote contract basis. You will design and maintain enterprise data pipelines in Azure Databricks, integrating data from Microsoft 365, ServiceNow, Splunk, and cybersecurity systems into analytics-ready Delta Lake tables.

The role emphasizes data governance, lineage, semantic search, natural-language analytics, and applied AI, with opportunities to contribute to lakehouse architectures, data

Qualifications

• Three or more years of hands-on data engineering experience.
• Strong Python and SQL development skills.
• Hands-on experience building data pipelines in Azure Databricks or a comparable Databricks environment.
• Experience with PySpark, Spark SQL, Databricks notebooks, and Delta Lake.
• Experience developing ETL or ELT pipelines and automated data workflows.
• Understanding of medallion architecture and Bronze, Silver, and Gold data layers.
• Experience with data modeling, data warehousing, schema design, and analytics-ready datasets.
• Experience building data-quality validations, monitoring, and error-handling processes.
• Ability to explain previous Databricks projects and individual technical contributions.
• Must be legally authorized to work in the United States without current or future sponsorship and meet any additional client eligibility requirements.

Responsibilities

• Design, build, and maintain enterprise data pipelines using Azure Databricks.
• Integrate ServiceNow, Splunk, Microsoft 365, identity, endpoint, infrastructure, and cybersecurity data.
• Use Python, PySpark, and Spark SQL to transform raw data into analytics-ready Delta Lake tables.
• Develop scalable ETL and ELT workflows using Delta Lake and medallion architecture.
• Build Bronze, Silver, and Gold data layers to support analytics, reporting, and machine learning use cases.
• Create and maintain data models that support Power BI reporting, operational analytics, and business decision-making.
• Build automated data-quality checks, validation processes, monitoring, and error handling.
• Develop data-lineage and governance solutions to identify upstream dependencies, downstream writes, undocumented transformations, stale datasets, ownership gaps, and redundant data products.
• Help create semantic data catalogs that allow users to identify authoritative datasets and understand their business context.
• Support natural-language analytics solutions that help users find relevant data and generate grounded analytical queries.
• Contribute to retrieval-augmented generation solutions that connect current ServiceNow incidents with relevant historical resolutions.
• Use vector embeddings, semantic search, metadata filtering, similarity matching, and clustering to improve enterprise knowledge management.
• Help identify stale, duplicate, underused, missing, or low-quality knowledge articles.
• Collaborate with infrastructure, identity, service desk, governance, cybersecurity, and analytics teams.
• Translate operational challenges into scalable data products, reporting solutions, automation, and applied AI use cases.
• Document pipelines, data models, transformations, dependencies, and engineering processes.

Skills

Python

SQL

Azure Databricks

PySpark

Spark SQL

Tools

Azure Databricks

Delta Lake

Databricks notebooks

Power BI

Job description

TEKsystems is seeking a Data Engineer to join its Enterprise Services Infrastructure team on a fully remote contract basis. You will design and maintain enterprise data pipelines in Azure Databricks, integrating data from Microsoft 365, ServiceNow, Splunk, and cybersecurity systems into analytics-ready Delta Lake tables.

The role emphasizes data governance, lineage, semantic search, natural-language analytics, and applied AI, with opportunities to contribute to lakehouse architectures, data