Data Engineer (Microsoft Fabric and Databricks)

Global Technical Talent, Inc.; Reston (VA); Hybrid; USD 82,000 - 106,000; Part time

Benefits offered by this job

Medical Insurance

Vision Insurance

Dental Insurance

401k Retirement

Job summary

Global Technical Talent, Inc. is seeking a Senior Data Engineer to design and operate enterprise data platforms on Microsoft Fabric and Databricks. You will drive ingestion pipelines, lakehouse architectures, and data governance across production environments.

The position requires 10–15 years of experience in data engineering, data platforms, cloud technologies, and cross-functional collaboration to deliver scalable, secure data solutions with cost efficiency.

Qualifications

• 10–15 years of overall relevant experience in data engineering, data platforms, analytics, and cloud technologies.
• Minimum 5 years of hands-on experience with Databricks and 2 years with Microsoft Fabric environments.
• Proven experience designing and supporting production-grade ETL/ELT and data integration solutions.

Responsibilities

• Own end-to-end technical design, development, and delivery of enterprise data engineering solutions.
• Design and implement scalable ETL/ELT, data integration, and data transformation pipelines for structured and unstructured data sources.
• Develop and maintain batch, near real-time, and streaming data ingestion pipelines from databases, APIs, files, cloud storage, and enterprise applications.
• Build reusable notebooks, data transformations, and processing frameworks using Databricks, Fabric, Python, PySpark, and SQL.
• Design Lakehouse architectures using Delta Lake, OneLake, and medallion layers (Bronze, Silver, Gold).
• Create trusted datasets, curated data products, and semantic data structures for analytics, AI, and ML initiatives.
• Implement data quality frameworks, validation rules, reconciliation controls, and exception management processes.
• Design scalable data models, dimensional schemas, and data marts to optimize reporting and analytics.
• Optimize Spark workloads, SQL queries, storage strategies, and data processing performance.

Skills

Databricks

Spark

Delta Lake

Microsoft Fabric

OneLake

Lakehouse

Data Warehouse

Python

PySpark

SQL

ETL/ELT

Ingestion patterns

Batch processing

Streaming data

Dimensional modeling

Education

B.Tech in Computer Science or Engineering

Microsoft Fabric Analytics Engineer (preferred)

Azure Data Engineer Associate (preferred)

Databricks Data Engineer Professional/Associate (preferred)

Tools

Databricks Admin

Fabric Admin

Azure AD / Entra

RBAC

Key Vault

Job description

Data Engineer (Microsoft Fabric and Databricks)

Location: Reston, VA

Onsite Flexibility: Hybrid

Contract Details

• Position Type: Contract
• Contract Duration: 12 months
• Pay Rate: $59.35–$77.03 / Hour (USD)
• Work Authorization: Applicants must be authorized to work for ANY employer in the U.S. We are unable to sponsor or take over sponsorship of an employment Visa at this time.

Job Summary

The Senior Data Engineer is responsible for the design, development, administration, governance, and operational support of enterprise data platforms built on Microsoft Fabric. This role combines deep data engineering expertise with platform administration capabilities to ensure scalable, secure, reliable, and cost-effective data solutions that support business operations, analytics, AI initiatives, and project delivery.

The position requires 10–15 years of overall relevant experience in data engineering, data platforms, cloud technologies, and enterprise data management.

The role will design, build, and support data ingestion, ETL/ELT pipelines, lakehouse architectures, and analytical datasets using Microsoft Fabric, Databricks, Python, PySpark, SQL, Spark, Delta Lake, OneLake, and Data Pipelines. In addition, the role will govern Databricks and Fabric environments, including security, access management, monitoring, performance tuning, capacity management, and platform operations.

The Senior Data Engineer will collaborate with business stakeholders, architects, and analytics teams to deliver robust data solutions while ensuring platform reliability, compliance, security, and operational excellence. This role will also drive continuous improvement in data quality, governance, automation, performance, and cost optimization.

Key Responsibilities

Data Engineering & Solution Delivery

• Own end-to-end technical design, development, and delivery of enterprise data engineering solutions.
• Design and implement scalable ETL/ELT, data integration, and data transformation pipelines for structured and unstructured data sources.
• Develop and maintain batch, near real-time, and streaming data ingestion pipelines from databases, APIs, files, cloud storage, and enterprise applications.
• Build reusable notebooks, data transformations, and processing frameworks using Databricks, Microsoft Fabric, Python, PySpark, and SQL.
• Design and implement Lakehouse architectures using Delta Lake, OneLake, and medallion architecture (Bronze, Silver, Gold layers).
• Create trusted datasets, curated data products, and semantic-ready data structures supporting analytics, reporting, AI, and machine learning initiatives.
• Implement data quality frameworks, validation rules, reconciliation controls, and exception management processes.
• Design scalable data models, dimensional schemas, and data marts to optimize reporting and analytical workloads.
• Optimize Spark workloads, SQL queries, storage strategies, partitioning, and data processing performance.

Databricks & Microsoft Fabric Administration

• Administer Databricks workspaces, Fabric capacities, Fabric workspaces, Lakehouses, Data Warehouses, Dataflows, and Pipelines.
• Configure and maintain user access, security groups, RBAC permissions, service principals, and identity integrations.
• Manage secrets, credentials, key vault integrations, and secure access to enterprise data sources.
• Monitor platform health, resource utilization, workload performance, and capacity consumption.
• Perform platform upgrades, feature enablement, release validation, and environment maintenance activities.
• Implement capacity planning, workload management, and resource allocation strategies.
• Support governance initiatives by enforcing workspace standards, naming conventions, tagging, and compliance controls.

Operations, Governance & Reliability

• Develop operational runbooks, support procedures, and disaster recovery documentation.
• Monitor data pipelines and platform services to ensure availability and reliability.
• Establish and maintain metadata management, data lineage, cataloging, and governance processes.
• Implement observability, logging, monitoring, and alerting across the data platform ecosystem.
• Drive cost optimization initiatives across compute, storage, Fabric capacities, and Databricks workloads.
• Establish Git-based source control and branching strategies for notebooks, pipelines, and infrastructure components.
• Implement CI/CD pipelines for automated deployment, testing, environment promotion, and release management.
• Develop reusable deployment templates and engineering accelerators for platform consistency and scalability.
• Communicate technical solutions, trade-offs, risks, dependencies, and delivery plans to both technical and non-technical stakeholders.

Required Skills

Data Engineering

• Strong expertise in Databricks, Spark, Delta Lake, Microsoft Fabric, OneLake, Lakehouse, and Data Warehouse architectures.
• Advanced proficiency in Python, PySpark, SQL, and Notebook-based development.
• Expertise in ETL/ELT frameworks, data ingestion patterns, incremental processing, CDC, and data transformation techniques.
• Strong understanding of dimensional modeling, data warehousing, and semantic data structures.
• Experience with batch processing, streaming architectures, and real-time data integration.

Platform Administration

• Strong knowledge of Databricks Workspace Administration and Microsoft Fabric Administration.
• Experience managing capacities, workspaces, governance controls, security policies, and platform operations.
• Expertise in Microsoft Entra ID (Azure AD), RBAC, access controls, service principals, and security integration.
• Experience with secrets management, key vaults, credential management, and secure data access.
• Knowledge of platform monitoring, observability, troubleshooting, incident management, and operational excellence.

Governance & DevOps

• Strong understanding of data governance, metadata management, lineage, data cataloging, and compliance controls.
• Experience with Git, Azure DevOps, CI/CD pipelines, release management, and automated testing.
• Knowledge of Infrastructure as Code, automation frameworks, and deployment standards.
• Experience implementing enterprise data quality and reliability frameworks.

Performance & Cost Optimization

• Experience tuning Spark applications, SQL workloads, storage structures, and distributed compute environments.
• Knowledge of Fabric capacity optimization, Databricks cluster optimization, workload management, and FinOps practices.
• Ability to balance performance, scalability, reliability, security, and cost considerations.
• Ability to work effectively in a fast-paced enterprise environment supporting multiple business priorities.

Preferred Skills

• Experience mentoring engineers and supporting enterprise platform operations.

Education Requirements

• B.Tech in Computer Science, Information Technology, Engineering, or an equivalent degree is required.
• Microsoft Certified: Fabric Analytics Engineer Associate (preferred)
• Microsoft Certified: Azure Data Engineer Associate (preferred)
• Databricks Certified Data Engineer Professional (preferred)
• Databricks Certified Data Engineer Associate (preferred)

Required Experience

• 10–15 years of overall relevant experience in data engineering, data platforms, analytics, and cloud technologies.
• Minimum 5 years of hands-on experience with Databricks and 2 years with Microsoft Fabric environments.
• Proven experience designing and supporting production-grade ETL/ELT and data integration solutions.
• Experience administering enterprise-scale Databricks and Microsoft Fabric platforms.
• Strong experience working with Azure data services and cloud-native architectures.
• Demonstrated experience in stakeholder engagement, solution ownership, and technical leadership.

Benefits

• Medical, Vision, and Dental Insurance Plans
• 401k Retirement Fund

About the Client

This client is a global leader in engineering, procurement, and construction, with a project portfolio spanning more than 25,000 engagements across 160 countries on all seven continents over more than 125 years. The organization delivers some of the world's most complex and consequential infrastructure projects — including power plants and pipelines, airports and transit systems, defense facilities and nuclear infrastructure, and large-scale clean energy initiatives — serving national governments, global energy companies, and the world's largest public agencies. With a Glassdoor rating of 4.0 stars and 78 percent of employees recommending the company, this privately held firm employs civil and structural engineers, project controls specialists, procurement managers, nuclear and environmental remediation professionals, and technical IT professionals who thrive in demanding, fast-paced project environments at a scale nearly impossible to find elsewhere.