Senior Azure Data Engineer / Architect

TechTrend, Inc.; Reston (VA); Remote; USD 140,000 - 200,000; Full time

Benefits offered by this job

Short-term disability

401(k) with match

Job summary

TechTrend, Inc. is seeking a Senior Cloud Data Engineer / Data Architect to design and operate enterprise-scale cloud data platforms in the Azure ecosystem.

You will work hands-on across Azure Databricks, ADLS Gen2, Synapse, Data Factory, SQL, Python/PySpark and related services to deliver scalable data solutions. You will translate business needs into robust data architectures, govern data quality and security, and collaborate with analytics, governance and AI teams.

Qualifications

• 7+ years of data engineering or architecture experience.
• 4+ years designing cloud-based data platforms on Azure.
• Strong Azure data services experience with Databricks and Spark.
• Proficiency in SQL and Python/PySpark for data processing.
• Experience with Lakehouse, Delta Lake, and governance concepts.

Responsibilities

• Architect and implement cloud data platforms and Lakehouse architectures on Azure.
• Design scalable data lakes, data warehousing, and analytical pipelines.
• Build batch and streaming pipelines using Azure Databricks, Data Factory, Synapse.
• Implement data governance, security, and lineage with Unity Catalog and Purview.
• Collaborate with AI/ML teams to prep governed enterprise data.

Skills

Azure Databricks

Azure Data Factory

Azure Synapse

SQL

Python

Spark/PySpark

Lakehouse

Unity Catalog

Tools

ADLS Gen2

Delta Lake

Databricks SQL

Power BI

Job description

Position Overview

TechTrend is seeking a Senior Cloud Data Engineer / Data Architect to design, build, modernize, and support enterprise-scale cloud data platforms, primarily within the Microsoft Azure ecosystem.

This is a senior, hands-on technical role. The ideal candidate will combine strong data architecture expertise with the engineering depth necessary to build and troubleshoot production data solutions using Azure Databricks, Azure Data Lake Storage (ADLS Gen2), Azure Synapse Analytics, Azure Data Factory, SQL, Python/PySpark, and related Azure Data & AI services.

The candidate should be comfortable working across architecture, data engineering, analytics, governance, security, and AI/ML integration and translating business and mission requirements into scalable production solutions.

Key Responsibilities

• Architect and implement modern cloud data platforms and Lakehouse architectures using Azure and Databricks.
• Design scalable data lakes, Lakehouse, data warehouse, and analytical architectures supporting structured, semi-structured, and unstructured data.
• Build and optimize batch and streaming data pipelines using technologies such as Azure Databricks, Spark/PySpark, Azure Data Factory, Synapse pipelines, Event Hubs, and related services.
• Design and implement solutions using ADLS Gen2, Delta Lake, Databricks SQL, Unity Catalog, Azure Synapse Analytics, and Azure SQL.
• Design ingestion and integration patterns for data originating from databases, APIs, files, applications, cloud platforms, and legacy/on-premises systems.
• Define data modeling strategies including dimensional models, curated data products, transformation layers, and analytical datasets.
• Establish data governance, lineage, cataloging, access-control, data-quality, and security patterns, including Unity Catalog and Microsoft Purview where appropriate.
• Design secure data architectures using Microsoft Entra ID, managed identities, RBAC, Key Vault, private endpoints, and least-privilege principles.
• Work with AI/ML teams to prepare governed enterprise data for machine learning, Generative AI, RAG, analytics, and other AI-enabled use cases.
• Support integration between Databricks and Azure AI/ML capabilities, APIs, applications, reporting platforms, and enterprise systems.
• Evaluate performance, scalability, availability, security, and cost characteristics of proposed architectures.
• Develop reusable engineering patterns, reference architectures, notebooks, pipelines, IaC, technical standards, and deployment documentation.
• Troubleshoot complex data pipeline, integration, Spark, SQL, performance, security, and connectivity issues.
• Collaborate directly with customers, architects, application developers, cybersecurity teams, cloud engineers, and business stakeholders.
• Provide technical leadership and mentoring to data engineers while remaining actively involved in design and implementation.

Required Qualifications

• 7+ years of professional experience in data engineering, data architecture, analytics engineering, cloud engineering, or related disciplines.
• 4+ years of hands-on experience designing or implementing cloud-based data platforms.
• Strong hands-on expertise with Microsoft Azure data services.
• Demonstrated experience with Azure Databricks and Apache Spark/PySpark.
• Strong understanding of Lakehouse and Data Lake architecture, including Delta Lake concepts.
• Strong SQL and Python/PySpark engineering skills.
• Experience designing enterprise data ingestion, ETL/ELT, transformation, orchestration, and integration pipelines.
• Experience with data modeling, schema design, performance optimization, partitioning, and large-scale dataset processing.
• Understanding of Databricks Unity Catalog, data governance, lineage, RBAC, and access-control concepts.
• Experience implementing secure cloud architectures for sensitive or regulated data.
• Experience integrating cloud data platforms with downstream analytics, applications, APIs, and AI/ML workloads.
• Strong analytical and troubleshooting skills.
• Ability to translate business requirements into scalable technical architectures.
• Excellent written and verbal communication skills, including the ability to explain architectural decisions and technical trade-offs.

Preferred Qualifications

• Experience with Azure Government and/or federal government environments.
• Previous experience supporting USDA or other federal civilian agencies.
• Experience with Microsoft Fabric and Power BI.
• Experience with Azure AI services, Azure Machine Learning, MLflow, Mosaic AI, Generative AI, RAG, vector search, or AI/ML pipelines.
• Experience with real-time/streaming architectures using Kafka, Event Hubs, or Spark Structured Streaming.
• Experience with Terraform, Bicep, GitHub, Azure DevOps, and CI/CD for data-platform deployment and automation.
• Experience with Microsoft Purview or other enterprise data governance/catalog platforms.
• Experience integrating Databricks with enterprise systems and APIs.
• Familiarity with federal security and governance frameworks including NIST 800-53, FedRAMP, and FISMA.
• Relevant certifications such as:
• Azure architecture certifications

What We're Looking For

We are not looking for someone who is purely an architect or purely a traditional ETL developer.

The successful candidate will be a hands-on Data Engineer / Architect who can move comfortably between:

They should be able to whiteboard the target architecture with stakeholders and then get into Databricks or Azure and help the team build, troubleshoot, optimize, and operationalize it.

Job Location

Primarily remote, with occasional onsite attendance at our Reston Town Center office, up to one day per week.

Short, Long Term Disability Insurance

401(k) Plan (matching component included)

TechTrend, Inc. is a Small Business providing IT products and services to both the government and commercial markets. We were founded in 2003 and continue to evolve as a leading provider of enterprise-wide services and technology solutions. TechTrend is a noted and trusted partner to our clients who seek our knowledge and guidance for their most critical and complex IT projects. TechTrend's mission is to provide comprehensive, enterprise-wide IT solutions to our clients' complex challenges. We serve as a liaison for positive organizational change by forming relationships and building bridges across functions, gaining buy-in from both leaders and end-users, and removing barriers to mission success. Our established processes ensure quality delivery of results by maximizing efficiency, productivity, and client satisfaction enterprise-wide.