Data Engineer - AWS/Databricks - Mid Level

Acuity INC; United States; On-site; USD 120,000 - 140,000; Full time

Job summary

Acuity Inc. seeks a Data Engineer to design and deliver AWS cloud-scale data platforms for federal clients. The role emphasizes Spark, Delta Lake, and distributed pipelines on Databricks, combining engineering with strategic enterprise data modernization.

Build PySpark pipelines, design Delta Lake tables, and develop ETL/ELT workflows while collaborating with data architects to ensure secure, scalable AWS storage and access. CI/CD with GitLab/Jenkins is expected.

Qualifications

• 4+ years of experience in data engineering and Agile analytics.
• 4+ years of experience creating software for retrieving, parsing and processing structured and unstructured data.
• 2+ years of experience building scalable ETL and ELT workflows for reporting and analytics.
• 2+ years experience building enterprise data engineering solutions in the cloud, with AWS and Databricks.
• Experience with data quality, validation frameworks, and storage optimization strategies.
• BA or BS degree

Responsibilities

• Build and maintain scalable PySpark-based data pipelines in Databricks notebooks to support ingestion, transformation, and enrichment of structured and semi-structured data.
• Design and implement Delta Lake tables optimized for ACID compliance, partition pruning, schema enforcement, and query performance across large datasets.
• Develop ETL and ELT workflows that integrate multiple source systems into a centralized, query-optimized data warehouse architecture.
• Leverage Spark SQL and DataFrame APIs to implement business rules, dimensional joins, and aggregation logic aligned to warehouse modeling best practices.
• Collaborate with data architects and engineers to implement cloud-native data solutions on AWS using S3, Glue, RDS, and IAM for secure, scalable storage and access control.
• Optimize pipeline performance through intelligent partitioning, caching, broadcast joins, and adaptive query tuning.
• Deploy and version data engineering assets using Git-integrated development workflows and automate deployment with CI/CD tools such as GitLab or Jenkins.
• Monitor pipeline health, job execution, and cluster utilization using native Databricks tools and AWS CloudWatch, identifying bottlenecks and optimizing cost-performance tradeoffs.
• Conduct technical discovery and mapping of legacy source systems, identifying required transformations and designing end-to-end data flows.
• Implement governance practices including metadata tagging, data quality validation, audit logging, and lineage tracking using platform-native features and custom logic.
• Support ad hoc data access requests, develop reusable data assets, and maintain shared notebooks that meet operational reporting and analytics needs across teams.

Skills

Data engineering

Distributed pipelines

Spark

Delta Lake

Databricks notebooks

Query optimization

Data governance

Metadata tagging

Education

Bachelor's degree

Tools

Spark

Databricks

AWS

S3

Glue

RDS

IAM

GitLab

Jenkins

Job description

Acuity Inc. is seeking a highly skilled Data Engineer to join our Engineering Team, helping drive the design and delivery of AWS cloud-scale data platforms for federal clients. This role requires knowledge and/or experience with Spark, Delta Lake, and distributed data pipelines on Databricks. The ideal candidate brings both engineering and strategic insight into enterprise data modernization.

Why Acuity?

Are you ready to use your expertise in the areas of IT Modernization, Data Enablement, and Hyperautomation to make a real difference? Join Acuity, Inc., a technology consulting firm that supports federal agencies. We combine industry partnerships and long-term federal experience with innovative technical leadership to support our customers' critical missions.

Responsibilities

• Build and maintain scalable PySpark-based data pipelines in Databricks notebooks to support ingestion, transformation, and enrichment of structured and semi-structured data.
• Design and implement Delta Lake tables optimized for ACID compliance, partition pruning, schema enforcement, and query performance across large datasets.
• Develop ETL and ELT workflows that integrate multiple source systems into a centralized, query-optimized data warehouse architecture.
• Leverage Spark SQL and DataFrame APIs to implement business rules, dimensional joins, and aggregation logic aligned to warehouse modeling best practices.
• Collaborate with data architects and engineers to implement cloud-native data solutions on AWS using S3, Glue, RDS, and IAM for secure, scalable storage and access control.
• Optimize pipeline performance through intelligent partitioning, caching, broadcast joins, and adaptive query tuning.
• Deploy and version data engineering assets using Git-integrated development workflows and automate deployment with CI/CD tools such as GitLab or Jenkins.
• Monitor pipeline health, job execution, and cluster utilization using native Databricks tools and AWS CloudWatch, identifying bottlenecks and optimizing cost-performance tradeoffs.
• Conduct technical discovery and mapping of legacy source systems, identifying required transformations and designing end-to-end data flows.
• Implement governance practices including metadata tagging, data quality validation, audit logging, and lineage tracking using platform-native features and custom logic.
• Support ad hoc data access requests, develop reusable data assets, and maintain shared notebooks that meet operational reporting and analytics needs across teams.

Qualifications

• 4+ years of experience in data engineering and Agile analytics
• 4+ years of experience creating software for retrieving, parsing and processing structured and unstructured data
• 2+ years of experience building scalable ETL and ELT workflows for reporting and analytics
• 2 + years experience building enterprise data engineering solutions in the cloud, with preferred experience with cloud native technologies from AWS and Databricks
• Experience with data quality, validation frameworks, and storage optimization strategies
• BA or BS degree

Clearance Requirement

Must be US Citizen with an ability to obtain and maintain US Suitability

Min USD $120,000.00/Yr.

Max USD $140,000.00/Yr.

Salary Range The salary range for this position represents Acuity's good faith estimate of the compensation for this job title at the time of posting. Final compensation will be determined based on factors including, but not limited to, the candidate's qualifications, experience, education, certifications, skills, geographic location, contract requirements, applicable labor categories, and, where applicable, contract award and funding.