Senior Data Engineer: Cloud Data Pipelines & AI

Capital Technology Group; United States; Hybrid; USD 130,000 - 165,000; Full time

Benefits offered by this job

Remote work options

Medical, Dental, Vision

401(k) with 4% matching

Continuing education budget

Wellness budget

Bonus programs

Job summary

Capital Technology Group is seeking a Senior Data Engineer to design, build, and maintain scalable data pipelines and analytics platforms in a federal-focused environment. You will partner with fellow data engineers to evaluate tools, prototype new technologies, and deliver secure, mission-driven data solutions on AWS.

Responsibilities include building ETL/ELT workflows, leveraging PySpark, Databricks, and SQL on PostgreSQL, while integrating diverse data sources and ensuring FedRAMP-aligned

Qualifications

• Bachelor's degree in Computer Science, Engineering, or a related technical field.
• 9+ years of professional experience in data engineering or related domains.

Responsibilities

• Design, build, and maintain scalable data pipelines, ETL/ELT workflows, and data models using Python, Spark, Databricks, dbt, SQL on PostgreSQL, and AWS Glue.
• Develop and optimize AWS-native data platforms leveraging AWS Glue, EMR, MWAA, Lambda, Step Functions, S3, Redshift, RDS, DMS, and CloudWatch.
• Build high-performance ingestion, transformation, and orchestration workflows for structured and semi-structured data using Iceberg, Parquet, ORC, Avro.
• Design and optimize analytical data platforms using Athena, Trino, Hive, OpenSearch, and enterprise data catalog technologies.
• Integrate enterprise and external data sources across relational and NoSQL platforms including PostgreSQL, Oracle, Redshift, GraphDB, and other NoSQL databases.
• Build AI-enabled data solutions using Amazon Bedrock, RAG pipelines, and vector search technologies including S3 Vector and OpenSearch vector indexes.
• Develop cloud infrastructure using CloudFormation, GitHub, Harness, and CI/CD pipelines while leveraging SNS, SQS, and EventBridge.
• Improve reliability, scalability, performance, and maintainability of enterprise data platforms through monitoring, troubleshooting, automation, and optimization.
• Lead modernization initiatives migrating legacy platforms to cloud-native AWS services.
• Mentor junior engineers and collaborate in an Agile environment to deliver high-quality data solutions.

Skills

Databricks

Apache Spark (PySpark)

Python

SQL (PostgreSQL)

Education

Bachelor's degree in Computer Science/Engineering or related field

Tools

AWS Glue

Amazon EMR

Amazon MWAA

AWS Lambda

AWS Step Functions

Amazon S3

Amazon Redshift

Amazon RDS

AWS DMS

Apache Iceberg

Parquet/ORC/Avro

Hive/OpenSearch

PostgreSQL/Oracle/GraphDB/NoSQL

Job description

Capital Technology Group is seeking a Senior Data Engineer to design, build, and maintain scalable data pipelines and analytics platforms in a federal-focused environment. You will partner with fellow data engineers to evaluate tools, prototype new technologies, and deliver secure, mission-driven data solutions on AWS.

Responsibilities include building ETL/ELT workflows, leveraging PySpark, Databricks, and SQL on PostgreSQL, while integrating diverse data sources and ensuring FedRAMP-aligned