Data Engineer: AWS OpenSearch & Real-Time Data Pipelines

Incedo Inc.; Fort Mill (SC); On-site; USD 110,000 - 160,000; Full time

Job summary

Incedo Inc. is seeking an experienced Data Engineer with strong AWS OpenSearch expertise to design and maintain scalable data pipelines and search solutions.

The role emphasizes Python, SQL, and cloud-native architectures to support analytics and real-time search across multiple business domains. You will build and optimize OpenSearch indexes, integrate with downstream systems, and implement IaC while collaborating with data science and analytics teams.

Qualifications

• 5+ years of experience as a Data Engineer.
• Strong experience with AWS OpenSearch (formerly Amazon Elasticsearch Service).
• Strong SQL and Python programming skills.
• Experience with AWS services: OpenSearch, S3, Glue, Lambda, IAM, CloudWatch, EC2, EMR, Step Functions.
• Experience building ETL/ELT pipelines.
• Experience with Docker and Kubernetes (preferred).
• Knowledge of distributed search architecture and query optimization.
• Familiarity with IaC (Terraform or CloudFormation) and CI/CD.

Responsibilities

• Design, develop, and maintain scalable data pipelines using AWS services.
• Build and optimize AWS OpenSearch clusters for indexing, search, and analytics.
• Develop ETL/ELT pipelines using Python, SQL, and AWS services such as Glue, Lambda, and EMR.
• Design and maintain OpenSearch indexes, mappings, analyzers, and query performance.
• Integrate OpenSearch with upstream and downstream applications for real-time search and analytics.
• Work with structured and semi-structured data from multiple data sources.
• Optimize OpenSearch cluster performance, shard allocation, indexing, and query execution.
• Implement Infrastructure as Code (Terraform or CloudFormation) for AWS resources.
• Build monitoring and alerting using CloudWatch, Grafana, or Prometheus.
• Collaborate with Data Scientists, Data Analysts, and application teams to support data and search requirements.
• Implement security best practices, IAM policies, encryption, and access controls.
• Troubleshoot production issues and optimize system performance.

Skills

AWS OpenSearch

SQL

Python

ETL/ELT pipelines

Data Pipelines

CI/CD

Security IAM

Cloud architecture

Distributed systems

Agile

Tools

OpenSearch

Docker

Kubernetes

Terraform

CloudWatch

Grafana

Prometheus

Git

CI/CD

Spark/PySpark

Kafka/MSK

Snowflake/Redshift/Databricks

Job description

Incedo Inc. is seeking an experienced Data Engineer with strong AWS OpenSearch expertise to design and maintain scalable data pipelines and search solutions.

The role emphasizes Python, SQL, and cloud-native architectures to support analytics and real-time search across multiple business domains. You will build and optimize OpenSearch indexes, integrate with downstream systems, and implement IaC while collaborating with data science and analytics teams.