Senior Security Data Engineer

Clearwater; India; On-site; INR 3,340,000 - 4,294,000; Full time

Job summary

Qualifications

• Proficiency in Python for ETL and SQL for querying.
• Experience with streaming tech: Kafka, Pub/Sub.
• Mastery of log parsing: Regex, Syslog, JSON, CEF.

Responsibilities

• Pipeline Management: Maintain high-throughput streaming pipelines for logs from varied sources.
• Log Normalization: Write parsers to standard schemas (OCSF/ECS).
• Cost Optimization: Route high-value data to SIEM, bulk data to Data Lake.
• Data Preparation: Clean and structure data for AI/ML models.

Skills

Python

SQL

Kafka

Pub/Sub

Regex

Syslog

CEF

JSON

Data Lake

Spark

Flink

Education

Bachelor's degree in CS/EE

Tools

Vector Databases

Log Observability Tools

Parquet/Avro

Spark

Flink

Job description

• Pipeline Management: Maintain high-throughput streaming pipelines to ingest logs from various sources (Firewalls, Cloud, Endpoints) to a central destination.
• Log Normalization: Write parsers to convert raw, messy logs into standard schemas (e.g., OCSF or ECS) for consistent querying.
• Cost Optimization: Implement routing logic to send "high-value" data to the SIEM and "bulk" data to low-cost Object Storage (Data Lake).
• Data Preparation: Clean and structure data to enable AI/ML detection models and advanced analytics.

Specific Job Responsibilities

• Pipeline Management: Maintain high-throughput streaming pipelines to ingest logs from various sources (Firewalls, Cloud, Endpoints) to a central destination.
• Log Normalization: Write parsers to convert raw, messy logs into standard schemas (e.g., OCSF or ECS) for consistent querying.
• Cost Optimization: Implement routing logic to send "high-value" data to the SIEM and "bulk" data to low-cost Object Storage (Data Lake).
• Data Preparation: Clean and structure data to enable AI/ML detection models and advanced analytics.

Experience Required

• Data Engineering: Proficiency in Python (for ETL) and SQL (for complex querying).
• Streaming Tech: Experience with Message Queues (e.g., Kafka, Pub/Sub) and stream processing concepts.
• Log Handling: Mastery of Regex and log parsing strategies for standard formats (Syslog, CEF, JSON).
• Storage Architecture: Understanding of Data Lake principles (Parquet/Avro formats) vs. Data Warehouses.

Qualifications, Skills, & Knowledge

• Experience with Vector Databases for storing embeddings.
• Knowledge of Log Observability/Routing tools (middleware that routes logs).
• Familiarity with Big Data frameworks (e.g., Spark, Flink).
• PROFESSIONAL DEVELOPMENT EXPECTATIONS

Ability to embrace Clearwater's CLEAR core values (Commitment to Client Success, Lead with Accountability, Integrity & Collaboration, Excellence in All That We Do, Advance Colleague Success, Respect & Transparency) and culture.

The base salary range for this role is 35,000- 45,000]. Base salary is part of our total rewards package which also includes the opportunity for merit-based salary increases, eligibility for our 401(k) plan, medical, dental, vision, life and disability insurances and leaves provided in line with your work state. Our robust time-off policy includes flexible paid time off, 11 paid holidays, and paid sick time. Total compensation, including base salary to be offered, will depend on elements unique to each candidate, including applicable candidate experience, skills, education and other factors permitted by law.

Disclaimer: The above statements are intended to describe the general nature and level of work being performed by people assigned to this position. They are not to be construed as an exhaustive list of all responsibilities, duties, and skills required of colleagues in the role. All colleagues may be required to perform duties outside of their normal responsibilities from time to time, as needed.

Clearwater is committed to providing reasonable accommodations for qualified individuals with disabilities and disabled veterans in our job application procedures. Please inform Clearwater/Redspin’s Recruiting team if you need any assistance completing any forms or to otherwise participating in the application process.

Mental/Physical Requirements: Fast paced environment handling multiple demands. Must be able to exercise appropriate judgment as necessary. Requires a high level of initiative and independence. Excellent written and oral communication skills required. Requires the ability to use a personal computer for extended periods of time.

Salary Description

[update per job]

Disclaimer: The above statements are intended to describe the general nature and level of work being performed by people assigned to this position. They are not to be construed as an exhaustive list of all responsibilities, duties, and skills required of colleagues in the role. All colleagues may be required to perform duties outside of their normal responsibilities from time to time, as needed.

Clearwater is committed to providing reasonable accommodations for qualified individuals with disabilities and disabled veterans in our job application procedures. Please inform Clearwater/Redspin’s Recruiting team if you need any assistance completing any forms or to otherwise participating in the application process.

Mental/Physical Requirements: Fast paced environment handling multiple demands. Must be able to exercise appropriate judgment as necessary. Requires a high level of initiative and independence. Excellent written and oral communication skills required. Requires the ability to use a personal computer for extended periods of time.