[Remote] Senior Security Observability Engineer (SIEM Data Pipeline)

Note: The job is a remote job and is open to candidates in USA. Prudent Technologies and Consulting, Inc. is seeking three Senior Security Observability Engineers to build and operationalize observability capabilities across the SIEM ecosystem. The role involves leading the design and implementation of integrated monitoring solutions, ensuring continuous visibility into system health, and enabling proactive fault detection and performance management.

Responsibilities

• Lead the architecture and implementation of a comprehensive observability strategy across the entire SIEM modernization ecosystem, spanning data pipeline layers (Cribl, Vector, NiFi), event transport (Event Hubs), intermediate storage (Blob), and multiple downstream platforms (Splunk, Snowflake, ADX, Log Analytics, Anvilogic)
• Design and build end-to-end telemetry and traceability for data events as they move across platforms, enabling real-time visibility into ingestion, transformation, routing, and storage processes
• Develop and maintain dashboards and alerting mechanisms to detect:
• Faults and failures (e.g., dropped messages, ingestion lags, retry loops)
• Latency or throughput bottlenecks across pipelines
• Schema mismatches or format errors
• Duplicate, delayed, or missing data
• Data quality anomalies at point of ingestion and final storage
• Instrument each pipeline component (e.g., Cribl workers, Vector agents, NiFi processors) with health and performance metrics, using native exporters, APIs, or custom collectors
• Ensure observability tooling is in place for Azure Event Hubs, including partition health, consumer group lag, and throttling events
• Monitor Blob storage utilization and access patterns to identify ingest failures, access permission issues, or object lifecycle gaps
• Implement and enforce correlation IDs or tracing metadata to follow data across systems and detect where in the pipeline an issue originates
• Integrate monitoring solutions with Grafana, Azure Monitor, and PowerBI to support multiple stakeholder needs (technical, operational, and executive-level views)
• Partner closely with Security Engineering, Platform Engineering, and Data Engineering to ensure observability insights are actionable and result in measurable improvements
• Automate reporting of SLO/SLA adherence for pipeline uptime, data integrity, and ingestion latency
• Design alert routing and severity classification, ensuring appropriate escalation workflows via systems such as PagerDuty, ServiceNow, or Microsoft Teams

Skills

• Experience in building and operationalizing observability capabilities across the SIEM ecosystem
• Proficiency in designing integrated monitoring solutions for tools like Cribl, Vector, Splunk, Snowflake, ADX, and Log Analytics
• Ability to lead the architecture and implementation of a comprehensive observability strategy across the entire SIEM modernization ecosystem
• Experience with data pipeline layers (Cribl, Vector, NiFi), event transport (Event Hubs), intermediate storage (Blob), and multiple downstream platforms (Splunk, Snowflake, ADX, Log Analytics, Anvilogic)
• Skills in designing and building end-to-end telemetry and traceability for data events
• Experience in developing and maintaining dashboards and alerting mechanisms to detect faults and failures
• Ability to instrument pipeline components with health and performance metrics
• Experience ensuring observability tooling is in place for Azure Event Hubs
• Skills in monitoring Blob storage utilization and access patterns
• Ability to implement and enforce correlation IDs or tracing metadata
• Experience integrating monitoring solutions with Grafana, Azure Monitor, and PowerBI
• Ability to partner closely with Security Engineering, Platform Engineering, and Data Engineering
• Skills in automating reporting of SLO/SLA adherence
• Experience in designing alert routing and severity classification

Company Overview

• For over 26+ years Prudent Technologies & Consulting has been helping customers secure the technical and functional resources needed to deliver mission-critical IT & Business initiatives. It was founded in 1998, and is headquartered in Dallas, Texas, USA, with a workforce of 201-500 employees. Its website is https://www.prudentconsulting.com.

Company H1B Sponsorship

• Prudent Technologies and Consulting, Inc. has a track record of offering H1B sponsorships, with 8 in 2025, 3 in 2024, 11 in 2023, 4 in 2022, 7 in 2021, 10 in 2020. Please note that this does not guarantee sponsorship for this specific role.

Apply tot his job Apply To this Job