Evaluate rules on our own again, or in tandem

[nilmerg]

Transferring SQL Queries is… sub-optimal. Let's get rid of that.

Version 0.1.0 expected sources to include extra_tags so that rules match events with corresponding recipients. We want to get back to this behavior, but we also want to avoid their previous shortcomings.

The properties of an event that are relevant in 0.2.0 are:

{
  "extra_tags": <object>,
  "rules_version": <hash>,
  "rule_ids": <list-of-ints>
}

These are fully obsolete now and only a single new one is required:

{
  "relations": <object>
}

Example, in case of Icinga DB:

{
  "relations": {
    "host": {
      "name": "web-1-a",
      "vars": {
        "os": "Linux"
      }
    },
    "hostgroups": [
      {
        "name": "web-servers",
        "display_name": "Web Servers"
      }
    ]
  }
}

The basic assumption here is that relations always have the same structure, so there is no variation requiring conditional checks. Let's take a look at a service why that matters:

{
  "relations": {
    "host": {
      "name": "web-1-a",
      "vars": {
        "os": "Linux"
      }
    },
    "hostgroups": [
      {
        "name": "web-servers",
        "display_name": "Web Servers"
      }
    ],
    "services": [
      {
        "name": "nginx",
        "vars": {
          "http_ports": [80, 443]
        }
      }
    ],
    "servicegroups": []
  }
}

Note that this simply contains an additional services and servicegroups key, services with just a single entry. In case of a service event, it is always a list with a single entry. It can simply be extended to a multiple of services in case of a host and the structure didn't change.

Icinga Notifications Web will serialize rule filters in a format easy to parse for Icinga Notifications. Though, if that's not the query string syntax already in use, Web will need to be able to parse it back as well in order to present it back to the user.

Upon retrieval of an event, Icinga Notifications tries to match all rules with it. If any rule cannot match due to missing attributes, they're collected in a list. If this happens, Icinga Notifications needs to contact the source and tell it about those and the source needs to re-transmit either the full event again or updated relations. (That's up to the implementer to decide.) This is called attribute negotiation.

Important

Generic sources are not meant to support attribute negotiation, so whether a source is capable of this must be signaled somehow to Icinga Notifications. A request header for example.

The source may signal that some relations are already complete and an update cannot be delivered for them. This is done by using a request header: X-Icinga-Complete-Relations: hostgroups, services[*].vars

If the source cannot (will not or is unable to) provide an update, or the the update is incomplete, Icinga Notifications evaluates the rules as far as it can but generates a log entry for rules that couldn't be evaluated.

Let's say Icinga DB missed to include the hostgroups and service custom variables. In this case, Icinga Notifications will ask it for:

• services[*].vars.http_ports
• hostgroups

And if a group did not contain the display name:

• hostgroups[*].display_name

The format used here is JSONpath. Its mature nature and widespread use should pose no problem for any source to understand.

The structure of said request and the expected response may look like this:

Request:

{
  "type": "attribute negotiation",
  "attributes": [
     "hostgroups",
     "services[*].vars.http_ports"
  ]
}

Response:

{
  "relations": {
    "hostgroups": [
      {
        "name": "web-servers",
        "display_name": "Web Servers"
      }
    ],
    "services": [
      {
        "vars": {
          "http_ports": [80, 443]
        }
      }
    ]
  }
}

[julianbrost]

What you're describing states nothing about ID tags. Would they remain as is and how would they interact with the relations you're describing here? You are adding a relation host where we already have host as an ID tag.

The basic assumption here is that relations always have the same structure

What exactly do you mean by that? In your examples, host is a JSON object and hostgroups a JSON list.

host [...] services

Looks inconsistent to me. You don't explicitly state this, but to me, this looks like you want to introduce the ability to filter hosts based on a "there exists a service on that host that" basis?

host.name~web*&host.vars.os=Linux|services[*].name=nginx&services[*].vars.http_ports[*]=443

So that filter syntax was gone in the meantime and we never had a pretty parser for that. What component(s) would need to be able to parse that filter? If the Go code needs to, I'd strongly prefer a trivial to parse JSON representation of that filter (which should be trivial to generate if that filter is already parsed on the PHP side, as that would more or less be a JSON-representation of the AST).

The format used here is JSONpath. Its mature nature and widespread use should pose no problem for any source to understand.

How much consideration did you put into JSONpath? I see a few potential problems for this use case: From my understanding, JSONpath is a format for selecting information from a JSON document. However, we don't already have a full JSON document here, but we want to know which information we should fetch somewhere else for populating a JSON document. So that raises the question how well this could be done with existing libraries without having to query everything and then filter it down, probably querying much information that isn't ever needed in the process.

I certainly don't want to start implementing JSONpath ourselves (which looks like a quite complex standard if you ask me) as we might need to dynamically query from SQL on the fly based on what's filtered for.

Additionally, I know what you want to convey with the example queries you're suggesting, however, when evaluating them, wouldn't the results look like this, so they lose the whole structure of the document?

• services[*].vars.http_ports

[80, 443]

• hostgroups

[{"name": "web-servers", "display_name": "Web Servers"}]

• hostgroups[*].display_name

"Web Servers"

Whereas you'd rather want something where you can put in all three selectors and receive the following result:

{
  "hostgroups": [
    {
      "name": "web-servers",
      "display_name": "Web Servers"
    }
  ],
  "services": [
    {
      "vars": {
        "http_ports": [80, 443]
      }
    }
  ]
}

[nilmerg]

What you're describing states nothing about ID tags. Would they remain as is and how would they interact with the relations you're describing here? You are adding a relation host where we already have host as an ID tag.

Why would I. They have nothing to do right now and never did, regarding rule filters. Yes, they remain and convey the same meaning as always: The object's unique identification.

What exactly do you mean by that? In your examples, host is a JSON object and hostgroups a JSON list.

This isn't about the structure between different relations, but between different objects. We have only a single filter for a given rule, in case of Icinga DB this filter must be able to match hosts and services. For this a unified schema is required and for Icinga DB the assumption is that it only transmits events for hosts and services, and in this case, there is never more than one host related but potentially multiple services. Basically, I'm repeating myself here. Not sure actually what else I need to explain…

The gist is, how is a single filter expression be able to match objects of different type with different relations? It's not without a highly flexible expression language, which evaluates filters at runtime. ((object.type == "host" ? object.name : object.host.name).matches("web.*") (CEL example).) But this poses another problem: How does Icinga Notifications identify now, what key is missing? (i.e. object.name or object.host or object.host.name.) Please think through this yourself before responding. It cost me the entire day, yesterday, to come up to the conclusion that this isn't a viable path. The solution to me is what I propose above.

So that filter syntax was gone in the meantime and we never had a pretty parser for that.

So it's gone? I thought it's still used given that escalation conditions use the same syntax. But of course, serializing a filter to JSON is by no means a problem.

How much consideration did you put into JSONpath?

The format used here is JSONpath. Its mature nature and widespread use should pose no problem for any source to understand.

I found already two go-libraries for this with a quick search. It's fairly straightforward compared to other solutions and it's designed for what we need it for: selecting information from a JSON document. And that's what relations are for Icinga Notifications.

Additionally, I know what you want to convey with the example queries you're suggesting, however, when evaluating them, wouldn't the results look like this, so they lose the whole structure of the document?

Forgive me, I've left out how the structure of this should look like and the expected response. Added above. Though, I've described it: (Rephrased slightly to avoid quoting an entire paragraph.)

Icinga Notifications needs to contact the source and tell it about the list of missing attributes and the source needs to re-transmit either the full event again or updated relations.

[nilmerg]

Well, to be honest, I'm currently about to define what this means for Icinga DB in detail and noticed a flaw in my proposal and the use of JSONPath only.

For attribute negotiation to reliably work, Icinga Notifications needs a way know whether a given attribute should be there or is not available. i.e. host.name and host.vars.os, respectively. JSONSchema might be the answer, but requires a more complex source setup.

[julianbrost]

Why would I. They have nothing to do right now and never did, regarding rule filters. Yes, they remain and convey the same meaning as always: The object's unique identification.

In 0.1, you could filter on them. So it's a question whether they can be used in filters as well.

((object.type == "host" ? object.name : object.host.name).matches("web.*") (CEL example).) But this poses another problem: How does Icinga Notifications identify now, what key is missing? (i.e. object.name or object.host or object.host.name.) Please think through this yourself before responding.

Only requesting the minimal amount of fields would probably require multiple round trips, though just requesting everything that could possible be needed in the filter expression, so object.type, object.name, and object.host.name in your example (and expecting something like "does not exist for this object") should work though?

So that filter syntax was gone in the meantime and we never had a pretty parser for that.

So it's gone? I thought it's still used given that escalation conditions use the same syntax. But of course, serializing a filter to JSON is by no means a problem.

You're right, it's not gone completely yet, but escalation conditions are very limited in comparison right now. It's also still the first implementation in use, which we wanted to redo (#41), but that went dormant after these expression were removed for object filters. But I would rather prefer abandoning that PR entirely and use something trivial to parse instead. (Or in other words: putting effort into implementing a parser for is a waste of time to me.)

and it's designed for what we need it for: selecting information from a JSON document

That's the point I made: we don't have a JSON document but a SQL database (and Redis, that indeed contains JSON, but in a format that's unsuitable for this use-case, you can't easily go from hosts to services or to custom vars) in Icinga DB. So if we take JSONpath as a hard requirement, we could end up with "query everything and throw away most of it" every time (depending of exact interface/capabilities of available libraries).

Forgive me, I've left out how the structure of this should look like and the expected response. Added above. Though, I've described it: (Rephrased slightly to avoid quoting an entire paragraph.)

Icinga Notifications needs to contact the source and tell it about the list of missing attributes and the source needs to re-transmit either the full event again or updated relations.

That's the question what the library provides. Assume that the $json contains the following JSON object:

  {
    "host": {
      "name": "web-1-a",
      "vars": {
        "os": "Linux"
      }
    },
    "hostgroups": [
      {
        "name": "web-servers",
        "display_name": "Web Servers"
      }
    ]
  }

What would you expect eval_jsonpath("hostgroups[*].display_name", $json) to return? If it was "Web Servers" or ["Web Servers"], that would be pretty useless, it would need to return something more like {"hostgroups":[{"display_name":"Web Servers"}]} to be useful.

Or in other words: using JSONpath probably won't be as easy as $json_to_send = eval_jsonpath($requested_filter, $all_relations_json).

For attribute negotiation to reliably work, Icinga Notifications needs a way know whether a given attribute should be there or is not available. i.e. host.name and host.vars.os, respectively. JSONSchema might be the answer, but requires a more complex source setup.

Why does it need to? Wouldn't is be good enough that if it gets told that the attribute doesn't exist?

[nilmerg]

That's the point I made: we don't have a JSON document but a SQL database (and Redis, that indeed contains JSON, but in a format that's unsuitable for this use-case, you can't easily go from hosts to services or to custom vars) in Icinga DB. So if we take JSONpath as a hard requirement, we could end up with "query everything and throw away most of it" every time (depending of exact interface/capabilities of available libraries).

You've mentioned already that you don't want to parse the filter query string syntax Icinga Web uses. Let me provide the previous example partially in JSON (not final):

{
  "type": "chain",
  "op": "&",
  "rules": [
    {
      "type": "condition",
      "column": "host.name"
      "op": "~",
      "value": "web*"
    },
    {
      "type": "condition",
      "column": "host.vars.os",
      "op": "=",
      "value": "Linux"
    }
  ]
}

Icinga Notifications still has to interpret host.name and host.vars.os. It's this reason why JSONPath. It has to be some standardized format. A format all sources have to abide to, as they are expected to provide relations as a JSON object. Of course it is also the format used when telling a source that a given attribute is required.

What would you expect eval_jsonpath("hostgroups[*].display_name", $json) to return? If it was "Web Servers" or ["Web Servers"], that would be pretty useless, it would need to return something more like {"hostgroups":[{"display_name":"Web Servers"}]} to be useful.

Given the condition hostgroups[*].display_name=Web Servers, I expect Icinga Notifications to traverse through the filter expression tree and evaluate conditions with the relations of an event. eval_jsonpath("hostgroups[*].display_name", $relations) returns ["Web Servers"] (which every library should return, [*] denotes an array after all…) and Icinga Notifications will verify that "Web Servers" is contained.

Why does it need to? Wouldn't is be good enough that if it gets told that the attribute doesn't exist?

And how does Icinga DB detect that a custom variable doesn't exist? Even with all custom variables already mapped to hosts and services in memory, this means that for a large amount of rule filters (given that custom variables are often used in filters) Icinga Notifications performs a needless round-trip to the very least. Even worse, Icinga DB issues a sql query each time.

Now consider that Icinga DB will include custom variables by default. Not every host or service has the same variable. How do we prevent said round-trip in this case? Icinga DB already includes all available custom variables, there is no need to ask for an update. At the same time, host.address might be missing and can be delivered as part of an update. This distinction must be possible. Of course, we may solve this ourselves with a request header (X-Icinga-Complete-Relations: host.vars, services[*].vars) or similar, but a JSONSchema would also allow Icinga Notifications Web to also show column suggestions and at least type validation as part of the filter configuration for generic sources. (Given that it's statically configured)

[nilmerg]

Dismissed the idea to use JSONSchema to verify relations. It added needless complexity with respect to setup and upgrade safety and the advantage for generic sources is negligible. The mentioned request header should suffice.