CA-404020: Do not fail when removing a non-existing datasource

[psafont]

The latest changes in the metrics daemon changes how archived and live metrics
are synchronised, and archived sources are created less often. This meant that
on some cases, like SR.forget, the operations failed because they could query
for existing sources, but the call to remove them failed, because the metrics
only exist in live form, not archived one.

(what happens with the live one, do they disappear when the SR is forgotten?)

Signed-off-by: Pau Ruiz Safont [email protected]

[gangj]

I think we don't need to sort here? Then still use Utils.setify ds_names?

[psafont]

We want to drop setify in as many places as possible from the codebase, it has a cost of O(n^2). sort_uniq should be faster in most cases. Since we don't need to maintain order of the elements the latter can be used

[edwintorok]

If we want to avoid O(n^2) behaviour then we could use List.sort instead of List.sort_uniq, and see how many consecutive equal elements we find: those are the duplicates that we need to remove.
List.filter_map + List.filter is still O(n^2).

[gangj]

Would you please share how archived and live metrics are synchronized? With this fix, is it possible for some entry to be synchronized from live to archived after an attempt to remove it by rrd_remove_ds? And is this a valid case?

[psafont]

Would you please share how archived and live metrics are synchronized?

The change is about datasources, not metrics. Archived datasources hold metadata when xapi asks to disable or enable a datasource. This bit of information is not stored in the datastructures of the live datasource, and its lifetime is also independent, which means it can't be stored easily in the live datastructure because the former needs to survive the latter when the datasource stops reporting.

[psafont]

With this fix, is it possible for some entry to be synchronized from live to archived after an attempt to remove it by rrd_remove_ds?

Remove can be called at any time, so data races are possible, theoretically. In practice I haven't observed a mechanism that synchronises live datasources with archival / default ones.

Datasources about SRs are in a specially awkward place because they are reported currently as hosts metrics. This means that once they exist as live datasources, they are never removed from memory, because there's no mechanism for that. There's a mechanism to manage SR datasources independently, and there's a lot of code for it, but it's currently unused. VM do not have this problem.

[edwintorok]

The optimization discussed to avoid List.filter + List.filter_map could be continued in another PR, this PR is already an improvement over what was there before.

[Vincent-lau]

The problem here, as @psafont suggested, is that query_possible_dss looks at the live and disabled datasources, and look for relevant ones to delete. In this particular case, it is avgqu_sz_296626b4 rrd, which turns out to be a live one. While the forget_host_ds removes a datasource by name in the disabled/archived datasources, i.e. the rrdi.rrd.rrd_dss, and ignoring all the live datasources, i.e. rrdi.dss. I am not sure why this is behaviour is like this, but if what we meant by forget_host_ds is to forget all the ds that is archived, then this PR would be good.

[psafont]

That's a limitation of the current design, and needs careful readjusting of all the SR-related metrics: all the names need to be changed (do multiple hosts report about the same SR?). I would avoid trying to fix it here, as it adds quite a lot of risk in the change.