Dynamic backends may get created while we transition a VCL to the WARM
temperature: Because we first set the new temperature and then post the event,
backends might get created when the temperature is already WARM, but before the
VCL_EVENT_WARM gets posted, which leads to double warming. This can be
demonstrated with an appropriate delay during vcl_send_event().

The solution to this problem is simple: Only post WARM events to backends from
before the start of the transition.

In code, it looks a little more involved, but it is not complicated:

Under the vcl mutex, we take all directors onto a separate "cold" list and
change the temperature, and send the warm event only to the "cold" directors.
When all are warm, we concatenate.

To undo, we basically do the inverse: Save the warm directors, put the cold ones
back, send a cold event to all warm ones and concatenate again.

Fixes varnishcache#4199

Dynamic backends may get created while we transition a VCL to the WARM
temperature: Because we first set the new temperature and then post the event,
backends might get created when the temperature is already WARM, but before the
VCL_EVENT_WARM gets posted, which leads to double warming. This can be
demonstrated with an appropriate delay during vcl_send_event().

The solution to this problem is simple: Only post WARM events to backends from
before the start of the transition.

In code, it looks a little more involved, but it is not complicated:

Under the vcl mutex, we take all directors onto a separate "cold" list and
change the temperature, and send the warm event only to the "cold" directors.
When all are warm, we concatenate.

To undo, we basically do the inverse: Save the warm directors, put the cold ones
back, send a cold event to all warm ones and concatenate again.

Fixes varnishcache#4199

Dynamic backends may get created while we transition a VCL to the WARM
temperature: Because we first set the new temperature and then post the event,
backends might get created when the temperature is already WARM, but before the
VCL_EVENT_WARM gets posted, which leads to double warming. This can be
demonstrated with an appropriate delay during vcl_send_event().

The solution to this problem is simple: Only post WARM events to backends from
before the start of the transition.

In code, it looks a little more involved, but it is not complicated:

Under the vcl mutex, we take all directors onto a separate "cold" list and
change the temperature, and send the warm event only to the "cold" directors.
When all are warm, we concatenate.

To undo, we basically do the inverse: Save the warm directors, put the cold ones
back, send a cold event to all warm ones and concatenate again.

Fixes varnishcache#4199

The context for this change is varnishcache#4142, but while it builds upon it, it is still
causally unrelated.

To motivate this change, let's look at vcl_set_state() from before, and consider
some non-issues and issues:

transition to COLD
------------------

                        Lck_Lock(&vcl_mtx);
                        vcl->temp = VTAILQ_EMPTY(&vcl->ref_list) ?
                            VCL_TEMP_COLD : VCL_TEMP_COOLING;
                        Lck_Unlock(&vcl_mtx);
			// *A*
                        vcl_BackendEvent(vcl, VCL_EVENT_COLD);

	non-issues:

	At point *A*, backends could get added. For VCL_TEMP_COOLING, this fails
	(see VRT_AddDirector()). For VCL_TEMP_COLD, the state is consistent,
	because backends get created cold and that's it.

	At point *A*, backends could get removed. vdire_resign() from varnishcache#4142
	ensures that the temperature passed to vcldir_retire() is consistent
	with the events the backends have received.

transition to WARM (success)
----------------------------

                        Lck_Lock(&vcl_mtx);
                        vcl->temp = VCL_TEMP_WARM;
                        Lck_Unlock(&vcl_mtx);
			// *B* ...
                        i = vcl_send_event(vcl, VCL_EVENT_WARM, msg);
                        if (i == 0) {
				// ... *B*
                                vcl_BackendEvent(vcl, VCL_EVENT_WARM);
                                break;
                        }

	issues:

	(1)

	In region *B*, backends could get removed. They will not have received a
	WARM event, but will be called with a WARM temperature, so they will
	receive a bogus COLD event.

	(2)

	Also in region *B*, backends could get added. They will implicitly
	receive a WARM event (see VDI_event() in VRT_AddDirector()), and then
	another from vcl_BackendEvent()

transition to WARM (failed)
---------------------------

                        AZ(vcl_send_event(vcl, VCL_EVENT_COLD, &nomsg));
                        AZ(nomsg);
                        vcl->temp = VCL_TEMP_COLD;

	issues:

	(3)

	Backends added in region *B* will have received the implicit WARM event,
	and thus need a COLD event for the "cancel".

To solve these issues, we need to do two things:

There needs to be some kind of transaction which begins with the temperature
change and ends when all backends have been notified appropriately. Backends can
not get deleted while the transaction is in progress.

We need a notion of "backends from before the temperature change" and "backends
from after".

The first part is already delivered by varnishcache#4142: The vdire facility already
provides the transaction during which backends do not actually get deleted and
it ensures that the right temperature gets passed when the deletion is carried
out. So for this part, we only need to use vdire.

But issues (2) and (3) are not yet covered. For these, we add a checkpoint, such
that we know which directors from the list are the "base" from before the
temperature change and which are the "delta" added after it.

That's this patch.

vdire_start_event() atomically (under the vcl_mtx) sets the checkpoint and the
new temperature.

vdire_end_event() just uses the existing vdire_end_iter() and clears the
checkpoint.

vcl_BackendEvent() gets split into two:

backend_event_base() notifies backends from before the checkpoint.
backend_event_delta() atomically sets a new temperature and notifies backends
from after the checkpoint (but not from after its temperature change).

Fixes varnishcache#4199

The context for this change is varnishcache#4142, but while it builds upon it, it is still
causally unrelated.

To motivate this change, let's look at vcl_set_state() from before, and consider
some non-issues and issues:

transition to COLD
------------------

                        Lck_Lock(&vcl_mtx);
                        vcl->temp = VTAILQ_EMPTY(&vcl->ref_list) ?
                            VCL_TEMP_COLD : VCL_TEMP_COOLING;
                        Lck_Unlock(&vcl_mtx);
			// *A*
                        vcl_BackendEvent(vcl, VCL_EVENT_COLD);

	non-issues:

	At point *A*, backends could get added. For VCL_TEMP_COOLING, this fails
	(see VRT_AddDirector()). For VCL_TEMP_COLD, the state is consistent,
	because backends get created cold and that's it.

	At point *A*, backends could get removed. vdire_resign() from varnishcache#4142
	ensures that the temperature passed to vcldir_retire() is consistent
	with the events the backends have received.

transition to WARM (success)
----------------------------

                        Lck_Lock(&vcl_mtx);
                        vcl->temp = VCL_TEMP_WARM;
                        Lck_Unlock(&vcl_mtx);
			// *B* ...
                        i = vcl_send_event(vcl, VCL_EVENT_WARM, msg);
                        if (i == 0) {
				// ... *B*
                                vcl_BackendEvent(vcl, VCL_EVENT_WARM);
                                break;
                        }

	issues:

	(1)

	In region *B*, backends could get removed. They will not have received a
	WARM event, but will be called with a WARM temperature, so they will
	receive a bogus COLD event.

	(2)

	Also in region *B*, backends could get added. They will implicitly
	receive a WARM event (see VDI_event() in VRT_AddDirector()), and then
	another from vcl_BackendEvent()

transition to WARM (failed)
---------------------------

                        AZ(vcl_send_event(vcl, VCL_EVENT_COLD, &nomsg));
                        AZ(nomsg);
                        vcl->temp = VCL_TEMP_COLD;

	issues:

	(3)

	Backends added in region *B* will have received the implicit WARM event,
	and thus need a COLD event for the "cancel".

To solve these issues, we need to do two things:

There needs to be some kind of transaction which begins with the temperature
change and ends when all backends have been notified appropriately. Backends can
not get deleted while the transaction is in progress.

We need a notion of "backends from before the temperature change" and "backends
from after".

The first part is already delivered by varnishcache#4142: The vdire facility already
provides the transaction during which backends do not actually get deleted and
it ensures that the right temperature gets passed when the deletion is carried
out. So for this part, we only need to use vdire.

But issues (2) and (3) are not yet covered. For these, we add a checkpoint, such
that we know which directors from the list are the "base" from before the
temperature change and which are the "delta" added after it.

That's this patch.

vdire_start_event() atomically (under the vcl_mtx) sets the checkpoint and the
new temperature.

vdire_end_event() just uses the existing vdire_end_iter() and clears the
checkpoint.

vcl_BackendEvent() gets split into two:

backend_event_base() notifies backends from before the checkpoint.
backend_event_delta() atomically sets a new temperature and notifies backends
from after the checkpoint (but not from after its temperature change).

Fixes varnishcache#4199

The context for this change is varnishcache#4142, but while it builds upon it, it is still
causally unrelated.

To motivate this change, let's look at vcl_set_state() from before, and consider
some non-issues and issues:

transition to COLD
------------------

                        Lck_Lock(&vcl_mtx);
                        vcl->temp = VTAILQ_EMPTY(&vcl->ref_list) ?
                            VCL_TEMP_COLD : VCL_TEMP_COOLING;
                        Lck_Unlock(&vcl_mtx);
			// *A*
                        vcl_BackendEvent(vcl, VCL_EVENT_COLD);

	non-issues:

	At point *A*, backends could get added. For VCL_TEMP_COOLING, this fails
	(see VRT_AddDirector()). For VCL_TEMP_COLD, the state is consistent,
	because backends get created cold and that's it.

	At point *A*, backends could get removed. vdire_resign() from varnishcache#4142
	ensures that the temperature passed to vcldir_retire() is consistent
	with the events the backends have received.

transition to WARM (success)
----------------------------

                        Lck_Lock(&vcl_mtx);
                        vcl->temp = VCL_TEMP_WARM;
                        Lck_Unlock(&vcl_mtx);
			// *B* ...
                        i = vcl_send_event(vcl, VCL_EVENT_WARM, msg);
                        if (i == 0) {
				// ... *B*
                                vcl_BackendEvent(vcl, VCL_EVENT_WARM);
                                break;
                        }

	issues:

	(1)

	In region *B*, backends could get removed. They will not have received a
	WARM event, but will be called with a WARM temperature, so they will
	receive a bogus COLD event.

	(2)

	Also in region *B*, backends could get added. They will implicitly
	receive a WARM event (see VDI_event() in VRT_AddDirector()), and then
	another from vcl_BackendEvent()

transition to WARM (failed)
---------------------------

                        AZ(vcl_send_event(vcl, VCL_EVENT_COLD, &nomsg));
                        AZ(nomsg);
                        vcl->temp = VCL_TEMP_COLD;

	issues:

	(3)

	Backends added in region *B* will have received the implicit WARM event,
	and thus need a COLD event for the "cancel".

To solve these issues, we need to do two things:

There needs to be some kind of transaction which begins with the temperature
change and ends when all backends have been notified appropriately. Backends can
not get deleted while the transaction is in progress.

We need a notion of "backends from before the temperature change" and "backends
from after".

The first part is already delivered by varnishcache#4142: The vdire facility already
provides the transaction during which backends do not actually get deleted and
it ensures that the right temperature gets passed when the deletion is carried
out. So for this part, we only need to use vdire.

But issues (2) and (3) are not yet covered. For these, we add a checkpoint, such
that we know which directors from the list are the "base" from before the
temperature change and which are the "delta" added after it.

That's this patch.

vdire_start_event() atomically (under the vcl_mtx) sets the checkpoint and the
new temperature.

vdire_end_event() just uses the existing vdire_end_iter() and clears the
checkpoint.

vcl_BackendEvent() gets split into two:

backend_event_base() notifies backends from before the checkpoint.
backend_event_delta() atomically sets a new temperature and notifies backends
from after the checkpoint (but not from after its temperature change).

Fixes varnishcache#4199

The context for this change is varnishcache#4142, but while it builds upon it, it is still
causally unrelated.

To motivate this change, let's look at vcl_set_state() from before, and consider
some non-issues and issues:

transition to COLD
------------------

                        Lck_Lock(&vcl_mtx);
                        vcl->temp = VTAILQ_EMPTY(&vcl->ref_list) ?
                            VCL_TEMP_COLD : VCL_TEMP_COOLING;
                        Lck_Unlock(&vcl_mtx);
			// *A*
                        vcl_BackendEvent(vcl, VCL_EVENT_COLD);

	non-issues:

	At point *A*, backends could get added. For VCL_TEMP_COOLING, this fails
	(see VRT_AddDirector()). For VCL_TEMP_COLD, the state is consistent,
	because backends get created cold and that's it.

	At point *A*, backends could get removed. vdire_resign() from varnishcache#4142
	ensures that the temperature passed to vcldir_retire() is consistent
	with the events the backends have received.

transition to WARM (success)
----------------------------

                        Lck_Lock(&vcl_mtx);
                        vcl->temp = VCL_TEMP_WARM;
                        Lck_Unlock(&vcl_mtx);
			// *B* ...
                        i = vcl_send_event(vcl, VCL_EVENT_WARM, msg);
                        if (i == 0) {
				// ... *B*
                                vcl_BackendEvent(vcl, VCL_EVENT_WARM);
                                break;
                        }

	issues:

	(1)

	In region *B*, backends could get removed. They will not have received a
	WARM event, but will be called with a WARM temperature, so they will
	receive a bogus COLD event.

	(2)

	Also in region *B*, backends could get added. They will implicitly
	receive a WARM event (see VDI_event() in VRT_AddDirector()), and then
	another from vcl_BackendEvent()

transition to WARM (failed)
---------------------------

                        AZ(vcl_send_event(vcl, VCL_EVENT_COLD, &nomsg));
                        AZ(nomsg);
                        vcl->temp = VCL_TEMP_COLD;

	issues:

	(3)

	Backends added in region *B* will have received the implicit WARM event,
	and thus need a COLD event for the "cancel".

To solve these issues, we need to do two things:

There needs to be some kind of transaction which begins with the temperature
change and ends when all backends have been notified appropriately. Backends can
not get deleted while the transaction is in progress.

We need a notion of "backends from before the temperature change" and "backends
from after".

The first part is already delivered by varnishcache#4142: The vdire facility already
provides the transaction during which backends do not actually get deleted and
it ensures that the right temperature gets passed when the deletion is carried
out. So for this part, we only need to use vdire.

But issues (2) and (3) are not yet covered. For these, we add a checkpoint, such
that we know which directors from the list are the "base" from before the
temperature change and which are the "delta" added after it.

That's this patch.

vdire_start_event() atomically (under the vcl_mtx) sets the checkpoint and the
new temperature.

vdire_end_event() just uses the existing vdire_end_iter() and clears the
checkpoint.

vcl_BackendEvent() gets split into two:

backend_event_base() notifies backends from before the checkpoint.
backend_event_delta() atomically sets a new temperature and notifies backends
from after the checkpoint (but not from after its temperature change).

Fixes varnishcache#4199