nit: TBD - cleanup info logs before merge.

There should be retry with exp. backoff. Esp. when CreateTags calls are throttled by AWS.
That probably implies a queue of PVs.

We don't have do this. If we are using factory.WithInformers, we will reconcile this within Sync function.

Hey @gnufied I was bit confused with this!! basically I don't want to run reconciliation on every change on InfraStructure resource. I need to run reconciliation only if there is a any change in infra.Status.PlatformStatus.AWS.ResourceTags.

Can you suggest a better way to do this so that we can remove the unnecessarily computes.

How will this work when controller is restarted? You are also doing WithInformers below and hence any change in infra object will still trigger Sync.

So - most bulletproof way of ensuring that, we don't unnecessarily process all PVs is to store the information that we have processed these PVs somewhere in a persistent way.

So, what we have currently is worst of both the worlds. If I were to design this, I will probably make a hash of sorted tags and annotate PV with tag hash. If tag hash annotation in PV and computed tag hash don't change, then I will not update the PV or else I will.

How will this work when controller is restarted? You are also doing WithInformers below and hence any change in infra object will still trigger Sync.

What I am thinking is on every restart I would like to run a sync function and update the volumes if there is a change and further On each change in resource-tags we would again run the sync.

So, what we have currently is worst of both the worlds. If I were to design this, I will probably make a hash of sorted tags and annotate PV with tag hash. If tag hash annotation in PV and computed tag hash don't change, then I will not update the PV or else I will.

This looks easy way to manage this!! I will then add a new field in controller struct which will have the updated hash for the sorted map of tags! now in each reconciliation I will update tags only if the hash is different from the other one? Does this sounds better to you??

But that is not what I said. I said, we should store hash of sorted tags in PV objects as annotation and compare those with current tags we are about to apply. We should only apply tags with AWS if hashes change.

Storing them just in-memory doesn't help us much.

Added Thanks For Suggestion!!

So, does this controller needs to be an opt-in or a default controller? I don't assume every OCP customer wants this feature and if tagging were to fail after OCP upgrade, their clusters will be degraded and we will have support nightmare.

cc @jsafrane

It is presented as enabled by default + no opt out for all HyperShift clusters in the enhancement.

I am not sure if we need this function at all.

So the reason I brought this change is I would like to retry the the batches which have failed to update tags!! Here I would like to update tags in a serial order(one By one) Discussuion link for the volumes. I cannot use the similar sync function here or else you could suggest a better way for this!!

If we are going to per-pv hash, then we are not going to need this right?

The points I want to make sure here are

1. I definitely need to batch volumes in order to not to hit throttling condition.
2. Now on failure we would need to retry!!
3. Since we are using batch APIs, so AWS SDK's APIs will give us error if any one of the VolumeID in the batch hits the any error(May be validation, Auth, Permission etc), All the VolumeIDs in that batch will not be able to update the tags! In this condition I would like to add a worker queue that will handle the the serial update of PV's tags and will retry to update the tags in exponential back-off time-period.

Since We need to figure out the trade-off between the either in first place we should update all PVs using AWS API's in serial order and retry using similar sync function or should we use Batch Volumes to be called in the sync Functions and later retry should be processed with the queue function serially until the queue is empty!!

I definitely need to batch volumes in order to not to hit throttling condition.

That is fine. You are already batching PVs in fetchPVsAndUpdateTags.

Now on failure we would need to retry!!

We are unnecessarily complicating this. The controller is going to resync every 20 minutes anyways, so it will try to tag all the PVs which aren't tagged. So do we even need to keep separate queue for failed PVs? I am also afraid that, your failed worker queue is going to race with normal controller resync.

What is the point of separate failed worker queue when every 20 minutes, we are going to try and sync tag for all PVs which doesn't have matching tag-hash? If you really want a separate worker queue, you will have to redesign the whole thing, so as at least they are not racy. But I don't really understand point of doing it.

Hey @gnufied I completely agree that this retrying the failure is unnecessarily making the changes complicated.

@jsafrane I had a chat with @TrilokGeer regarding this, IMHO we should drop the idea to add degraded condition on tags update failure. As the cluster should not be degraded for failure to apply Tags. as the sync will anyhow retry to tag the volumes withing resync period, in this way we will not immediately require to retry the failures and can remove this queue worker.
Also anyways we are emitting the warning events from the this controller if the tags update is failed. so user will know that the tags update has been failed due the the certain reason also can think of alerts based on that.

/cc @TrilokGeer
Can you also put your views on this!!

BTW are we planning to backport this PR to older releases?

@gnufied I guess we would need to backport this to 4.17. Slack Thread.

So I noticed that you removed failed worker logic. The thing is - I was talking to @jsafrane offline and he has me convinced that, we do need some kind of additional logic so as we can try retagging of "failed" PVs one-by-one, rather than in a batch. This will ensure that, one bad apple in a batch doesn't prevent tagging of rest of the PVs.

But - we need to be careful when doing this.

• We should make sure that, PVs which will be tagged via failed worker, doesn't get processed via regular controller resync (so no race).
• I would move the entire failed worker code in a separate file.

Ack;
I believe we can remove the ResyncEvery parameter from the controller builder as this is a overkill for us why do we want to run the sync function in every 20 mins if nothing has changed in the resource Tags.

Alternatively, If you really think resync is important here then, I think that we can add the handle the race condition by using another annotation in PVs for Tagging status and filtering based on tagshash and status in the annotation. But this will also cost us some volume Update calls and further need to handle cases where we are not able to update the status within batches. WDYT?

Can you create a type for these, to prevent someone from setting updateType = "foobar" where behavior is undefined? Something like the following:

type UpdateType string

const (
    updateTypeBatch UpdateType = "batch"
    updateTypeIndividual UpdateType = "individual"
)

type pvUpdateQueueItem struct {
	updateType UpdateType
	pvNames    []string
}

Please first check my other comments though, I wonder if we really need updateType.

Broader question: do we really need to separate out the batch and individual update types? I could be missing something, but the individual case looks mostly the same as the batch case, just with an array length of 1. Can this function handle both cases with out an if/else for updateType?

If there are special cases for batch updates, you could use len(item.pvNames) > 1 the same as item.updateType == "batch". But it would be better to minimize those special cases and handle an array length of 1 the same as >1 as a general rule.

I was thinking of doing that in the same way but it makes the code a bit messy and a bit hard to understand.
putting updateType clarifies what kind of operation is expected to happen where In batch operation there can be a batch that will have only 1 volume.
Putting some points down:

1. AWS errors are not very detailed. In a batch operation, they will return the first error message let's say in a batch operation: if the nth volume ID does not exist, it will return an error for that volume only maybe n+xth volume might give some different error message. which we are not sure of.
2. Better readability of code.
3. Better Events can be published with error messages in case of batch and individual updates.
4. There can be cases where the user has again changed the infrastructure tags to previous tags, then we would not need to make more requests for a particular PV. IMHO including them within the same function will make it a bit confusing

I just want to put my reasoning for this field else I am open to refactoring it in the same way as you suggested.
WDYT?

Ok, thanks for explaining, I'm okay with keeping the update types as you have them.

One more idea to simplify this function a bit: would it be possible to refactor the code inside the if / else block into separate methods? So that processVolumes could just have the common code and then call the right method, instead of nesting the code inside the if / else?

switch item.updateType {
    case updateTypeBatch:
        c.processBatchVolumes(ctx, item, infra, ec2Client)
    case updateTypeIndividual:
        c.processIndividualVolumes(ctx, item, infra, ec2Client)
    default:
        panic("invalid update type: %v", item.updateType)
}

Thanks; I have added the same.