Skip to content

Optimize cleaner run time #6815

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Open
wants to merge 2 commits into
base: master
Choose a base branch
from

Conversation

alexqyle
Copy link
Contributor

What this PR does:

Two optimization being made for cleaner:

  • Use non-block way to push cleaner jobs to active user channel and delete user channel. So that long run time deletion job would not block cleaning job in the next cycle.
  • Introduced concurrent way to do DeletePrefix. So it is faster to delete directory with large number of files. Tested with 1k files in one directory. Sequential deletion took 17s and Concurrent deletion took 1.3s

Which issue(s) this PR fixes:

Checklist

  • Tests updated
  • Documentation added
  • CHANGELOG.md updated - the order of entries should be [CHANGE], [FEATURE], [ENHANCEMENT], [BUGFIX]

@danielblando
Copy link
Contributor

We can add an ENHANCEMENT in the changelog. Good to know what is being deployed in a new rev

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Development

Successfully merging this pull request may close these issues.

3 participants