Search does not prioritize exact match when package name contains numbers

[di]

For example, a search for python-miio has the python2-miio as the first result, likely because the 2 is being tokenized out of the query.

[gsb-eng]

@di Any progress on this? If not, Can I pick this?

[brainwane]

@gsb-eng Yes, please go ahead and pick it up! At least within Warehouse, our rule is: If there is not a corresponding pull request for this issue (visibly linked by issue number/PR number), and it's not marked as assigned to anyone, it is up for grabs. So in cases like this, you don't even need to ask. :)

As always, if you have questions along the way as you work on this, please feel free to ask them here, in #pypa-dev on Freenode, or the pypa-dev mailing list. Thank you!

[waseem18]

@di Could you please help me in providing me more information about this issue?

When I searched for python2-miio the results on my local environment were completely different and irrelevant than that of pypy.org

[di]

@waseem18 Unfortunately I don't know much more than you here. We got a report that packages with a number in them could be prioritized higher in search than a package with the exact name as the search query.

However looking at https://pypi.org/search/?q=python-miio now, it seems like python-miio has the highest rank now. We will likely need to do some debugging of the elasticsearch query and the results to see why it (at some point) was possible.

[waseem18]

Yeah - I was feeling the same. Because now the search queries seems to work as expected.

Will do some debugging and check.

[di]

Here's another current example of this happening:

https://pypi.org/search/?q=psycopg2

[waseem18]

@di I presume the issue is because of usage of lowercase filter in normalized_name which is tokenizing psycopg2 to psycopg and 2

As per Elastic search,

lowercase tokenizer breaks text into terms whenever it encounters a character which is not a letter

[di]

Sounds like the same thing would happen with package names containing other valid non-alphanumeric characters like -, _, and .. Perhaps we need to tweak that tokenizer a bit.

[waseem18]

Yeah - We should decide on which tokenizer to use. What about whitespace tokenizer? I'll check search results accuracy with whitespace tokenizer vs lowercase tokenizer.

[di]

@honzakral is able to help with this issue