case where the path is the same in the links.txt

[anandijain]

hey again, love dlm! and thanks for your help in the past

https://static.case.law/us/1/CasesMetadata.json
https://static.case.law/us/2/CasesMetadata.json
https://static.case.law/us/3/CasesMetadata.json
...

the problem i ran into was

[2024-12-18 00:13:52] Skipping CasesMetadata.json because the file is already completed [299.39KiB]                 
[2024-12-18 00:13:52] Skipping CasesMetadata.json because the file is already completed [299.39KiB]                 
[2024-12-18 00:13:52] Skipping CasesMetadata.json because the file is already completed [299.39KiB]           
...      

the problem is that these files are actually distinct, but i couldn't find a way to tell dlm the names that i wanted to use or to canonicalize them to be unique in some way.

i dont want to complicate this beautifully simple package too much, but this case might be useful to handle

[agourlay]

Thanks for opening this!

I believe this is a recurring issue that could be handled by dlm natively.

My first idea would be to use the segments in the URL to name the files.

e.g.

https://static.case.law/us/1/CasesMetadata.json
https://static.case.law/us/2/CasesMetadata.json
https://static.case.law/us/3/CasesMetadata.json

to

us-1-CasesMetadata.json
us-2-CasesMetadata.json
us-3-CasesMetadata.json

I guess there are edge cases requiring additional chars sanitization to make sure the names are valid 🤔

WDYT?

[anandijain]

great solution to me! means not adding more args