String improvement and benchmark setup

[CrazyboyQCD]

• Use compact_string as variant of Ident, Float and Integer gated behind feature(enabled by default, local benchmark shows 13~15% performance improvement).
• Simple benchmark setup of preprocessing(this includes many files, should I only take some of them?).

[Kangz]

I wouldn't change the default just yet TBH. But also to get the best wins, we could instead make the Token use a Cow<&str> that points directly in the source buffer. It would make each token have the same lifetime as the source buffer but in the very vast majority of cases, 0 copiers would be happening, ending up in the best performance. WDYT?

[Kangz]

Also I'm asking to see if anyone from Naga can review the change at a high level.

[CrazyboyQCD]

we could instead make the Token use a Cow<&str> that points directly in the source buffer. It would make each token have the same lifetime as the source buffer but in the very vast majority of cases, 0 copiers would be happening, ending up in the best performance. WDYT?

I did try to do this, but the calculation of range is harder than I thought, since this requires that lexer tracks skipped characters, which brings many branches in parse_* and increases complexity, also there are tons of lifetimes, so I think is not worth it.

[teoxoy]

Using CompactString sounds fine to me, I recently switched from &str to CompactString for a parser I'm working on and I expected performance to be much worse but it turned out to be fine. IIRC it was marginally slower (1-2%) with the benefit of not having to have lifetimes everywhere. I guess having most strings inline and having to copy them at least once is about the same as not having to copy them but instead reach out in the source string. Cow<&str> might be even slower than CompactString since any access will need to take a branch; but it would be great to have a benchmark prove it.

[Kangz]

Where do these bench files come from? I think glslpp-rs will very quickly produce an error when it encounters the #include so we're just benchmarking the parsing of the first few lines.

Also gathering files like this with no license disclosure could be a problem.

[CrazyboyQCD]

I think glslpp-rs will very quickly produce an error when it encounters the #include so we're just benchmarking the parsing of the first few lines.

I checked locally, it did produce err variants, but not exit since benchmark call collect here:

glslpp-rs/benches/benchmark.rs

Line 13 in 54a58e6

let _ = Preprocessor::new(input).collect::<Vec<_>>();

Where do these bench files come from?
Also gathering files like this with no license disclosure could be a problem.

I mark them here, the original project is MIT licensed, do I need to add license disclosure somewhere else? Or do you have some better source to benchmark with?

glslpp-rs/benches/benchmark.rs

Line 5 in 54a58e6

// Copied from https://github.com/qiutang98/flower/tree/dark/install/shader (MIT License)