parse5 and streaming

[jeromew]

Is there a way to pipe a file into parse5 (this feature is available in htmlparser2).

similarly, is there a way to pause / resume the sax parser ? I believe that the getNextToken method could be used to decide when the parsing should pause/resume.

I have been using the html-tokenize streaming parser lately and its suite (html-select, trumpet) but reaching html5 conformity on html-tokenize is still a long way to go so I am trying to see how parse5 could fit in and be used with html-select

[jeromew]

the Tokenizer and Preprocessor take the html content as one big chunk. Have you considered the possibility of working on partial chunks that would be written one after the other ?

parser.write('<div>')
parser.write('hello')
parser.write('</div>')
parser.end();

[inikulin]

Hi,
currently parse5 doesn't support streaming. However we've discussed possibility of it's implementation with jsdom maintainers in context of this thread.

I would like to know which goal you would like to achieve using streaming? If it's primary focus is performance then I would like to warn you that I find it a little bit questionable. For streaming we need to teach tokenizer to invalidate non-emitted tokens if end of chunk was encountered. This behavior requires introduction of the tokenizer state snapshots mechanism: if we have invalidated token we need to rollback tokenizer to the last valid state and retreat preprocessor to the point there last valid token was emitted. Since tokenizer and preprocessor are the most performance-sensitive parts of parse5 this may end up with significant performance degradation, so you can lose more than you win.

Long story short. I definetly would like to see streaming API in parse5 too. But it requires quite complex research and I'm afraid I will not be able to get my hands on this soon, since we need to land more important features (like <template> support) first and I'm already suffering from the lack of the spare time. Therefore, any PR on this task will be highly appreciated.

[jeromew]

Ok I see thanks for your feedback & for the link to the other thread. I agree that there is a big question mark on the result of all this performance wise.

I'll start my research with understanding the caveats of the html5 spec and try to understand how the approach taken by html-tokenize is compatible or not with all this.

[stevenvachon]

Streaming would be useful in parsing HTML that contains template logic such as handlebars/etc.

[inikulin]

@stevenvachon Sounds interesting. Can you describe this scenario in details, please? How streaming can help here?

[stevenvachon]

@inikulin Having worked on my idea further, I realize that I was mistaken. Instead, #33 makes more sense.

[stevenvachon]

The biggest benefit to streaming is in memory usage and garbage collection, not parse performance. Not having to parse an entire html file in a list of thousands has great benefits.

[aredridel]

Also having a DOM to start working on before all the data has arrived.

[stevenvachon]

Any progress on this at all?

[angelozerr]

In my case I'm using https://github.com/isaacs/sax-js and it work's great. The feature with sax-js that I needed where:

• speed
• parse an HTML even if it's not valid (a XML element which is not closed for instance)
• retrieve location of each elements, attributes
• works with web browser (without using browserify)

And sax-js provides me those features (for speed, I have not benchmark).

I tell me if it's a good idea for parse5 to spend time to implement a new sax parser although sax-js works great.

But perhaps I have missed something?

[inikulin]

@angelozerr sax-js is not the HTML parser.
@stevenvachon I will be able to get my hands on it not earlier than end of July/August. Any PR is still welcome. If someone would like to start implementing it, then you can count on my assistance.

[angelozerr]

@angelozerr sax-js is not the HTML parser.

You could implement sax-js callback to create DOM node and having an HTML Document, no?

[inikulin]

@angelozerr No, HTML parser requires html preprocessing/tokenization/parsing algorithms.