Helmet not working with Facebook scraper. #26

sampurcell93 · 2015-08-17T05:10:32Z

Hi there, I've just started using Helmet to implement meta tags, so apps like Facebook and Twitter can get some useful information about my page and put it into previews. However, I notice that there is a delay between when I render a page, and when the meta tags/title get set on the page. This seems to be causing Facebook and Twitter to miss the tags, as they are not present at the instant of page load. I'm wondering if I am doing something wrong, or there is a fix I am not aware of.

I am currently using the recommended server-side technique - IE

str = React.renderToString(componentwithhelmet);
Helmet.rewind();
res.send(str)

I'm using express. And I have confirmed that when I hard code in the description (that is, it is helmet-independent), FB picks up the data.

I really like the package, but if it doesn't take care of this kind of thing, I am afraid I'll have to roll my own architecture for tags like these. Any insight would be appreciated!

Thanks,
Sam

The text was updated successfully, but these errors were encountered:

potench · 2015-08-22T19:48:54Z

When you "ViewSource" on your app, you should see the og:meta tags in the source. In this case str should contain the og:meta tags; can you check and see what str contains? Does it contain any og:meta data or perhaps the wrong og:meta data?

sampurcell93 · 2015-08-23T15:21:25Z

It actually does not contain any at render time. It takes a second for the tags to be injected - I know this, because I see the Tab title flicker from the default to the one set by Helmet after a split second.

doctyper · 2015-08-23T16:41:05Z

@sampurcell93 This may be an implementation issue. We use Helmet in production and Helmet does successfully pre-render data (view source).

In your example, you are calling Helmet.rewind() but are not ingesting its payload. Note the example, the call returns the stringified payload necessary to prerender your data:

React.renderToString(<Handler />);
let head = Helmet.rewind();
// head = { title, meta, link }

Which you can then send to the server (pseudo-code):

res.send(`
<!DOCTYPE html>
<html>
    <head>
        <meta charset="utf-8" />
        ${head.meta}
        ${head.link}
        <title>${head.title}</title>
    </head>
    <body>
    </body>
</html>
`);

PaulieScanlon · 2017-10-05T17:08:51Z

Hi, i'm also seeing problems with Facebook scraper not picking up the og tags. I have for the moment just hard coded them on my index but my posts use helmet to inject the image and title. Is this an issue with Helmet or React?

site in question is here: pauliescanlon.io it's the posts page after you view a portfolio item that's my current problem.

rus-yurchenko · 2017-10-13T14:50:28Z

+1

PaulieScanlon · 2017-10-13T16:09:35Z

@RuslanYurchenko The issue is not with Helmet. Crawlers can only read what’s in the meta tag that’s hard coded. Using helmet to inject this data at run time doesn’t mean it’s crawlable... unless server side rendered.

If like me you’re doing this on the client it won’t work. Shame but never mind ay!

rus-yurchenko · 2017-10-13T16:24:44Z

@PaulieScanlon yeah, I realized this a little later. Sorry!

lipenco · 2017-10-18T13:48:38Z

Is there any other solution I can consider to solve FB problem when using react-helmet on front-end.

cjimmy · 2017-10-24T20:57:29Z

@lipenco
For those coming here from Google, it seems there are few options (after much research). This assumes you've built a client-side React app (with Create React App, for example), and social media crawlers (Open Graph, Twitter Cards) can't see your meta data because it's serving the pre-rendered index.html

Convert your app to server-side rendering
This is the most obvious solution but the most onerous. You won't be able to use client-side definitions like window in your js. If you're using React Router, you'll have to find a way to mirror the routes between server and client. If you're like me, you might be serverless, and running a server would be a lot more work. On the other hand, your page will likely load faster, and crawlers will see what your users would. This is a non-exhaustive list of tradeoffs.

Use a pre-rendering service
Prerender.io, Render-tron, and Prerender.cloud to name a few, give you a way to server-side render when the user-agent is a bot. Some CDNs like Netlify and Roast.io do this for you so you don't have to run your own server.
The downside to this is this is yet another service to pay for. EDIT: Netlify is free, and prerendering is one-click, albeit in beta.

Pre-render on your own
A couple of packages exist for rendering your React app statically. Graphcool's Prep, React-Snap, React-Snapshot were ones I've found that all essentially run a local server to render the site and download the html files. The files won't be pretty, but if all you're looking for is the <head> generated by React Helmet, this will do.

In the end, I ended up using react-snap to render static files, and it hardly changed my build workflow. This was sufficient for me!

Any others I'm missing?

Edit 07/20/18: I've recently started hosting on Netlify (which has an option to turn on prerendering) to remove react-snap. It was causing an unsightly flash of unstyled content when loading the page.

riccardolardi · 2018-05-02T11:54:22Z

Be sure to check the Facebook Crawler Docs & the debugger to get more insight on how it crawls the content. Helped me better understand what's going on: https://developers.facebook.com/docs/sharing/webmasters/crawler

ChristiaanScheermeijer · 2019-03-20T14:10:34Z

In our case, a GTM trigger caused an <iframe> to be inserted directly after the <html> tag. This made the Netlify prerender or Facebook crawler to move the body tag before the iframe. This caused all of our meta tags to be inside the body tag. Apparently, the Facebook crawler will ignore some of them.

This didn't scrape all og tags: 😭

<!DOCTYPE html>
<html lang="en" class="wf-montserrat-n5-inactive wf-montserrat-n7-inactive wf-averiasanslibre-n5-inactive wf-averiasanslibre-n7-inactive wf-sansserif-n4-inactive wf-inactive">
<body>
<iframe height="0" width="0" style="display: none; visibility: hidden;" src=""></iframe>
<meta http-equiv="content-type" content="text/html; charset=utf-8">
<style>
...
</style>
<meta property="og:image" content="http://...." data-react-helmet="true">

After removing the trigger that added the iframe, the prerendered HTML was valid again. All og tags were being scraped again. 🚀

<!DOCTYPE html>
<html lang="en" class="wf-montserrat-n5-inactive wf-montserrat-n7-inactive wf-averiasanslibre-n5-inactive wf-averiasanslibre-n7-inactive wf-sansserif-n4-inactive wf-inactive">
<head>
<meta http-equiv="content-type" content="text/html; charset=utf-8">
<style>
...
</style>
<meta property="og:image" content="http://...." data-react-helmet="true">

ghost · 2019-06-06T15:24:31Z

In our case, a GTM trigger caused an <iframe> to be inserted directly after the <html> tag. This made the Netlify prerender or Facebook crawler to move the body tag before the iframe. This caused all of our meta tags to be inside the body tag. Apparently, the Facebook crawler will ignore some of them.

This didn't scrape all og tags: 😭
<!DOCTYPE html>
<html lang="en" class="wf-montserrat-n5-inactive wf-montserrat-n7-inactive wf-averiasanslibre-n5-inactive wf-averiasanslibre-n7-inactive wf-sansserif-n4-inactive wf-inactive">
<body>
<iframe height="0" width="0" style="display: none; visibility: hidden;" src=""></iframe>
<meta http-equiv="content-type" content="text/html; charset=utf-8">
<style>
...
</style>
<meta property="og:image" content="http://...." data-react-helmet="true">
After removing the trigger that added the iframe, the prerendered HTML was valid again. All og tags were being scraped again. 🚀
<!DOCTYPE html>
<html lang="en" class="wf-montserrat-n5-inactive wf-montserrat-n7-inactive wf-averiasanslibre-n5-inactive wf-averiasanslibre-n7-inactive wf-sansserif-n4-inactive wf-inactive">
<head>
<meta http-equiv="content-type" content="text/html; charset=utf-8">
<style>
...
</style>
<meta property="og:image" content="http://...." data-react-helmet="true">

@ChristiaanScheermeijer We are experiencing the same issue. Can I ask what the GTM trigger was and if you modified your implementation of it or just completely removed it? I can't see an option where I can just remove our page view events from happening as that is really the only thing being triggered. Any insight would be greatly appreciated.

ChristiaanScheermeijer · 2019-06-11T11:38:00Z

Hi @smschick,

At first, we've removed the trigger completely. However, since we really needed the trigger to work, we enabled it again and temporarily (until Netlify fixes the bug) removed the <DOCTYPE html> part.

But beware for some unpredictable side-effects while doing so. Especially in older browsers.

In our case, it was a Floodlight trigger. But it could be caused by any trigger who adds an iframe to the page.

sophylee · 2022-03-18T20:01:06Z

Hi, I can confirm that this is still an issue. Facebook developer tools only sees the following when it crawls our home page:

<!DOCTYPE html>
<html lang="en">
<head>
<script async src="...">
<title>Pledge Ukraine</title>
<script ...>
</head>
<body>
<noscript>You need to enable JavaScript to run this app.</noscript>
<div id="root"></div>
</body>
</html>

Even though opening up inspector in a browser shows all of the meta tags present. Our un-ideal workaround was to hard code our meta og and twitter tags into index.html.

doctyper closed this as completed Sep 9, 2015

RichardJECooke mentioned this issue Oct 4, 2016

Allow meta tags to be seen by Facebook & Twitter scraper #181

Closed

csantiago132 mentioned this issue Mar 10, 2018

Social media crawler (Facebook, Twitter, Whatsapp, etc) doesn't detect meta tags generated by React Helmet react-boilerplate/react-boilerplate#2149

Closed

broskoski mentioned this issue Feb 12, 2019

Missing Twitter card for profiles aredotna/ervell#1793

Open

GabrielMajeri mentioned this issue Apr 1, 2020

Add social og tags code4romania/covid-19-cine-ce-face#11

Closed

5 tasks

swdilip mentioned this issue Mar 13, 2023

Topic Metatags [Blocked] Sigma-Labs-XYZ/Explain-AI#118

Closed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Helmet not working with Facebook scraper. #26

Helmet not working with Facebook scraper. #26

sampurcell93 commented Aug 17, 2015

potench commented Aug 22, 2015

sampurcell93 commented Aug 23, 2015

doctyper commented Aug 23, 2015

PaulieScanlon commented Oct 5, 2017

rus-yurchenko commented Oct 13, 2017

PaulieScanlon commented Oct 13, 2017

rus-yurchenko commented Oct 13, 2017

lipenco commented Oct 18, 2017

cjimmy commented Oct 24, 2017 •

edited

Loading

riccardolardi commented May 2, 2018 •

edited

Loading

ChristiaanScheermeijer commented Mar 20, 2019 •

edited

Loading

ghost commented Jun 6, 2019

ChristiaanScheermeijer commented Jun 11, 2019

sophylee commented Mar 18, 2022

Helmet not working with Facebook scraper. #26

Helmet not working with Facebook scraper. #26

Comments

sampurcell93 commented Aug 17, 2015

potench commented Aug 22, 2015

sampurcell93 commented Aug 23, 2015

doctyper commented Aug 23, 2015

PaulieScanlon commented Oct 5, 2017

rus-yurchenko commented Oct 13, 2017

PaulieScanlon commented Oct 13, 2017

rus-yurchenko commented Oct 13, 2017

lipenco commented Oct 18, 2017

cjimmy commented Oct 24, 2017 • edited Loading

riccardolardi commented May 2, 2018 • edited Loading

ChristiaanScheermeijer commented Mar 20, 2019 • edited Loading

ghost commented Jun 6, 2019

ChristiaanScheermeijer commented Jun 11, 2019

sophylee commented Mar 18, 2022

cjimmy commented Oct 24, 2017 •

edited

Loading

riccardolardi commented May 2, 2018 •

edited

Loading

ChristiaanScheermeijer commented Mar 20, 2019 •

edited

Loading