How to create link previews like social media apps (open source API)

For modern social media apps its now the norm to have good link previews. Many apps such as WhatsApp, Signal, Facebook, Twitter, and more generate a preview whenever you share a link. These previews usually include a title, description and snappy image.

Signal message with a link and a link preview — one of the more reliable implementations around

Unfortunately when searching around for an easy way to implement this I found that most of the online APIs, tutorials and libraries to be outdated and unreliable. Even apps like Signal fail for some links (e.g. try out a link such as https://www.bloomberg.com/, this usually fails on a lot of preview generators due to the site’s antibot detection and is a good acid test).

So I built an API to generate link previews as good as these social media apps if not better! The entire project is open source and available here: https://github.com/krthush/link-preview. Before going over how it works, you can quickly try out the API on this demo site: https://favorited-link-preview.herokuapp.com/

Demo on https://favorited-link-preview.herokuapp.com/ (works on “tricky” sites such as Bloomberg)

Client Side Link Preview?

When trying to implement a link preview, it might be possible to do a client-side fetch of the link’s site (on the user’s device) to obtain the relevant preview data instead of using a server-side API. There are various reasons for/against this strategy.

The primary negative is that for web apps (using browsers) its generally not possible due to CORS restrictions from most sites (see the section on CORS for client-side web scraping here).

For native apps like mobile/desktop apps, client-side fetching is doable (Signal actually does this discussed here), but by then “offloading” the fetch to user’s device, this puts more work on the user. The user device will have to ping the external site (additional user data usage) and lose some anonymity.

The positive for a client-side implementation though, is that you avoid server-side fetchs that can run into rate limits from sites, thus being unable fetch the preview. The solution for correcting the server-side fetchs leads to requiring multiple proxy server instances and further antibot precautions.

Using a client-side implementation also makes it much harder to have a general solution regardless of language/framework — with an API, you would just need to ping the link preview API server whenever you app detects a link, rather than having to build a custom client-side implementation for fetching link previews.

Now let’s get into how the API works!

Code Breakdown

The main strategy for the API is to fetch the relevant data from the meta tags of the site for a given link.

To do this we first get the site html, try a basic request (through axios), and then fallback to more advanced methods such as browser emulation through Puppeteer (with additional packages such as puppeteer-extra-stealth-plugin and puppeteer-extra-plugin-adblocker to prevent detection).

After obtaining the site html, we parse through it using cheerio to grab the site’s meta tag elements (such as title, description, image, favicon, sitename, etc.) — by now most links will have returned the data required for a link preview.

Finally we have additional fall backs of using Bing image search (searching the site domain name and/or title for images) as well as fetching the largest image from the html.

I summarised the major techniques above instead of going through all the code line by line as it would lead to a very lengthy article. As mentioned before, the code is available at https://github.com/krthush/link-preview with fairly good documentation and comments. I suggest cloning/forking the repo, alongside following this article — then with some tinkering you should have a good understanding of the whole project!

Getting Started

Prerequisites:

• Git
• Node.js — the API is built on Nextjs, but can be adapted to be used in any other Node.js frameworks.
• Microsoft Azure account with Bing Search Key (optional)

To run the code locally, first download the project:

git clone https://github.com/krthush/link-preview
cd link-preview

Then install the packages:

npm install
# or
yarn install

Finally run the development server:

npm run dev
# or
yarn dev

Open http://localhost:3000 with your browser and you should see the landing page. The link preview API route should work on http://localhost:3000/api/link-preview?url=

Taking The API To Production

If you’re happy using this API as is with Next.js, you should be able to get the API live in a few minutes! See this section of the readme: https://github.com/krthush/link-preview#heroku---deploy-to-production-in-5-minutes

Otherwise if using other Node.js environments, you’ll need to take precautions to ensure puppeteer is running smoothly see: https://github.com/puppeteer/puppeteer/blob/main/docs/troubleshooting.md#running-puppeteer-in-the-cloud

You should now have a near perfect link preview API 🎉 that works in production!

At favedd.com we use this API for our creators/influencers so that they can easily add their favorite products as elegant cards without having to add their own title/description/image. We’re constantly looking to improve this API and would love any feedback/contributions!