AJAX: What it is, how it works, and its impact on SEO

AJAX (from English Asynchronous Javascript and XML) is a technology for dynamically loading data on a site, for example, for endless loading of content on the main page or another page. A characteristic feature of AJAX is continuous loading of content. Simply put, AJAX is refreshing the page without reloading it.

Background data exchange with the server improves the user experience, because you do not need to spend time loading the page and, as a rule, pressing pagination buttons. To implement dynamic loading of data, you need to add the appropriate one to the site template script.

Compared to other web technologies, AJAX is quite young: it was first discussed as an independent approach to the formation of web application interfaces in 2005. The author of the technology is considered to be Jesse James Garrett.

How AJAX is arranged technically

AJAX uses several technologies: Dynamic approach HTML for constantly changing the content of the web page and several technologies of dynamic access to the server at once. In particular, dynamic generation of img and script tags and dynamic generation of side frames.

As for data formats, XML, standard, can be used for their transfer in AJAX text, JSON and standard HTML.

AJAX and SEO: Indexing issues

Can Google crawl AJAX content? Short answer: yes, it can. The longer answer is yes, but it's harder for crawlers to do. Single-page web applications using Ajax frameworks have historically been very problematic from an SEO perspective. Here are their main disadvantages:

• Problems with scanning. The content important to the crawlers was hidden inside JavaScript that was only displayed on the client side, which meant that Google's robots were essentially seeing a blank screen.
• Problems with the navigation bar (The "Back" button does not work in the browser or does not work correctly).
• Disguise. With the AJAX approach, webmasters created two versions of the content. The first - for the user, the second - for search engines. And this is prohibited and punishable by sanctions.

For years, Google has advised webmasters to use a proper AJAX crawling scheme — to tell crawlers that a site has AJAX content. An AJAX crawl scheme using the _escaped_fragment parameter allowed Google to retrieve a pre-processed version of the page.

This version had standard static HTML that Google could easily parse and index. In other words, server instructed crawlers to crawl a page other than the one available in the source code.

Everything changed in 2015. Google has announced that its crawlers have now learned to crawl, read, and parse content inside JavaScript without any problems, making the use of the AJAX crawling scheme with the _escaped_fragment parameter obsolete.

What happens with the indexing of AJAX pages

Google is often deceptive. Today, he claims that he has no problems crawling and indexing AJAX sites. But it would be risky to just take his word for it and lose what he had gained over the years traffic and site positions.

Indeed, Google can index dynamic AJAX content. But there are non-obvious points:

• Hidden HTML. If important content is hidden in JavaScript, it can be more difficult for crawlers to access it. Indexing (and subsequent ranking) can be stopped artificially. To avoid this, make sure that the content that is important to the user is stored in HTML format. In this case, Google crawlers will easily index it.
• Missing links. Google uses internal links as a signal to establish a connection between the pages of the site. And external links are one of the ranking factors. In other words, when the content is high-quality, expert, other trusted domains link to it. It is very important that the links on the site are accessible to crawlers and not hidden in JavaScript.

How AJAX affects SEO

It turns out, observing the above conditions, it is possible not to worry about the indexing of AJAX content at all?

To do this, let's go back in time and find Google's official answer on this matter:

"…as long as you don't block Googlebot from crawling JavaScript or CSS, Google will show your pages in search results."

The wording is somewhat vague, but the point is clear. Google seems to be saying: "It's not our problem, it's yours." So no more workarounds and crutches to let the search engine know where is AJAX content and where is plain content. He knows how to scan it on his own.

Google actively advocates for content and user experience. AJAX-content somewhat contradicts this approach: take at least an incorrect one URL- the address of the page that is formed using this approach. And this is important for crawlers: the URL must reflect the actual location of the page!

To solve the problem with URL pages during AJAX access, you need to use the History API with the pushState() function. It modifies the URL displayed on the client side.

Using pushState allows you to save Ajax content on the site and solves the problem of an incorrect URL page.

Another thing that Google pays attention to is the hashbang (#!). Google looks for hashed parameters to identify dynamic URLs and process them (in different ways). The crawler scans everything after the hashtag and passes that as a URL parameter, then simply requests a static version of the page that it can read, index, and rank.

How to make AJAX content accessible to Google

Here are some more tips to help make AJAX pages and AJAX content search engine friendly.

• Optimize URL structure. We recommend using the History API c pushState method instead of protocol_escaped_fragment (which is deprecated as we remember).

PushState updates the URL in the address bar so that any content that is in JavaScript is rendered without any issues.

• Optimize speed site When the client-side browser creates the DOM (application programming interface), there is a high probability of creating bloated code inside the HTML. Such code will reduce the page loading speed and, therefore, make it more difficult for Google crawlers. And do you remember that they also have limits on the number of visits to the site when crawling, the so-called Crawl Budget?

The crawling budget is the limit of a search robot based on the number of hits to a certain domain.

So make sure you remove all render-blocking resources so that Google can effectively parse the CSS on the page and all the scripts.

Sites entirely based on AJAX have been a headache for owners for a long time, because search engines did not know how to index them correctly. Now the problem of indexing AJAX content has been successfully solved.

However, the AJAX approach can still be a bit cumbersome for large sites and even bad for SEO if you don't know how to implement such content correctly.

Use the History API function and all the tips we gave above: this way you will insure your site against problems with indexing and you will be able to use all the advantages of the technology without fear of sanctions. And prerender will help make sure that AJAX content is available every time Googlebot calls.

Prerender is the process of preloading all elements on a page in preparation for their viewing by a search engine. The prerendering service intercepts the page request to see if the user-agent (browsing your site) is a bot

If so, the prerender will send a cached version of your site (to be rendered with all JavaScript, images, etc.). This will be a static page. If the user-agent is not a bot, everything is loaded as usual. Thus, the prerender is used to optimize interaction with bots only.

Why do search engines have such a hard time trying to index AJAX sites? To answer this question, it is necessary to consider how the client interacts with the server.

Sequence of requests to the server: standard and AJAX
The standard approach to building a web application interface involves the following sequence of interaction between the client and the server:

1. The user opens the website page.
2. Interacts with any element on the page.
3. When the corresponding click is made, the browser creates an HTTP request.
4. The generated HTTP request is sent to the server.
5. The server creates a new version of the page.
6. The browser receives the corresponding request and reloads the original version of the web page to display the modified page.

The AJAX approach involves a completely different sequence of interaction between the client and the server:

1. The user opens the website page.
2. Interacts with any element on the page.
3. The script on the site identifies the type of data that is needed to get the updated version of the web page.
4. The browser or other web client sends the required data type inside the HTTP request.
5. The server responds by not returning the entire document, but only the part that is needed to update the page
6. The script on the site changes the content of the page taking into account the data received in the HTTP request that came from the server.

AJAX benefits

The AJAX approach is especially valuable for media content. If the site uses a video or audio player, then AJAX allows you to broadcast the stream continuously. This technology is used by large projects such as SoundCloud, Last.fm, YouTube, Vimeo.

Here are three more scenarios where AJAX will be especially relevant on a site for the user and the webmaster:

1. The user cannot download large volumes of traffic. Sites created entirely on AJAX load much faster and save user traffic.
2. A webmaster wants to implement a window with dynamic data. For example, display the product balance in the online store or a window with tips. AJAX is ideal for displaying such data. At the same time, the content is displayed instantly, without delays or uploads.
3. The webmaster encountered server overload. In most cases, AJAX allows you to significantly reduce the load of the site created on the server. The fact is that only the changed part of the page or even a bare data set is uploaded, which is later changed due to the script.

Disadvantages of AJAX

First of all, the difficulty of indexing pages can be called a disadvantage of AJAX. Such problems can arise if the approach is not sufficiently thought out and without appropriate preparation: for example, due to the installation of a plugin that adds dynamic loading of content.

Here are some more disadvantages of AJAX for the webmaster:

• The URL of the page does not reflect its actual location within the site.
• Visits to AJAX pages may not be recorded by web analytics. The fact is that the browser does not record them as visited due to the technical features of the "client-server" chain interaction, which we already described at the beginning.
• Not all AJAX content can be indexed.
• JS must be enabled in the browser. A very indirect drawback, because today you simply cannot open any modern site without JavaScript enabled.
• The user does not have the "Back" browser function available.
• Crawlers can display page content that users won't see. According to the rules of search engines, this is a serious violation

Summary: Proper AJAX for SEO

To summarize all the above and not get confused, here are the main conclusions:

1. Google already knows how to index AJAX content. But in particularly complex technical cases indexing An AJAX page can be really tricky. To solve these problems, as well as implement AJAX-based tools, plugins for CMS from independent developers will help.
2. If no methods help your site and PS AJAX content is persistently not seen - set up prerender. This method ensures that crawlers will see the HTML code of the page even when AJAX is active. It will not be difficult to implement prerender: third-party plugins are available for popular CMS.
3. After performing all the above steps regenerate or update XML map. So you will make the task of crawlers much easier.