A blog about WordPress plugin settings and search engine optimization for beginner webmasters. A quick way to check the indexing of pages in Yandex and Google First indexing of a page in Yandex

Hello Friends! Very often you ask me the following questions: - “why does my site take so long to be indexed”, “how often does Yandex index sites”, “can my content be stolen before indexing and accuse me of theft”, how to speed up the indexing of a site by Yandex” and so on. I sent more than 50 replies to everyone interested and decided to simplify my work and answer the questions described above in this article, so to speak, kill two birds with one stone))) So, let's start the hunt!

Let's first look at two concepts, indexing and updating. I will try to briefly write what they are and what they are eaten with.

Site indexing- this is site processing, that is, programs that scan the site and fill search engine databases with results for search queries. If indexing is done correctly, the user will easily find required pages site. Otherwise, the web resource will not be displayed in search engines; in other words, users will not see it and will not purchase the product, use the service, and so on.

The term " update " comes from the English word - update, which means update. Among specialists involved in the field, it is often used and literally means “updating data.” Varieties of such an update would be refining the interface of the site or making changes to a specific page. Among specialists, an abbreviated form of the word is also used - “ap”.

The appearance of new search results after a request for the same key phrase in a search engine, for example, Google or Yandex - that’s what an “update” is in relation to. Search engines have , in which all pages are arranged in order. The update leads to changes in the positions of existing web resource records or the addition of new ones in responses to user requests. The process associated with the update leads to updating the search engine cache and indexing sites in its database.

You can read more about these two concepts on the pages of my website, here are links to what it is and what it is.

Well, I think we’ve sorted that out, now let’s move on to your questions.

Why is my site taking so long to be indexed?

It all depends on the search robot. It is he who visits sites according to a certain schedule, known only to Yandex, and scans millions of WEB projects for updates. For example, if you add or change content (articles, photographs, videos, product cards) on your website once a month, then the robot will sooner or later understand this and will visit your resource only once a month. Hence, the period for indexing the site will be the same.

The more often you add content, the more often the robot will come to your site. On news resources such as Fontanka.ru or RBK, robots simply “live” and do not leave there. News on such sites is updated and added daily in dozens, or even hundreds, and search bots scan and index them almost instantly.

Can content be stolen before the site is indexed?

Yes they can! And that's exactly how it's done. While the site is young and just beginning its life, it is indexed once a month, or even twice. All content that is published on such a site can be stolen, either manually or automatically (which happens more often), working in self-populating sites. Most of my students are very surprised when I analyze their site and identify content that is not unique.

My last student published 15 articles on his young website at once and a month later he was very surprised that Yandex accused him of copy-paste and imposed a filter on the site. It was not possible to prove to Yandex (Platon Shchukin) that he was not the thief, but on the contrary, his articles were stolen. Doing this now is very problematic, and besides, in order to get the site out of the filter, you will have to delete all these articles and write new ones. The result of ignorance and incorrect actions is loss of money, loss of content and falling under the filter.

You can read how to protect your content from theft, or watch my video if you are too lazy to read))) I will not write about this in detail here.

How to protect content from theft

Now let's talk about how to speed up the indexing process.

How to speed up the indexing process

To make the indexing of your site and newly added content faster, you need to “ask” the Yandex bot to visit your site. I do this simply and in several ways.

The first way to speed up site indexing

I attract a robot to my site by sharing posts of published articles through social media. I do this using a block of social network buttons. As soon as a post is published, robots see it, since they live on social networks constantly and do not leave there, as well as from news feeds.

Publishing articles on social networks transmits social signals to the site and, in addition to speeding up indexing, has a beneficial effect on site promotion in Google and Yandex. Let me remind you that before sharing texts, be sure to let Yandex know that they are yours (see video above).

The second way to speed up site indexing

Here we inform Yandex-Bot that new content has appeared on our website through Yandex Web Master, a section of which is popularly called Adurilka.

We go to Yandex Webmaster, register, add your site and go to the indexing tab - re-crawling pages. Then copy the URL of the page that you want to quickly index and add it to the column, as in the picture below, and click the submit button.

After this, we wait for the robot to walk around. Adding new URLs in Adurilka significantly speeds up the indexing process, and when you work in conjunction with the service, you also completely eliminate the theft of your content. Everything is actually very simple if you know all this.

It’s also easy to promote sites in Yandex. You can do this yourself and absolutely free, but again you need knowledge. If you want, I can teach you this too (information below).

Website promotion training

That's all for today, good luck to everyone and see you again!

Have you created a website, but can’t find it in search engines? No problem! In this material you will learn how to index a website in Yandex and Google in the shortest possible period of time. It’s probably unnecessary to talk about the advantages of quickly getting into the search engine index. After all, anyone understands that the sooner his website is shown in search results, the faster new clients will appear. And for this to work, you need to get into the search engine database.

By the way, thanks to the right approach, new materials on our site are quite well, and most importantly, always quickly indexed by search engines. Perhaps you came to this page after using the corresponding request in the search bar. Let’s move on from the lyrics to practice.

How to find out if a site is indexed?

The first thing you need to do is find out whether the site is indexed by search engines. It may be that the site simply is not on the first page of the search for the query you entered. These may be high-volume queries that need to be worked on to show up SEO promotion, and not just make and launch a website.

So, to check, we go to all the search engines that make sense to go to ( Yandex, Google, Mail, Rambler) and enter the site address in the search query line.

If your resource has not yet been indexed, nothing will be shown in the search results, or other sites will appear.

How to index a site in Yandex?

First, we’ll tell you how to index a website in Yandex. But before you add your resource, check that it works correctly, opens correctly on all devices and contains only unique content. For example, if you add a site at the development stage, you can simply fall under the filter - this happened to us once and we had to wait a whole month for Yandex to understand that we have a high-quality site and lift the sanctions.

To inform Yandex about a new site, you must add it to Yandex Webmaster And install Yandex Metrica, the first tool is responsible for additional information about the resource ( region, structure, quick links) and how the site looks in organic results, the second for collecting data on the site ( attendance, behavior, etc.), which, according to our experience, also affects the indexing of the site and its position in the search. Also, be sure to make a sitemap and indicate it in the webmasters panel.

How to index a site in Google?

Most often, Google itself quickly finds new sites and puts them into search, but waiting for Google to come and do all the work for us is too presumptuous, so let’s figure out how to index a site in Google.

After the sites are added, it should go through from 3 to 7 days, before search engines update the data and index the site.

You always want new website pages to appear in search results as quickly as possible, and for this there are several secret (and very simple) ways to speed up the indexing of website pages in search engines.

3. To perform the manipulation described in paragraph 2, only for the search engine Google systems, go to Search Console. Select "Scan" and "See like Googlebot"-add the address of the new page and click “Crawn”, after which we request indexing.

Site indexing analysis

In conclusion, it should be noted that even after successful indexing of the site in search engines, the work does not end there. It is necessary to periodically analyze the indexing of the site, as well as remove positions for popular queries. This will allow you to keep your finger on the pulse and not end up in a situation where a significant part of the traffic from organic search results has simply disappeared.

This happened to many old sites that used old promotion methods after the release Yandex algorithm Baden-Baden. At the same time, Yandex announced in advance that it was launching this algorithm and over-optimized pages would be excluded from the search, while Google never reports the release of new algorithms. Therefore, only tireless monitoring will allow you to remain or become a topic leader!

Quite often, a new site cannot be found in Yandex. Even if you type its name in the search bar. The reasons for this may be different. Sometimes search engines simply don’t yet know that a new resource has appeared. To figure out what’s going on and solve the problem, you need to register your site with Yandex.Webmaster.

What is site indexing in Yandex

First, let's figure out how search engines generally find out about new sites or changes to them. Yandex has a special program called a search robot. This robot surfs the Internet and looks for new pages. Sometimes he goes to old ones and checks to see if anything new has appeared on them.

When the robot finds a useful page, it adds it to its database. This base is called search index. When we look for something in the search, we see sites from this database. Indexing is when the robot adds new documents there.

A robot cannot crawl the entire Internet every day. He doesn't have enough power for that. Therefore, he needs help - to report about new pages or changes to old ones.

What is Yandex.Webmaster and why is it needed?

Yandex.Webmaster is an official service from Yandex. You need to add a website to it so that the robot knows about its existence. With its help, resource owners (webmasters) can prove that this is their site.

You can also see in Webmaster:

• when and where the robot entered;
• which pages it indexed and which it did not;
• what keywords do people search for?
• are there any technical errors?

Through this service you can set up a website: set the region, prices of goods, protect your texts from theft. You can ask the robot to re-visit the pages where you made changes. Yandex.Webmaster makes it easy to move to https or another domain.

How to add a new website to Yandex.Webmaster

Go to the Webmaster panel. Click "Login". You can enter the login and password that you use to log into Yandex mail. If you don't have it yet account, you will have to register.

After logging in, you will be taken to a page with a list of added resources. If you have not used the service before, the list will be empty. To add a new resource, click the “+” button.

On the next page, enter the address of your site and confirm its addition.

At the last stage you need to confirm your rights- prove to Yandex that you are the owner. There are several ways to do this.

How to confirm rights to a website in Yandex.Webmaster

The easiest way to confirm rights in Yandex.Webmaster is add a file to the site. To do this, click on the “HTML File” tab.

A small file will download. You'll need this file now, so save it somewhere you can see it. For example, on the desktop. Do not rename the file! There is no need to change anything about it.

Now upload this file to your website. Typically used for this file managers, But users don't need to do any of this. Just go to the back office, click "Files". Then at the top of the page - “Add file”. Select the file you downloaded earlier.

Then return to the Yandex.Webmaster panel and click the “Check” button. After successfully confirming access rights, your site will appear in the list of added ones. Thus, you have informed Yandex.Webmaster about the new site.

Meta tag Yandex.Webmaster

Sometimes the method described above does not work, and the owners cannot confirm the rights to the site in Webmaster. In this case, you can try another method: add a line of code to the template.

In Webmaster go to the "Meta Tag" tab. You will see a line that needs to be added to the HTML code.

Users can contact technical support and ask to insert this code. This will be done as part of a free revision.

When they do this in Webmaster, click the “Check” button. Congratulations, you have registered your site in a search engine!

Preliminary setup of Yandex.Webmaster

The site has been added to the search, now the robot will definitely come to you and index it. This usually takes up to 7 days.

Add a link to your sitemap

In order for the robot to index the resource faster, add the sitemap.xml file to Webmaster. This file contains the addresses of all pages of the resource.

Online stores already have this file configured and should be added to Webmaster automatically. If this does not happen, add a link to sitemap.xml in the “Indexing” - “Sitemap Files” section.

Check robots.txt

In the robots.txt file indicate pages that the robot does not need to visit. These are the cart, checkout, back office and other technical documents.

By default, it creates robots.txt, which does not need to be modified. Just in case, we recommend checking for errors in robots. To do this, go to “Tools” - “Analysis of robots.txt”.

Set the site region

On the “Site Information” - “Region” page, you can set the region of the site. For online stores, these are the cities, regions and countries where purchased goods are delivered. If you don’t have a store, but a directory or blog, then the region will be the whole world.

Set the sales region as shown in the screenshot:

What else is Webmaster useful for?

On the page " Search queries» you can see the phrases that come to you from the search.

The “Indexing” section displays information about when the robot was on the site and how many pages it found. The “Site Moving” subsection will help you if you decide to install and switch to https. The “Page Retraversal” subsection is also extremely useful. In it you can indicate to the robot the pages on which the information has changed. Then, on your next visit, the robot will index them first.

On the “Products and Prices” page of the “Site Information” section, you can provide information about your online store. To do this, the resource must be configured to upload data on products and prices in YML format. At correct setting Prices and delivery information will be displayed in the search results of product pages.

If you want to improve the visibility of your company in Yandex services, you should use the “Useful Services” section. In Yandex.Directory, you can specify the phone number, address of your store, and opening hours. This information will be displayed directly in Yandex results. This will also add you to Yandex.Maps.

Yandex.Metrica - another important tool for the owner of an Internet resource, showing traffic data. Statistics and dynamics of site traffic are displayed in easy-to-analyze tables, charts and graphs.

After connecting to the Yandex.Webmaster and Yandex.Metrica services, you will receive a sufficient amount of information to manage the site’s positions and traffic. These are indispensable tools for website owners who want to promote their resources in the most popular search engine in Russia.

The next step in website promotion is through a similar service Search Console. That's all, good luck with your promotion!

With Google everything is very simple. You need to add your site to webmaster tools at https://www.google.com/webmasters/tools/, then select the added site, thus getting into the Search Console of your site. Next, in the left menu, select the “Scanning” section, and in it the “View as Googlebot” item.

On the page that opens, in the empty field, enter the address of the new page that we want to quickly index (taking into account the already entered domain name of the site) and click the “Crawl” button to the right. We wait until the page is scanned and appears at the top of the table of addresses previously scanned in a similar way. Next, click on the “Add to Index” button.

Hurray, your new page is instantly indexed by Google! In just a couple of minutes you will be able to find it in Google search results.

Fast indexing of pages in Yandex

IN new version webmaster tools became available similar tool to add new pages to the index. Accordingly, your site must also first be added to Yandex Webmaster. You can also get there by selecting the desired site in the webmaster, then go to the “Indexing” section, select the “Page Re-Crawling” item. In the window that opens, enter the addresses of new pages that we want to quickly index (via a link on one line).

Unlike Google, indexing in Yandex does not yet occur instantly, but it is trying to strive for it. Using the above actions you will inform the Yandex robot about new page. And it will be indexed within half an hour to an hour - this is what my personal experience shows. Perhaps the speed of page indexing in Yandex depends on a number of parameters (the reputation of your domain, account and/or others). In most cases, you can stop there.

If you see that the pages of your site are poorly indexed by Yandex, that is, several general recommendations how to deal with this:

• The best, but also difficult, recommendation is to install the Yandex quickbot on your website. To do this, it is advisable to add fresh materials to the site every day. Preferably 2-3 or more materials. Moreover, add them not all at once, but after a while, for example, in the morning, afternoon and evening. It would be even better to maintain approximately the same publication schedule (approximately maintain the same time for adding new materials). Also, many recommend creating an RSS feed for your site so that search robots can read updates directly from it.
• Naturally, not everyone will be able to add new materials to the site in such volumes - it’s good if you can add 2-3 materials per week. In this case, you can’t really dream about the speed of Yandex, but try to get new pages into the index in other ways. The most effective of which is considered to be posting links to new pages to upgraded Twitter accounts. By using special programs like Twidium Accounter, you can “pump up” the number of Twitter accounts you need and, with their help, quickly insert new site pages into the search engine index. If you do not have the opportunity to post links to upgraded Twitter accounts yourself, you can buy such posts through special exchanges. One post with your link will cost on average 3-4 rubles and more (depending on the strength of the selected account). But this option will be quite expensive.
• The third option for quick indexing is to use the http://getbot.guru/ service, which for just 3 rubles will help you achieve the desired effect with a guarantee of results. Well suited for sites with a rare schedule of adding new publications. There are also cheaper rates. It is better to look at their details and differences on the website of the service itself. Personally, I have services of this service I'm very pleased with it as an indexing accelerator.

Of course, you can also add new posts to social bookmarks, which in theory should also help the site get indexed quickly. But the effectiveness of such an addition will also depend on the level of your accounts. If you have little activity on them and you use accounts only for such spam, then there will be practically no useful output.

P.S. with extensive experience is always relevant - contact us!

What is site indexing? How does it happen? You can find answers to these and other questions in the article. in search engines) is the process of adding information about a site to a database by a search engine robot, which is subsequently used to search for information on web projects that have undergone this procedure.

Data about web resources most often consists of keywords, articles, links, documents. Audio, images, and so on can also be indexed. It is known that the algorithm for identifying keywords depends on the search device.

There are some restrictions on the types of information indexed (flash files, javascript).

Inclusion management

Indexing a website is a complex process. To manage it (for example, prohibit the inclusion of a particular page), you need to use the robots.txt file and regulations such as Allow, Disallow, Crawl-delay, User-agent and others.

Tags are also used for indexing and props , hiding the contents of the resource from Google robots and Yandex (Yahoo uses the tag ).

In the Goglle search engine, new sites are indexed from a couple of days to one week, and in Yandex - from one week to four.

Do you want your site to show up in search engine results? Then it must be processed by Rambler, Yandex, Google, Yahoo, and so on. You must report search engines(spiders, systems) about the existence of your website, and then they will crawl it in whole or in part.

Many sites have not been indexed for years. The information contained on them is not seen by anyone except their owners.

Processing methods

Site indexing can be done in several ways:

1. The first option is to add it manually. You need to enter your site data through special forms offered by search engines.
2. In the second case, the search engine robot itself finds your website using links and indexes it. He can find your site using links from other resources that lead to your project. This method is the most effective. If a search engine finds a site this way, it considers it significant.

Deadlines

Site indexing is not very fast. The terms vary, from 1-2 weeks. Links from authoritative resources (with excellent PR and Tits) significantly speed up the placement of the site in the search engine database. Today Google is considered the slowest, although until 2012 it could do this job in a week. Unfortunately, everything is changing very quickly. It is known that Mail.ru has been working with websites in this area for about six months.

Not every specialist can index a website in search engines. The timing of adding new pages to the database of a site that has already been processed by search engines is affected by the frequency of updating its content. If fresh information constantly appears on a resource, the system considers it frequently updated and useful for people. In this case, its work is accelerated.

You can monitor the progress of website indexing in special sections for webmasters or on search engines.

Changes

So, we have already figured out how the site is indexed. It should be noted that search engine databases are frequently updated. Therefore, the number of pages of your project added to them may change (either decrease or increase) for the following reasons:

• search engine sanctions against the website;
• presence of errors on the site;
• changes in search engine algorithms;
• disgusting hosting (inaccessibility of the server on which the project is located) and so on.

Yandex answers to common questions

"Yandex" - search engine, which is used by many users. It ranks fifth among search systems in the world in terms of the number of research requests processed. If you added a site to it, it may take too long to add it to the database.

Adding a URL does not guarantee it will be indexed. This is just one of the methods by which the system informs the robot that a new resource has appeared. If your site has few or no links from other sites, adding it will help you discover it faster.

If indexing does not occur, you need to check whether there were any failures on the server at the time the request was created by the Yandex robot. If the server reports an error, the robot will terminate its work and try to complete it in a comprehensive crawl. Yandex employees cannot increase the speed of adding pages to the search engine database.

Indexing a website in Yandex is a rather difficult task. You don't know how to add a resource to a search engine? If there are links to it from other websites, then you do not need to add the site specifically - the robot will automatically find and index it. If you don't have such links, you can use the Add URL form to tell search engines that your site exists.

It is important to remember that adding a URL does not guarantee that your creation will be indexed (or how quickly it will be indexed).

Many people are interested in how long it takes to index a website in Yandex. Employees of this company do not make guarantees or predict deadlines. As a rule, since the robot has learned about the site, its pages appear in searches within two days, sometimes after a couple of weeks.

Processing process

Yandex is a search engine that requires accuracy and attention. Site indexing consists of three parts:

1. The search robot crawls the resource pages.
2. The content of the site is recorded in the database (index) of the search system.
3. After 2-4 weeks, after updating the database, you can see the results. Your site will appear (or not appear) in search results.

Indexing check

How to check site indexing? There are three ways to do this:

1. Enter the name of your business in the search bar (for example, “Yandex”) and check each link on the first and second page. If you find the URL of your brainchild there, it means the robot has completed its task.
2. You can enter your site's URL in the search bar. You will be able to see how many Internet sheets are shown, that is, indexed.
3. Register on the webmasters' pages in Mail.ru, Google, Yandex. After you pass the site verification, you will be able to see the results of indexing and other search engine services created to improve the performance of your resource.

Why does Yandex refuse?

Indexing a site in Google is carried out as follows: the robot enters all pages of the site into the database, low-quality and high-quality, without selecting. But only useful documents are included in the ranking. And Yandex immediately excludes all web junk. It can index any page, but the search engine eventually eliminates all garbage.

Both systems have an additional index. Both have low quality pages that affect the ranking of the website as a whole. There is a simple philosophy at work here. A particular user's favorite resources will rank higher in search results. But this same individual will have difficulty finding a site that he didn’t like last time.

That is why it is first necessary to protect copies of web documents from indexing and check the presence blank pages and do not allow low-quality content to appear in search results.

Speeding up Yandex

How can I speed up site indexing in Yandex? The following steps must be followed:

Intermediate actions

What needs to be done until the web page is indexed by Yandex? A domestic search engine should consider the site the primary source. That is why, even before publishing an article, it is imperative to add its content to the “Specific Texts” form. Otherwise, plagiarists will copy the entry to their resource and end up first in the database. In the end, they will be recognized as the authors.

Google Database

Prohibition

What is a site indexing ban? You can apply it either to the entire page or to a separate part of it (a link or a piece of text). In fact, there is both a global indexing ban and a local one. How is this implemented?

Let's consider prohibiting adding a website to the search engine database in Robots.txt. Using the robots.txt file, you can exclude indexing of one page or an entire resource category like this:

1. User-agent: *
2. Disallow: /kolobok.html
3. Disallow: /foto/

The first point indicates that the instructions are defined for all subsystems, the second indicates that indexing of the kolobok.html file is prohibited, and the third does not allow adding the entire contents of the foto folder to the database. If you need to exclude several pages or folders, specify them all in Robots.

In order to prevent the indexing of an individual Internet sheet, you can use the robots meta tag. It differs from robots.txt in that it gives instructions to all subsystems at once. This meta tag obeys general principles html format. It should be placed in the page header between the Ban entry, for example, could be written like this: .

Ajax

How does Yandex index Ajax sites? Today, Ajax technology is used by many web site developers. Of course, she has great opportunities. Using it, you can create fast and productive interactive web pages.

However, the system “sees” the web sheet differently than the user and the browser. For example, a person looks at a comfortable interface with movably loaded Internet sheets. For a search robot, the content of the same page may be empty or presented as other static HTML content, for the generation of which scripts are not used.

To create Ajax sites, you can use a URL with #, but the search engine robot does not use it. Usually the part of the URL after the # is separated. This needs to be taken into account. Therefore, instead of a URL like http://site.ru/#example, he makes a request home page resource located at http://site.ru. This means that the content of the Internet sheet may not be included in the database. As a result, it will not appear in search results.

To improve the indexing of Ajax sites, Yandex supported changes in search robot and rules for processing URLs of such websites. Today, webmasters can indicate to the Yandex search engine the need for indexing by creating an appropriate scheme in the resource structure. To do this you need:

1. Replace the # symbol in the page URL with #!. Now the robot will understand that it can request an HTML version of the content for this Internet sheet.
2. The HTML version of the content of such a page should be placed at a URL where #! replaced by?_escaped_fragment_=.