kato13
10-09-2017, 11:35 AM
Overnight for some reason Bing, Yandex(Russia Ukraine), and Baidu (China) search engines decided to reindex every page on our site. I don't pay for bandwitdh thank goodness but the sheer number of hits has slowed us down.
I had to simply ban Yandex as the were not following rules and were botting like 30 pages a second. Baidu seems to be behaving after I made some changes but Bing is still hitting us hard. If they don't stop in 24 hours I will have to put in some restrictions.
Long term it might make finding pages harder if you use bing for find stuff on this site. So I was wondering how popular it was.
I use the site search mostly, but using google site search
example
site:forum.juhlin.com EMP kato13
can sometimes be easier.
So I was wondering if people used bing the same way.
It already seems to be slowing down a bit (60 kbs a second compared to 340kbs overnight) so it may not be an issue, but I wanted to make sure I was not making things more complicated for users.
As much as google gets some flak they are efficient in their botting. They have a more complete record of our site than any other and I have never had issues with them overloading us. I think the others are lazy and will simply bot multiple times rather than copy data.
I had to simply ban Yandex as the were not following rules and were botting like 30 pages a second. Baidu seems to be behaving after I made some changes but Bing is still hitting us hard. If they don't stop in 24 hours I will have to put in some restrictions.
Long term it might make finding pages harder if you use bing for find stuff on this site. So I was wondering how popular it was.
I use the site search mostly, but using google site search
example
site:forum.juhlin.com EMP kato13
can sometimes be easier.
So I was wondering if people used bing the same way.
It already seems to be slowing down a bit (60 kbs a second compared to 340kbs overnight) so it may not be an issue, but I wanted to make sure I was not making things more complicated for users.
As much as google gets some flak they are efficient in their botting. They have a more complete record of our site than any other and I have never had issues with them overloading us. I think the others are lazy and will simply bot multiple times rather than copy data.