Several webcrawlers found on Magistream
-
- MagiStream Donor
- Creatures • Trade
- Posts: 83
- Joined: May 22nd, 2024, 1:58:12 am
- Gender: Male
- Location: 砂隠れの里
Several webcrawlers found on Magistream
There are several webcrawling bots on magistream website scraping data.
Re: Several webcrawlers found on Magistream
The search engines ones have been here for a very long time.
I am a bit curious whether the AI bots are at least partly the source of the lag - I've heard about some of them being very aggressive with their scraping including some listed in the Who Is Online section.
I am a bit curious whether the AI bots are at least partly the source of the lag - I've heard about some of them being very aggressive with their scraping including some listed in the Who Is Online section.
-
- Creatures • Trade
- Posts: 12958
- Joined: October 26th, 2010, 12:57:20 am
- Gender: Female
- Location: Befriending peacock phoenixes
Re: Several webcrawlers found on Magistream
The AI bots were, in fact, causing some of the lag before Rose was able to identify them as bots and tell the forum software to treat them as such, which, if I understood correctly, throttles them so that they stop being able to scrape so aggressively and take up so much server resources.Katnilb wrote: ↑May 23rd, 2024, 6:34:39 am The search engines ones have been here for a very long time.
I am a bit curious whether the AI bots are at least partly the source of the lag - I've heard about some of them being very aggressive with their scraping including some listed in the Who Is Online section.
Kestrad has been eaten by life. She'll probably pop back in occasionally.
Keep story | Portal Guild | Graphics Shop
Please do not click my hatchlings. Thank you.
Avatar by Kingsfisher, sig art by herinbon
Keep story | Portal Guild | Graphics Shop
Please do not click my hatchlings. Thank you.
Avatar by Kingsfisher, sig art by herinbon
-
- MagiStream Donor
- Creatures • Trade
- Posts: 83
- Joined: May 22nd, 2024, 1:58:12 am
- Gender: Male
- Location: 砂隠れの里
Re: Several webcrawlers found on Magistream
Thanks for letting me know that they are already on top of it! I appreciate the replies!