Results 1 to 5 of 5

Thread: 301s and bots

  1. #1
    Thinkin' out loud again Builder's Avatar
    Join Date
    Nov 2002
    Location
    Illinois
    Posts
    2,088
    Rep Power
    19

    301s and bots

    So every spring I create a new folder for my troop's summer camp information for that year. I delete the previous year's folder and 301 it to the current folder. I generally leave the 301 directives in my .htaccess for 2 years or so; i.e. "/camp12/" and "/camp13/" redirect to "/camp14/". Works great. No issues with the .htaccess.

    But every once in a while I see this:
    Code:
    207.46.13.7 - - [17/Jul/2014:19:41:31 -0400] "GET /camp11/signup.htm HTTP/1.1" 404 1082 "-" "Mozilla/5.0 (compatible; bingbot/2.0; +http://www.bing.com/bingbot.htm)"
    207.46.13.7 - - [17/Jul/2014:19:41:32 -0400] "GET /camp13/form-info.htm HTTP/1.1" 301 247 "-" "Mozilla/5.0 (compatible; bingbot/2.0; +http://www.bing.com/bingbot.htm)"
    207.46.13.7 - - [17/Jul/2014:19:41:32 -0400] "GET /camp10/when.htm HTTP/1.1" 404 1082 "-" "Mozilla/5.0 (compatible; bingbot/2.0; +http://www.bing.com/bingbot.htm)"
    207.46.13.7 - - [17/Jul/2014:19:41:32 -0400] "GET /camp10/ HTTP/1.1" 404 1082 "-" "Mozilla/5.0 (compatible; bingbot/2.0; +http://www.bing.com/bingbot.htm)"
    207.46.13.86 - - [17/Jul/2014:19:41:39 -0400] "GET /camp11/mb.htm HTTP/1.1" 404 1082 "-" "Mozilla/5.0 (compatible; bingbot/2.0; +http://www.bing.com/bingbot.htm)"
    207.46.13.86 - - [17/Jul/2014:19:41:39 -0400] "GET /camp11/cost.htm HTTP/1.1" 404 1082 "-" "Mozilla/5.0 (compatible; bingbot/2.0; +http://www.bing.com/bingbot.htm)"
    157.55.39.103 - - [17/Jul/2014:19:41:49 -0400] "GET /camp11/other.htm HTTP/1.1" 404 1082 "-" "Mozilla/5.0 (compatible; bingbot/2.0; +http://www.bing.com/bingbot.htm)"
    157.55.39.103 - - [17/Jul/2014:19:41:49 -0400] "GET /camp13/extracost.htm HTTP/1.1" 301 247 "-" "Mozilla/5.0 (compatible; bingbot/2.0; +http://www.bing.com/bingbot.htm)"
    157.55.39.103 - - [17/Jul/2014:19:41:49 -0400] "GET /camp11/ HTTP/1.1" 404 1082 "-" "Mozilla/5.0 (compatible; bingbot/2.0; +http://www.bing.com/bingbot.htm)"
    157.55.39.103 - - [17/Jul/2014:19:41:49 -0400] "GET /camp10/mb.htm HTTP/1.1" 404 1082 "-" "Mozilla/5.0 (compatible; bingbot/2.0; +http://www.bing.com/bingbot.htm)"
    157.55.39.103 - - [17/Jul/2014:19:41:49 -0400] "GET /camp11/adults.htm HTTP/1.1" 404 1082 "-" "Mozilla/5.0 (compatible; bingbot/2.0; +http://www.bing.com/bingbot.htm)"
    207.46.13.85 - - [17/Jul/2014:19:42:47 -0400] "GET /camp13/progresp.htm HTTP/1.1" 301 246 "-" "Mozilla/5.0 (compatible; bingbot/2.0; +http://www.bing.com/bingbot.htm)"
    The actual bot doesn't matter, Googlebot does it, Yahoo, all of them.

    I thought part of the reason of redirecting (301 anyway) was to help search engines, but perhaps I'm wrong, or perhaps that's no longer the case. If bingbot et al is trying to access folders and files that haven't existed for at least 3 years, it's not worth my effort to update the .htaccess file. I'll just let all the old stuff 404 and be done with it.

    I checked my sitemap.xml. No traces of any of those "old" folders.

    The question being (I suppose), is it better to let those old files/folders 404, or 301 them?

    Hints? Comments? Suggestions? Educate me please!

    Thanks,
    Kevin
    A good friend will come and bail you out of jail...
    but a true friend will be sitting next to you saying,
    "Damn... that was fun!"

  2. #2
    Rick
    Join Date
    May 2002
    Location
    Minneapolis, MN
    Posts
    1,753
    Rep Power
    19
    Yeah, I think you've gotten a bit off-track here. 301's are best used when you're moving content, not just changing it. Most often, in SEO terms they're useful when you change your site's directory/URL structure or changing domain names.

    Going forward, I would suggest setting up a new permanent location for these pages and simply update them every year as needed so that the URLs remain the same. You can then set up 301's for all of the old URLs and point them to this one new location to capture the value from any external links that may be pointing to them, as well as redirecting real users who bookmarked the old location(s) in the past.

    In terms of actual rankings, I doubt that it matters much whether you take my advice or not. I would expect your site to rank pretty well for searches related to your troop either way. As a practical matter, the effect would simply be to reduce errors and make it easier to deal with any future problems because your log files wouldn't get more and more cluttered with these errors over time. You see, search engines never forget a valid URL, and once discovered, they'll keep trying to crawl those URLs despite any 404s or 301s they encounter indefinitely - just less frequently as time passes. Good luck!
    Rick Trethewey

  3. #3
    Thinkin' out loud again Builder's Avatar
    Join Date
    Nov 2002
    Location
    Illinois
    Posts
    2,088
    Rep Power
    19
    Haha! "Off-track". Sometimes my derailments can be pretty spectacular . Fortunately in this case it's only a minor annoyance.

    I must have misinterpreted something I read years ago about 301s, 404s and search engines. But glad to get put back on track. SEO doesn't matter much here. The pages in question are really internal-use-only even though they are publicly viewable.

    Re: setting up a "permanent" folder. That was a thought of mine several years ago, but for some reason I decided against it. It could have been something as simple as I was lazy. I'll see about getting off my lazy butt and doing it for next year.

    Quote Originally Posted by rainbore View Post
    You see, search engines never forget a valid URL, and once discovered, they'll keep trying to crawl those URLs despite any 404s or 301s they encounter indefinitely - just less frequently as time passes.
    Gack, I was kind of afraid of that. Although as I stated, it really and truly is a minor annoyance. Take-away for Kev: make one folder that gets updated every year and let all the old ones 404.

    Thanks again Rick! I always seem to be indebted to you for wise and sage advice.
    A good friend will come and bail you out of jail...
    but a true friend will be sitting next to you saying,
    "Damn... that was fun!"

  4. #4
    Rick
    Join Date
    May 2002
    Location
    Minneapolis, MN
    Posts
    1,753
    Rep Power
    19
    That's not sage, it's time. ;-)
    Rick Trethewey

  5. #5
    Thinkin' out loud again Builder's Avatar
    Join Date
    Nov 2002
    Location
    Illinois
    Posts
    2,088
    Rep Power
    19
    Badum-bum!
    "Thank you and good night! Please tip your waitresses!"


    Since I'm old the first thing that entered my brain was:
    http://www.lyricsfreak.com/s/simon+a..._20124689.html
    Even sadder is that I knew the actual name of the song...
    A good friend will come and bail you out of jail...
    but a true friend will be sitting next to you saying,
    "Damn... that was fun!"

Posting Permissions

  • You may not post new threads
  • You may not post replies
  • You may not post attachments
  • You may not edit your posts
  •