Over the last few months we’ve been seeing some strange behavior on our site. For instance, records in my running log would disappear regularly, but Shane’s records would not. The lists in the wishlist pages would also disappear regularly. For a while we suspected that there was something wrong with the code used to record new entries — maybe new data was being written incorrectly, which resulted in old data being corrupted.
Finally, on Anna’s suggestion, we took a look at the web site logs that our hosting provider gives us. The logs list each web page requested, the ip address of the requester, the name of the web page used to reach our page, and finally information about the software used to access our site.
To our surprise, we found entries that looked like this:
207.46.98.70 - - [08/Oct/2004:06:42:10 -0500] "GET /potpourri/marathon/deleteitem.php?id=9&name_id=1 HTTP/1.0" 302 0 "-" "msnbot/0.3 (+http://search.msn.com/msnbot.htm)"
Basically, this entry says that MSN’s search bot requested a web page to delete item 9 from our marathon running logs. There were a number of such entries, which resulted in all of my running log entries being deleted whenever the MSN search bot would come by. Shane’s log entries were unaffected because there is no direct link starting from our home page to the delete links for his log entries. One would need to select his name in a dropdown list in order to reach those links. The addition of a “/robots.txt” file was all that was needed to keep search bots from hitting the wrong pages.
Curious about what else we could learn by looking at our web site logs, we downloaded the logs for the first half of October and looked through them. We found that someone is using a goldfish cracker picture we took as the background for her personal web page: Palindrome85’s Profile
We also found, to our surprise, that a number of requests for our site came from search engines. People have been reaching our site through Google, MSN Search, Yahoo! and other search engines. We thought we would list some of them here:
Yahoo! Search
http://www.bernardandagnes.com/technology/PDA/PDA_05-06-2003.html
- omap 1510
- transcriber for palm Vx
http://www.bernardandagnes.com/potpourri/wordusage.html
- word usage lay
- sentence using the word hung
- the word fundamentally in a sentence
- compose a sentence using lain
- proper word usage
- confusing word usage
- word usage different
- between among usage
http://www.bernardandagnes.com/technology/home_networking.html
- changing mpbs xp
- router number of computers
http://www.bernardandagnes.com/entertainment/weddingdays.shtml
- common wedding days
http://www.bernardandagnes.com/technology/zire71faq.html
- mp3 files to sd card
- 71 screen
- zire 71 mp3
- zire 71 sd card
- zire 71 and lexar
- zire 71 case pictures
- palm 71 digitizer problem
- mp3 Kit for Zire 71
http://www.bernardandagnes.com/food/oyakodonburi_recipe.shtml
- oyako recipe
- oyako donburi
http://www.bernardandagnes.com/food/beefnoodlesoup_recipe.shtml
- pressure cooker recipe
- easy beef noodle soup recipe
- chinese beef noodle soup recipe
- recipe beef shin
- how long to cook chinese beef tendons
- beef soup recipe
- NOODLE soup recipe
- beef noodle soup
- tough noodle
- beef noodle soup recipe
http://www.bernardandagnes.com/potpourri/goodyearblimp.shtml
- david Samuels goodyear blimp
- goodyear blimp carson
- goodyear blimp rides
- The Goodyear Blimp
- goodyear blimp ride in carson
- goodyear blimp carson
- goodyear blimp
http://www.bernardandagnes.com/food/kungpaochicken_recipe.shtml
- kung pao spaghetti
- kung pao chicken recipe
- kung pao recipe
- kung pao chicken
- kung chicken recipe
- kung pao chicken recipe
- kung pao sauce recipe
http://www.bernardandagnes.com/potpourri/grumpybug.html
- grumpy bug
- robert scull grumpy bug
- grumpy bug
- The Grumpy Bug
- story the grumpy bug
- The Grumpy Bug by Robert Scull
- nick jr bug
- the grumpy bug robert
- grumpy story
- The Grumpy Bug Nick jr
http://www.bernardandagnes.com/food/snowcrab.shtml
- how to cook snow crab legs
- snowcrab legs
- ohio and crab legs
- how to cook snowcrab legs
- how to cook snowcrab
- different kinds of crab
- cook crab legs how long
- how to cook king crab legs
- how to cook snow crab legs
- cook snowcrab
- how long to cook snow crab
- how to cook king crab legs
- snow crab legs
http://www.bernardandagnes.com/food/cheesesteak.shtml
- philly cheesesteak
- how to make a philly cheesesteak
- philly cheesesteaks
- how to make cheesesteak
- best philly cheesesteak
- philly cheesesteaks
- Philly cheesesteak
http://www.bernardandagnes.com/journal/london/sandwiches.html
- pret-a-manger in new york
http://www.bernardandagnes.com/food/restaurants.shtml
- balucci restaurant
- ramen restaurants in Torrance
- sapporo ramen porter
- tea station irvine
http://www.bernardandagnes.com/food/nuts.shtml
- what to do with stale peanuts
- kinds of nuts
- walnuts hazelnuts chestnuts nuts
http://www.bernardandagnes.com/potpourri/goodyearblimp.shtml
- goodyear blimp balloon
- goodyear blimp ride carson
- where are the goodyear blimps
- blimp pilot
- goodyear blimp job
- goodyear spirit of america
- goodyear blimp torrance
http://www.bernardandagnes.com/journal/heater.shtml
- pictures of hand food and mouth disease
- heater
- bad foot
- foot
- Foot Pictures
- foot good
http://www.bernardandagnes.com/journal/raccoon.shtml
- raccoon poison
- rat raccoon poison
- get rid of raccoon
- getting rid of grubs
- raccoon
- how to get rid of a raccoon
MSN Search
http://www.bernardandagnes.com/food/guacamole_recipe.shtml
- Guacamole recipe
- guacamole lemon recipe
- guacamole recipe 4 avocados
http://www.bernardandagnes.com/food/cheesesteak.shtml
- big jo’s santa monica
- philly cheesesteak
- cheesesteak torrance
http://www.bernardandagnes.com/food/restaurants.shtml
- tea station irvine
- boba in torrance
- cobras and matadors
http://www.bernardandagnes.com/food/steak.shtml
- perfect steak
- how make the steak tender
- cook the perfect steak
- the perfect steak and oven
- grilling the perfect steak
- cook perfect steak
http://www.bernardandagnes.com/entertainment/weddingdays.html
- wedding days
- say congratulate someone wedding
http://www.bernardandagnes.com/potpourri/laflamme.html
- laflamme harvard square
- la flamme barbershop
- LaFlamme barbershop Harvard Square
- la flamme harvard square
- la flamme dunster street
- la flamme barber harvard square
http://www.bernardandagnes.com/potpourri/trigonometry/trigonometry.html
- sohcahtoa conversions
http://www.bernardandagnes.com/potpourri/pythagorean_theorem.html
- Pythagoran Proof
- pythagoran
http://www.bernardandagnes.com/food/restaurants.shtml
- balucci’s
- balucci, indian restaurant
- balucci’s new york indian
- balucci indian food
- balucci’s indian restaurant
- balucci’s new york