SEO News » Tags » and

Feeds

2082 items (1557 unread) in 3 feeds

Google (154 unread)

• Official Google Webmaster Central Blog (154 unread)

SEO (1403 unread)

• Emily Chang (95 unread)
• Search Engine Roundtable (1308 unread)

80 items tagged "and"

Related tags: indexing [+], crawling [+], feedback [+], communication [+], services [+], products [+]

Official Google Webmaster Central Blog

• 

  Google's SEO Report Card

  Posted: March 2nd, 2010, 5:02pm CET by Maile Ohye

  Tags:  feedback and communication  

  Webmaster Level: All
  
  How many of Google's web pages use a descriptive title tag? Do we use description meta tags? Heading tags? While we always try to focus on the user, could our products use an SEO tune up? These are just some of the questions we set out to answer with Google's SEO Report Card.
  
  Google's SEO Report Card is an effort to provide Google's product teams with ideas on how they can improve their products' pages using simple and accepted optimizations. These optimizations are intended to not only help search engines understand the content of our pages better, but also to improve our users' experience when visiting our sites. Simple steps such as fixing 404s and broken links, simplifying URL choice, and providing easier-to-understand titles and snippets for our pages can benefit both users and search engines. From the start of the project we also wanted to release the report card publicly so other companies and webmasters could learn from the report, which is filled with dozens of examples taken straight from our products' pages.
  
  The project looked at the main pages of 100 different Google products, measuring them across a dozen common optimization categories. Future iterations of the project might look at deeper Google product web pages as well as international ones. We released the report card within Google last month and since then a good number of teams have taken action on it or plan to.
  
  We hope you find our SEO Report Card useful and we'd love to hear your feedback in the comments below or in the Webmaster Central Help Forum. And if you'd like to do your own SEO tune up, a good place to start is by reading our free SEO Beginner's Guide.
  
  Written by Brandon Falls, Adi Goradia, and Charlene Perez, Search Quality Team
• 

  Is your site hacked? New Message Center notifications for hacking and abuse

  Posted: March 1st, 2010, 4:00pm CET by Michael Wyszomierski

  Tags:  feedback and communication  

  Webmaster Level: All
  As we crawl the web, we see bad content inserted on to thousands of hacked sites each day. The number of sites attacked is staggering and the problem is only getting worse. Hackers and spammers target and successfully compromise any sites they can - small personal sites, schools and universities, even multinational corporations. Spam attacks against forums and user content sections of sites, though not as shocking, are even more widespread.
  You may have read in an earlier post that we've begun notifying webmasters about new software versions via Webmaster Tools to help protect their sites. Continuing with our effort to provide more useful information to webmasters, we're happy to announce that we'll soon be sending even more notifications to the Message Center.
  Starting this month, we will notify more webmasters of more potential issues we've detected on their websites, including:

  • Spammy or abused user-generated content
  • Abused forum pages or egregious amounts of comment spam
  • Suspected hacking

  These notifications are meant to alert webmasters of potential issues and provide next steps on how to get their sites fixed and back into Google's search results. If it pertains to a hacking or abuse issue, the notification will point to example URLs exhibiting this type of behavior. These notifications will run in parallel with our existing malware notifications.
  A notice of suspected hacking, for example, will look like this:
  
  We've been notifying webmasters of suspected hacking for years, but a recent upgrade to our systems will allow us to notify many more site owners that have been hacked. We hope webmasters will find these notifications useful in making sure their sites are clean and secure, ultimately providing a better user experience for their visitors. In the future, we may extend this effort even further to include other types of vulnerabilities or abuse issues.
  Just as before, webmasters who have not already signed up for Webmaster Tools may still do so and retrieve previously sent messages within one year of their send date. And if you don't want to miss out on any important messages, remember to use the email forwarding feature to receive these alerts in your inbox.
  If you have any questions, please feel free to ask in our Webmaster Help Forum or leave your comments below.
  Posted by Jessica Wong and Jason Morrison, Search Quality Team
• 

  Protect your site from spammers with reCAPTCHA

  Posted: January 26th, 2010, 11:00am CET by Michael Wyszomierski

  Tags:  products and services  

  Webmaster Level: All
  If you allow users to publish content on your website, from leaving comments to creating user profiles, you’ll likely see spammers attempt to take advantage of these mechanisms to generate traffic to their own sites. Having this spammy content on your site isn't fun for anyone. Users may be subjected to annoying advertisements directing them to low-quality or dangerous sites containing scams or malware. And you as a webmaster may be hosting content that violates a search engine's quality guidelines, which can harm your site's standing in search results.
  There are ways to handle this abuse, such as moderating comments and reviewing new user accounts, but there is often so much spam created that it can become impossible to keep up with. Spam can easily get to this unmanageable level because most spam isn’t created manually by a human spammer. Instead, spammers use computer programs called “bots” to automatically fill out web forms to create spam, and these bots can generate spam much faster than a human can review it.
  To level the playing field, you can take steps to make sure that only humans can interact with potentially spammable features of your website. One way to determine which of your visitors are human is by using a CAPTCHA , which stands for "completely automated public Turing test to tell computers and humans apart." A typical CAPTCHA contains an image of distorted letters which humans can read, but are not easily understood by computers. Here's an example:
  
  You can easily take advantage of this technology on your own site by using reCAPTCHA, a free service owned by Google. One unique aspect of reCAPTCHA is that data collected from the service is used to improve the process of scanning text, such as from books or newspapers. By using reCAPTCHA, you're not only protecting your site from spammers; you're helping to digitize the world's books.
  Luis Von Ahn, one reCAPTCHA's co-founders, gives more details about how the service works in the video below:
  
  If you’d like to implement reCAPTCHA for free on your own site, you can sign up here. Plugins are available for easy installation on popular applications and programming environments such as WordPress and PHP.
  Posted by Michael Wyszomierski, Search Quality Team
• 

  Introducing a new Rich Snippets format: Events

  Posted: January 22nd, 2010, 11:20am CET by Maile Ohye

  Tags:  crawling and indexing  

  Webmaster Level: All
  
  Last year we introduced Rich Snippets, a new feature that makes it possible to surface structured data from your pages on Google's search results. So far, user reaction to Rich Snippets has been enthusiastic -- after all, Rich Snippets help people make more informed clicks and find what they need even faster.
  
  We originally introduced Rich Snippets with two formats: reviews and people. Later in the year we added support for marking up video information which is used to improve Video Search. Today, we're excited to kick off the new year by adding support for events.
  
  Events markup is based off of the hCalendar microformat. Here's an example of what the new events Rich Snippets will look like:
  
  
  The new format shows links to specific events on the page along with dates and locations. It provides a fast and convenient way for users to determine if a page has events they may be interested in.
  
  If you have event listings on your site, we encourage you to review the events documentation we've prepared to help you get started. Please note, however, that marking up your content is not a guarantee that Rich Snippets will show for your site. Just as we did for previous formats, we will take a gradual approach to incorporating the new event snippets to ensure a great user experience along the way.
  
  Stay tuned for more developments in Rich Snippets throughout the year!
  
  Written by Mike Danylchuk and Nitin Shetti
• 

  State of the Index 2009

  Posted: January 20th, 2010, 3:50pm CET by Michael Wyszomierski

  Tags:  feedback and communication  

  Webmaster Level: All
  At PubCon in Las Vegas in November 2009, I gave a "State of the Index" talk which covers what Google has done for users, web developers, and webmasters in the last year. I recently recreated it on video for those of you who didn't make it to the conference. You can watch it below:
  
  And here are the slides if you'd like to follow along:
  
  Posted by Matt Cutts, Search Quality Team
• 

  Answering your December Grab Bag questions

  Posted: January 14th, 2010, 10:16am CET by Michael Wyszomierski

  Tags:  feedback and communication  

  Webmaster Level: All
  You asked and Matt Cutts answered. It's time to answer the latest round of Grab Bag questions! Here's the first answer, complete with Matt's new hairstyle:
  
  We have a lot of videos ready to share, so we're not currently taking new questions for the Grab Bag. If you have a question that you would like answered, your best bet as always is to head to our Webmaster Help Forum, where you'll find plenty of knowledgeable webmasters, including some Bionic Posters!
  To be kept up-to-date on our latest video releases, you can follow @googlewmc on Twitter, where we'll announce new videos and blog posts as they're published.
  Posted by Michael Wyszomierski, Search Quality Team
• 

  Helping webmasters from user to user

  Posted: December 24th, 2009, 3:30pm CET by John Mueller

  Tags:  feedback and communication  

  You have to have some kind of super-powers to keep up with all of the issues posted in our Webmaster Help Forum—that's why we call our Top Contributors the "Bionic Posters." They're able to leap through tall questions in a single bound, providing helpful and solid information all around. We're thankful to the Bionics for tackling problems both hard and easy (well, easy if you know how). Our current Bionic Posters are: Webado (Christina), Phil Payne, Red Cardinal (Richard), Shades1 (Louis), Autocrat, Tim Abracadabra, Aaron, Cristina, Robbo, John, Becky Sharpe, Sasch, BbDeath, Beussery (Brian), Chibcha (Terry), Luzie (Herbert), 奥宁 (Andy), Ashley, Kaleh and Redleg!
  
  With thousands of webmasters visiting the English Help Forum every day, some questions naturally pop up more often than others. To help catch these common issues, the Bionic Posters have also helped to create and maintain a comprehensive list of frequently asked questions and their answers. These FAQs cover everything from "Why isn't my site indexed?" to diagnosing difficult issues with the help of Google Webmaster Tools, often referring to our Webmaster Help Center for specific topics. Before you post in the forum, make sure you've read through these resources and do a quick search in the forum; chances are high that your question has been answered there already.
  
  Besides the Bionic Posters, we're lucky to have a number of very active and helpful users in the forum, such as: squibble, Lysis, yasir, Steven Lockey, seo101, RickyD, MartinJ and many more. Thank you all for making this community so captivating and—most of the time—friendly.
  
  Here are just a few (well, a little more than a few) of the many comments that we've seen posted in the forum:
  
  

  • "Thank you for this forum... Thank you to those that take the time to answer and care!"
  • "I've only posted one question here, but have received a wealth of knowledge by reading tons of posts and answers. The time you experts put into helping people with their problems is very inspiring and my hat's off to each of you. Anyway, I just wanted to let you know that your services aren't going unnoticed and I truly appreciate the lessons."
  • "Thank you very much cristina, what you told me has done the trick. I really appriciate the help as this has been bugging me for a while now and I didn't know what was wrong."
  • "thank you ssssssssssoooo much kaleh. "
  • "OK, Phil Payne big thanks to You! I have made changes and maybe people are starting to find me in G! Thanks to Ashley, I've started to make exclusive and relevant content for people."
  • "If anything, it has helped me reflect on the sites and projects of days gone by so as to see what I could have done better - so that I can deliver that much more and better results going forward. I've learned that some things I had done right, were spot on, and other issues could have been handled differently, as well as a host of technical information that I've stored away for future use. Bottom Line: this forum rocks and is incredibly helpful."
  • "I asked a handful of questions, got GREAT help while doing a whole lot of lurking, and now I've got a site that rocks!! (...) Huge thanks to all the Top Contributors, and a very special mention to WEBADO, who helped me a TON with my .htaccess file."
  • "Over the years of reading (and sometimes contributing) to this forum I think it has helped to remove many false assumptions and doubts over Google's ranking systems. Contrary to what many have said I verily believe Google can benefit small businesses. Keep up the good work. "
  • "The forum members are awesome and are a most impressive bunch. Their contribution is immeasurable as it is huge. Not only have they helped Google in their success as a profitable business entity, but also helped webmasters both aspiring and experienced. There is also an engender(ment) of "family" or "belonging" in the group that has transcended the best and worst of times (Current forum change still TBD :-) ). We can agree, disagree and agree to disagree but remain respectful and civil (Usually :-) )."
  • "Hi Redleg, Thank you very much for all of the information. Without your help, I don't think I would ever have known how to find the problem. "
  • "What an amazing board. Over the last few days I have asked 1 question and recieved a ton of advice mainly from Autocrat. "
  • "A big thank you to the forum and the contributors that helped me get my site on Google . After some hassle with my web hosters and their naff submission service, issues over adding pages Google can see, issues over Sitemaps, I can now say that when I put my site name into the search and when i put in [custom made watch box], for instance, my site now comes up."
  • "Thank you Autocrat! You are MAGNIFICENT! (...) I am your biggest fan today. : ) Imagine Joe Cocker singing With a Little Help from My Friends...that's my theme song today."
  • "I've done a lot of reading since then and I've learned more in the last year than I learned in the previous 10. When I stumbled into this forum I had no idea what I was getting into but finding this forum was a gift from God! Words cannot express the amount of gratitude I feel for the help you have given me and I wish I could repay you some how.... I don't mean to sound so mushy, but I write this with tears in my eyes and I am truly, truly grateful..."

  
  Are you new to the Webmaster Help Forum? Tell us a little bit about yourself and then join us to learn more and help others!
  
  Posted by John Mueller, Webmaster Trends Analyst, Google Zürich
• 

  Handling legitimate cross-domain content duplication

  Posted: December 15th, 2009, 3:47pm CET by John Mueller

  Tags:  crawling and indexing  

  Webmaster level: Intermediate
  
  We've recently discussed several ways of handling duplicate content on a single website; today we'll look at ways of handling similar duplication across different websites, across different domains. For some sites, there are legitimate reasons to duplicate content across different websites — for instance, to migrate to a new domain name using a web server that cannot create server-side redirects. To help with issues that arise on such sites, we're announcing our support of the cross-domain rel="canonical" link element.
  
  
  
  Ways of handling cross-domain content duplication:
  

  • Choose your preferred domain
    When confronted with duplicate content, search engines will generally take one version and filter the others out. This can also happen when multiple domain names are involved, so while search engines are generally pretty good at choosing something reasonable, many webmasters prefer to make that decision themselves.

  • Reduce in-site duplication
    Before starting on cross-site duplicate content questions, make sure to handle duplication within your site first.

  • Enable crawling and use 301 (permanent) redirects where possible
    Where possible, the most important step is often to use appropriate 301 redirects. These redirects send visitors and search engine crawlers to your preferred domain and make it very clear which URL should be indexed. This is generally the preferred method as it gives clear guidance to everyone who accesses the content. Keep in mind that in order for search engine crawlers to discover these redirects, none of the URLs in the redirect chain can be disallowed via a robots.txt file. Don't forget to handle your www / non-www preference with appropriate redirects and in Webmaster Tools.

  • Use the cross-domain rel="canonical" link element
    There are situations where it's not easily possible to set up redirects. This could be the case when you need to move your website from a server that does not feature server-side redirects. In a situation like this, you can use the rel="canonical" link element across domains to specify the exact URL of whichever domain is preferred for indexing. While the rel="canonical" link element is seen as a hint and not an absolute directive, we do try to follow it where possible.

  
  
  Still have questions?
  
  Q: Do the pages have to be identical?
  A: No, but they should be similar. Slight differences are fine.
  
  Q: For technical reasons I can't include a 1:1 mapping for the URLs on my sites. Can I just point the rel="canonical" at the homepage of my preferred site?
  A: No; this could result in problems. A mapping from old URL to new URL for each URL on the old site is the best way to use rel="canonical".
  
  Q: I'm offering my content / product descriptions for syndication. Do my publishers need to use rel="canonical"?
  A: We leave this up to you and your publishers. If the content is similar enough, it might make sense to use rel="canonical", if both parties agree.
  
  Q: My server can't do a 301 (permanent) redirect. Can I use rel="canonical" to move my site?
  A: If it's at all possible, you should work with your webhost or web server to do a 301 redirect. Keep in mind that we treat rel="canonical" as a hint, and other search engines may handle it differently. But if a 301 redirect is impossible for some reason, then a rel="canonical" may work for you. For more information, see our guidelines on moving your site.
  
  Q: Should I use a noindex robots meta tag on pages with a rel="canonical" link element?
  A: No, since those pages would not be equivalent with regards to indexing - one would be allowed while the other would be blocked. Additionally, it's important that these pages are not disallowed from crawling through a robots.txt file, otherwise search engine crawlers will not be able to discover the rel="canonical" link element.
  
  We hope this makes it easier for you to handle duplicate content in a user-friendly way. Are there still places where you feel that duplicate content is causing your sites problems? Let us know in the Webmaster Help Forum!
  
  
  Posted by John Mueller, Webmaster Trends Analyst, Google Zürich
• 

  New User Agent for News

  Posted: December 2nd, 2009, 9:02am CET by Jonathan Simon

  Tags:  crawling and indexing  

  Webmaster Level: Intermediate
  
  Today we are announcing a new user agent for robots.txt called Googlebot-News that gives publishers even more control over their content. In case you haven't heard of robots.txt, it's a web-wide standard that has been in use since 1994 and which has support from all major search engines and well-behaved "robots" that process the web. When a search engine checks whether it has permission to crawl and index a web page, the "check if we're allowed to crawl this page" mechanism is robots.txt.
  
  Publishers could easily contact us via a form if they didn't want to be included in Google News but did want to be in Google's web search index. Now, publishers can manage their content in Google News in an even more automated way. Site owners can just add Googlebot-News specific directives to their robots.txt file. Similar to the Googlebot and Googlebot-Image user agents, the new Googlebot-News user agent can be used to specify which pages of a website should be crawled and ultimately appear in Google News.
  
  Here are a few examples for publishers:
  
  Include pages in both Google web search and News:
  User-agent: Googlebot
  Disallow:
  
  This is the easiest case. In fact, a robots.txt file is not even required for this case.
  
  Include pages in Google web search, but not in News:
  User-agent: Googlebot
  Disallow:
  
  User-agent: Googlebot-News
  Disallow: /
  
  This robots.txt file says that no files are disallowed from Google's general web crawler, called Googlebot, but the user agent "Googlebot-News" is blocked from all files on the website.
  
  Include pages in Google News, but not Google web search:
  User-agent: Googlebot
  Disallow: /
  
  User-agent: Googlebot-News
  Disallow:
  
  When parsing a robots.txt file, Google obeys the most specific directive. The first two lines tell us that Googlebot (the user agent for Google's web index) is blocked from crawling any pages from the site. The next directive, which applies to the more specific user agent for Google News, overrides the blocking of Googlebot and gives permission for Google News to crawl pages from the website.
  
  Block different sets of pages from Google web search and Google News:
  User-agent: Googlebot
  Disallow: /latest_news
  
  User-agent: Googlebot-News
  Disallow: /archives
  
  The pages blocked from Google web search and Google News can be controlled independently. This robots.txt file blocks recent news articles (URLs in the /latest_news folder) from Google web search, but allows them to appear on Google News. Conversely, it blocks premium content (URLs in the /archives folder) from Google News, but allows them to appear in Google web search.
  
  Stop Google web search and Google News from crawling pages:
  User-agent: Googlebot
  Disallow: /
  
  This robots.txt file tells Google that Googlebot, the user agent for our web search crawler, should not crawl any pages from the site. Because no specific directive for Googlebot-News is given, our News search will abide by the general guidance for Googlebot and will not crawl pages for Google News.
  
  For some queries, we display results from Google News in a discrete box or section on the web search results page, along with our regular web search results. We sometimes do this for Images, Videos, Maps, and Products, too. This is known as Universal search results. Since Google News powers Universal "News" search results, if you block the Googlebot-News user agent then your site's news stories won't be included in Universal search results.
  
  We are currently testing our support for the new user agent. If you see any problems please let us know. Note that it is possible for Google to return a link to a page in some situations even when we didn't crawl that page. If you'd like to read more about robots.txt, we provide additional documentation on our website. We hope webmasters will enjoy the flexibility and easier management that the Googlebot-News user agent provides.
  
  Written by Jonathan Simon, Webmaster Trends Analyst
• 

  Region Tags in Google Search Results

  Posted: December 1st, 2009, 5:25pm CET by Maile Ohye

  Tags:  crawling and indexing  

  Webmaster Level: All
  
  Country-code top-level domains (or ccTLDs) can provide people with a quick and valuable clue about the location of a website—for example, ".fr" for France or ".co.jp" for Japan. However, for certain top level domains like .com, .info and .org, it's not as easy to figure out the location. That's why today we're adding region information supplied by webmasters to the green address line on some Google search results.
  
  This feature is easiest to explain through an example. Let's say you've heard about a boxing club in Canada called "Capital City Boxing." You try a search for [capital city boxing] to find out more, but it's hard to tell which result is the one you're looking for. Here's a screen shot:
  
  
  None of the results provide any location information in the title or snippet, nor do they have a regional TLD (such as .ca for Canada). The only way to find the result you're looking for is to refine your search ([capital city boxing canada] works) or click through the various links to figure it out. Clicking through the first result reveals that there's apparently another "Capital City Boxing" club in Alabama.
  
  Region tags improve search results by providing valuable information about website location right in the green URL line. Continuing our prior example, here's a screen shot of the new region tag (circled in red):
  
  
  
  As you can see, the fourth result now includes the region name "Canada" after the green URL, so you can immediately tell that this result relates to the boxing club in Canada. With the new display, you no longer need to refine your search or click through the results to figure out which page is the one you're looking for. In general, our hope is that these region tags will help searchers more quickly identify which results are most relevant to their queries.
  
  As a webmaster, you can control how this feature works by adjusting your Geographic Targeting settings. Log in to Webmaster Tools and choose Site configuration > Settings > Geographic Target. From here you can associate a particular country/region with your site. These settings will determine the name that appears as a region tag. You can learn more about using the Geographic Target tool in a prior blog post and in our Help Center.
  
  We currently show region tags only for certain domains such as .com and .net where the location information would otherwise be unclear. We don't show region tags for results on domains like .br for Brazil, because the location is already implied by the green URL line in our default display. In addition, we only display region tags when the region supplied by the site owner is different from the domain where the search was entered. For example, if you do a search from the Singapore Google domain (google.com.sg), we won't show you region tags for all the websites webmasters have targeted to Singapore because we'd end up tagging too many results, and the tag is really most relevant for foreign regions. For the initial release, we anticipate roughly 1% of search results pages will include webpages with a region tag.
  
  We hope you'll find this new feature useful, and we welcome your feedback.
  
  Written by Piyush Prahladka, Software Engineer
• 

  Changes in First Click Free

  Posted: December 1st, 2009, 10:02am CET by John Mueller

  Tags:  products and services  

  Webmaster level: intermediate
  
  We love helping publishers make their content available to large groups of readers, and working on ways to make the world's information useful and accessible through our search results. At the same time, we're also aware of the fact that creating high-quality content is not easy and, in many cases, expensive. This is one of the reasons why we initially launched First Click Free for Google News and Google Web Search -- to allow publishers to sell access to their content in general while still allowing users to find it through our search results.
  
  While we're happy to see that a number of publishers are already using First Click Free, we've found that some who might try it are worried about people abusing the spirit of First Click Free to access almost all of their content. As most users are generally happy to be able to access just a few pages from these premium content providers, we've decided to allow publishers to limit the number of accesses under the First Click Free policy to five free accesses per user each day. This change applies to both Google News publishers as well as websites indexed in Google's Web Search. We hope that this encourages even more publishers to open up more content to users around the world!
  
  Questions and answers about First Click Free
  
  Q: Do the rest of the old guidelines still apply?
  A: Yes, please check the guidelines for Google News as well as the guidelines for Web Search and the associated blog post for more information.
  
  Q: Can I apply First Click Free to only a section of my site / only for Google News (or only for Web Search)?
  A: Sure! Just make sure that both Googlebot and users from the appropriate search results can view the content as required. Keep in mind that showing Googlebot the full content of a page while showing users a registration page would be considered cloaking.
  
  Q: Do I have to sign up to use First Click Free?
  A: Please let us know about your decision to use First Click Free if you are using it for Google News. There's no need to inform us of the First Click Free status for Google Web Search.
  
  Q: What is the preferred way to count a user's accesses?
  A: Since there are many different site architectures, we believe it's best to leave this up to the publisher to decide.
  
  (Please see our related blog post for more information on First Click Free for Google News.)
  
  Posted by John Mueller, Webmaster Trends Analyst, Google Zürich
• 

  GENERIC CIALIS on my website? I think my site has been hacked!

  Posted: November 26th, 2009, 4:23am CET by John Mueller

  Tags:  crawling and indexing  

  How to use "Fetch as Googlebot", part 1
  Webmaster level: Intermediate
  
  Has your site ever dropped suddenly from the index or disappeared mysteriously from search results? Have you ever received a notice that your site is using cloaking techniques? Unfortunately, sometimes a malicious party "hacks" a website: they penetrate the security of a site and insert undesirable content. Sophisticated attackers can camouflage this spammy or dangerous content so that it doesn't appear for normal users, and appears only to Googlebot, which could negatively impact your site in Google's results.
  
  In such cases it used to be very difficult to detect the problem, because the site would appear normal in the eyes of the user. It may be possible that only requests with a User-agent: of Googlebot and coming from Googlebot's IP could see the hidden content. But that's over: with Fetch as Googlebot, the new Labs feature in Webmaster Tools, you can see exactly what Googlebot is seeing, and avoid any kind of cloaking problems. We'll show you how:
  
  Let's imagine that Bob, the administrator of www.example.com, is searching for his site but he finds this instead:
  
  
  
  That's strange, because when he looks at the source code of www.example.com, it looks fine:
  
  
  
  With much surprise Bob may receive a notice from Google warning him that his site is not complying with Google's quality guidelines. Fortunately he has his site registered with Webmaster Tools, let's see how he can check what Googlebot sees:
  
  First Bob logs into Webmaster Tools and selects www.example.com. The Fetch as Googlebot feature will be at the bottom of the navigation menu, in the Labs section:
  
  
  
  The page will contain a field where you can insert the URL to fetch. It can also be left blank to fetch the homepage.
  
  
  
  Bob can simply click Fetch and wait a few seconds. After refreshing the page, he can see the status of the fetch request. If it succeeds, he can click on the "Success" link...
  
  
  
  ...and that will show the details, with the content of the fetched page:
  
  
  
  Aha! There's the spammy content! Now Bob can be certain that www.example.com has been hacked.
  
  Confirming that the website has been hacked (and perhaps is still hacked) is an important step. It is, however, only the beginning. For more information, we strongly suggest getting help from your server administrator or hoster and reading our previous blog posts on the subject of hacked sites:
  
  

  • My site's been hacked - now what?
  
  
  • A quick security checklist for webmasters
  
  
  • Message center warnings for hackable sites
  
  
  • Best practices against hacking

  
  If you have any questions about how to use the Fetch as Googlebot feature, feel free to drop by the Webmaster Help Forum. If you feel that your website might be hacked but are having problems resolving it, you might want to ask the experts in our "Malware and Hacked sites" category.
  
  PS Keep in mind that once you have removed hacked content from your site, it will generally still take time for us to update our search results accordingly. There are a number of factors that affect crawling and indexing of your content so it's impossible to give a time-frame for that.
  
  Written by Javier Tordable, Software Engineer and John Mueller, Webmaster Trends Analyst
• 

  Running desktop and mobile versions of your site

  Posted: November 18th, 2009, 11:15pm CET by Maile Ohye

  Tags:  crawling and indexing  

  (This post was largely translated from our Japanese version of the Webmaster Central Blog )
  
  Recently I introduced several methods to ensure your mobile site is properly indexed by Google. Today I'd like to share information useful for webmasters who manage both desktop and mobile phone versions of a site.
  
  One of the most common problems for webmasters who run both mobile and desktop versions of a site is that the mobile version of the site appears for users on a desktop computer, or that the desktop version of the site appears when someone finds them from a mobile device. In dealing with this scenario, here are two viable options:
  
  Redirect mobile users to the correct version

  When a mobile user or crawler (like Googlebot-Mobile) accesses the desktop version of a URL, you can redirect them to the corresponding mobile version of the same page. Google notices the relationship between the two versions of the URL and displays the standard version for searches from desktops and the mobile version for mobile searches.
  
  If you redirect users, please make sure that the content on the corresponding mobile/desktop URL matches as closely as possible. For example, if you run a shopping site and there's an access from a mobile phone to a desktop-version URL, make sure that the user is redirected to the mobile version of the page for the same product, and not to the homepage of the mobile version of the site. We occasionally find sites using this kind of redirect in an attempt to boost their search rankings, but this practice only results in a negative user experience, and so should be avoided at all costs.
  
  On the other hand, when there's an access to a mobile-version URL from a desktop browser or by our web crawler, Googlebot, it's not necessary to redirect them to the desktop-version. For instance, Google doesn't automatically redirect desktop users from their mobile site to their desktop site, instead they include a link on the mobile-version page to the desktop version. These links are especially helpful when a mobile site doesn't provide the full functionality of the desktop version -- users can easily navigate to the desktop-version if they prefer.

  
  Switch content based on User-agent

  Some sites have the same URL for both desktop and mobile content, but change their format according to User-agent. In other words, both mobile users and desktop users access the same URL (i.e. no redirects), but the content/format changes slightly according to the User-agent. In this case, the same URL will appear for both mobile search and desktop search, and desktop users can see a desktop version of the content while mobile users can see a mobile version of the content.
  
  However, note that if you fail to configure your site correctly, your site could be considered to be cloaking, which can lead to your site disappearing from our search results. Cloaking refers to an attempt to boost search result rankings by serving different content to Googlebot than to regular users. This causes problems such as less relevant results (pages appear in search results even though their content is actually unrelated to what users see/want), so we take cloaking very seriously.
  
  So what does "the page that the user sees" mean if you provide both versions with a URL? As I mentioned in the previous post, Google uses "Googlebot" for web search and "Googlebot-Mobile" for mobile search. To remain within our guidelines, you should serve the same content to Googlebot as a typical desktop user would see, and the same content to Googlebot-Mobile as you would to the browser on a typical mobile device. It's fine if the contents for Googlebot are different from the one for Googlebot-Mobile.
  
  One example of how you could be unintentionally detected for cloaking is if your site returns a message like "Please access from mobile phones" to desktop browsers, but then returns a full mobile version to both crawlers (so Googlebot receives the mobile version). In this case, the page which web search users see (e.g. "Please access from mobile phones") is different from the page which Googlebot crawls (e.g. "Welcome to my site"). Again, we detect cloaking because we want to serve users the same relevant content that Googlebot or Googlebot-Mobile crawled.

  
  Diagram of serving content from your mobile-enabled site
  
  We're working on a daily basis to improve search results and solve problems, but because the relationship between PC and mobile versions of a web site can be nuanced, we appreciate the cooperation of webmasters. Your help will result in more mobile content being indexed by Google, improving the search results provided to users. Thank you for your cooperation in improving the mobile search user experience.
  
  Posted by Jun Mukai, Software Engineer, Mobile Search Team
• 

  Pros and cons of watermarked images

  Posted: November 17th, 2009, 12:40pm CET by Maile Ohye

  Tags:  crawling and indexing  

  Webmaster Level: All
  What's our take on watermarked images for Image Search? It's a complicated topic. I talked with Peter Linsley—my friend at the 'plex, video star, and Product Manager for Image Search—to hear his thoughts.
  Maile: So, Peter... "watermarked images". Can you break it down for us?

  Peter: It's understandable that webmasters find watermarking images beneficial.

  Pros of watermarked images

  • Photographers can claim credit/be recognized for their art.
  • Unknown usage of the image is deterred.

  If search traffic is important to a webmaster, then he/she may also want to consider some of our findings:

  Findings relevant to watermarked images

  • Users prefer large, high-quality images (high-resolution, in-focus).
  • Users are more likely to click on quality thumbnails in search results. Quality pictures (again, high-res and in-focus) often look better at thumbnail size.
  • Distracting features such as loud watermarks, text over the image, and borders are likely to make the image look cluttered when reduced to thumbnail size.

  In summary, if a feature such as watermarking reduces the user-perceived quality of your image or your image's thumbnail, then searchers may select it less often. Preview your images at thumbnail size to get an idea of how the user might perceive it.

  Maile: Ahh, I see: Webmasters concerned with search traffic likely want to balance the positives of watermarking with the preferences of their users -- keeping in mind that sites that use clean images without distracting artifacts tend to be more popular, and that this can also impact rankings. Will Google rank an image differently just because it's watermarked?

  Peter: Nope. The presence of a watermark doesn't itself cause an image to be ranked higher or lower.

  
  Do you have questions or opinions on the topic? Let's chat in the webmaster forum.
  Written by Maile Ohye, Developer Programs Tech Lead
• 

  Help Google index your mobile site

  Posted: November 13th, 2009, 2:48pm CET by Maile Ohye

  Tags:  crawling and indexing  

  (This post was largely translated from our Japanese Webmaster Central Blog.)
  It seems the world is going mobile, with many people using mobile phones on a daily basis, and a large user base searching on Google’s mobile search page. However, as a webmaster, running a mobile site and tapping into the mobile search audience isn't easy. Mobile sites not only use a different format from normal desktop site, but the management methods and expertise required are also quite different. This results in a variety of new challenges. As a mobile search engineer, it's clear to me that while many mobile sites were designed with mobile viewing in mind, they weren’t designed to be search friendly. I'd like to help ensure that your mobile site is also available for users of mobile search.
  Here are troubleshooting tips to help ensure that your site is properly crawled and indexed:
  Verify that your mobile site is indexed by Google
  If your web site doesn't show up in the results of a Google mobile search even using the 'site:' operator, it may be that your site has one or both of the following issues:

  Googlebot may not be able to find your site
  Googlebot, our crawler, must crawl your site before it can be included in our search index. If you just created the site, we may not yet be aware of it. If that's the case, create a Mobile Sitemap and submit it to Google to inform us to the site’s existence. A Mobile Sitemap can be submitted using Google Webmaster Tools, in the same way as with a standard Sitemap.

  Googlebot may not be able to access your site
  Some mobile sites refuse access to anything but mobile phones, making it impossible for Googlebot to access the site, and therefore making the site unsearchable. Our crawler for mobile sites is "Googlebot-Mobile". If you'd like your site crawled, please allow any User-agent including "Googlebot-Mobile" to access your site. You should also be aware that Google may change its User-agent information at any time without notice, so it is not recommended that you check if the User-agent exactly matches "Googlebot-Mobile" (which is the string used at present). Instead, check whether the User-agent header contains the string "Googlebot-Mobile". You can also use DNS Lookups to verify Googlebot.

  
  Verify that Google can recognize your mobile URLs
  Once Googlebot-Mobile crawls your URLs, we then check for whether the URL is viewable on a mobile device. Pages we determine aren't viewable on a mobile phone won't be included in our mobile site index (although they may be included in the regular web index). This determination is based on a variety of factors, one of which is the "DTD (Doc Type Definition)" declaration. Check that your mobile-friendly URLs' DTD declaration is in an appropriate mobile format such as XHTML Mobile or Compact HTML. If it's in a compatible format, the page is eligible for the mobile search index. For more information, see the Mobile Webmaster Guidelines.
  If you have any question regarding mobile site, post your question to our Webmaster Help Forum and webmasters around the world as well as we are happy to help you with your problem.
  Posted by Jun Mukai, Software Engineer, Mobile Search Team
• 

  New personalization features in Google Friend Connect

  Posted: November 4th, 2009, 8:40am CET by Maile Ohye

  Tags:  products and services  

  Webmaster Level: All
  Just a few weeks ago, we made Google Friend Connect a lot easier to use by dramatically simplifying the setup process. Today, we're excited to announce several new features that make it possible for website owners to get to know their users, encourage users to get to know each other, and match their site content (including Google ads) to visitors' interests.
  
  To learn more about these new features, check out the Google Social Web Blog.
  Posted by Mussie Shore, Product Manager, Google Friend Connect
• 

  Using RSS/Atom feeds to discover new URLs

  Posted: October 29th, 2009, 6:50pm CET by Maile Ohye

  Tags:  crawling and indexing  

  Webmaster Level: Intermediate
  
  Google uses numerous sources to find new webpages, from links we find on the web to submitted URLs. We aim to discover new pages quickly so that users can find new content in Google search results soon after they go live. We recently launched a feature that uses RSS and Atom feeds for the discovery of new webpages.
  
  RSS/Atom feeds have been very popular in recent years as a mechanism for content publication. They allow readers to check for new content from publishers. Using feeds for discovery allows us to get these new pages into our index more quickly than traditional crawling methods. We may use many potential sources to access updates from feeds including Reader, notification services, or direct crawls of feeds. Going forward, we might also explore mechanisms such as PubSubHubbub to identify updated items.
  
  In order for us to use your RSS/Atom feeds for discovery, it's important that crawling these files is not disallowed by your robots.txt. To find out if Googlebot can crawl your feeds and find your pages as fast as possible, test your feed URLs with the robots.txt tester in Google Webmaster Tools.
  
  Written by Raymond Lo, Guhan Viswanathan, and Dave Weissman, Crawl and Indexing Team
• 

  Help us make the web better: An update on Rich Snippets

  Posted: October 26th, 2009, 3:00pm CET by Maile Ohye

  Tags:  crawling and indexing  

  Webmaster Level: All
  
  In May this year we announced Rich Snippets which makes it possible to show structured data from your pages on Google's search results.
  
  
  We're convinced that structured data makes the web better, and we've worked hard to expand Rich Snippets to more search results and collect your feedback along the way. If you have review or people/social networking content on your site, it's easier than ever to mark up your content using microformats or RDFa so that Google can better understand it to generate useful Rich Snippets. Here are a few helpful improvements on our end to enable you to mark up your content:
  
  Testing tool. See what Google is able to extract, and preview how microformats or RDFa marked-up pages would look on Google search results. Test your URLs on the Rich Snippets Testing Tool.
  
  
  Google Custom Search users can also use the Rich Snippets Testing Tool to test markup usable in their Custom Search engine.
  
  Better documentation. We've extended our documentation to include a new section containing Tips & Tricks and Frequently Asked Questions. Here we have responded to common points of confusion and provided instructions on how to maximize the chances of getting Rich Snippets for your site.
  
  Extended RDFa support. In addition to the Person RDFa format, we have added support for the corresponding fields from the FOAF and vCard vocabularies for all those of you who asked for it.
  
  Videos. If you have videos on your page, you can now mark up your content to help Google find those videos.
  
  As before, marking up your content does not guarantee that Rich Snippets will be shown for your site. We will continue to expand this feature gradually to ensure a great user experience whenever Rich Snippets are shown in search results.
  
  Written by Kavi Goel, Pravir Gupta, and Othar Hansson
• 

  One million YouTube views!

  Posted: October 21st, 2009, 2:34pm CEST by Michael Wyszomierski

  Tags:  feedback and communication  

  Earlier this year, we launched our very own Webmaster Central channel on YouTube. Just today, we saw our total video views exceed one million! On the road to this milestone, we uploaded 154 videos, for a total of nearly 11 hours of webmaster-focused media. These videos have brought you conference presentations, updates on tools for webmasters, general tips, and of course answers to your "Grab bag" questions for Matt Cutts.
  
  To celebrate our one million views, we're sharing a fun video with you in which Matt Cutts shows us what happened when he lost a bet with his team:
  
  
  
  We're also pleased to announce that we've added captions to all of our videos and plan to do so for our future videos as well. Thank you to everyone who has watched, shared, and commented on our videos. We look forward to the next million views!
  
  Posted by Michael Wyszomierski, Search Quality Team
• 

  Managing your reputation through search results

  Posted: October 15th, 2009, 5:00pm CEST by Susan Moskwa

  Tags:  crawling and indexing  

  (Cross-posted on the Official Google Blog)
  
  A few years ago I couldn't wait to get married. Because I was in love, yeah; but more importantly, so that I could take my husband's name and people would stop getting that ridiculous picture from college as a top result when they searched for me on Google.
  
  After a few years of working here, though, I've learned that you don't have to change your name just because it brings up some embarrassing search results. Below are some tips for "reputation management": influencing how you're perceived online, and what information is available relating to you.
  
  Think twice
  
  The first step in reputation management is preemptive: Think twice before putting your personal information online. Remember that although something might be appropriate for the context in which you're publishing it, search engines can make it very easy to find that information later, out of context, including by people who don't normally visit the site where you originally posted it. Translation: don't assume that just because your mom doesn't read your blog, she'll never see that post about the new tattoo you're hiding from her.
  
  Tackle it at the source
  
  If something you dislike has already been published, the next step is to try to remove it from the site where it's appearing. Rather than immediately contacting Google, it's important to first remove it from the site where it's being published. Google doesn't own the Internet; our search results simply reflect what's already out there on the web. Whether or not the content appears in Google's search results, people are still going to be able to access it — on the original site, through other search engines, through social networking sites, etc. — if you don't remove it from the original site. You need to tackle this at the source.

  • If the content in question is on a site you own, easy — just remove it. It will naturally drop out of search results after we recrawl the page and discover the change.
  • It's also often easy to remove content from sites you don't own if you put it there, such as photos you've uploaded, or content on your profile page.
  • If you can't remove something yourself, you can contact the site's webmaster and ask them to remove the content or the page in question.

  After you or the site's webmaster has removed or edited the page, you can expedite the removal of that content from Google using our URL removal tool.
  
  Proactively publish information
  
  Sometimes, however, you may not be able to get in touch with a site's webmaster, or they may refuse to take down the content in question. For example, if someone posts a negative review of your business on a restaurant review or consumer complaint site, that site might not be willing to remove the review. If you can't get the content removed from the original site, you probably won't be able to completely remove it from Google's search results, either. Instead, you can try to reduce its visibility in the search results by proactively publishing useful, positive information about yourself or your business. If you can get stuff that you want people to see to outperform the stuff you don't want them to see, you'll be able to reduce the amount of harm that that negative or embarrassing content can do to your reputation.
  
  You can publish or encourage positive content in a variety of ways:

  • Create a Google profile. When people search for your name, Google can display a link to your Google profile in our search results and people can click through to see whatever information you choose to publish in your profile.
  • If a customer writes a negative review of your business, you could ask some of your other customers who are happy with your company to give a fuller picture of your business.
  • If a blogger is publishing unflattering photos of you, take some pictures you prefer and publish them in a blog post or two.
  • If a newspaper wrote an article about a court case that put you in a negative light, but which was subsequently ruled in your favor, you can ask them to update the article or publish a follow-up article about your exoneration. (This last one may seem far-fetched, but believe it or not, we've gotten multiple requests from people in this situation.)

  Hope these tips have been helpful! Feel free to stop by our Web Search Forum and share your own advice or stories about how you manage your reputation online.
  
  Posted by Susan Moskwa, Webmaster Trends Analyst
• 

  A proposal for making AJAX crawlable

  Posted: October 7th, 2009, 12:51pm CEST by John Mueller

  Tags:  crawling and indexing  

  Webmaster level: Advanced
  
  Today we're excited to propose a new standard for making AJAX-based websites crawlable. This will benefit webmasters and users by making content from rich and interactive AJAX-based websites universally accessible through search results on any search engine that chooses to take part. We believe that making this content available for crawling and indexing could significantly improve the web.
  
  While AJAX-based websites are popular with users, search engines traditionally are not able to access any of the content on them. The last time we checked, almost 70% of the websites we know about use JavaScript in some form or another. Of course, most of that JavaScript is not AJAX, but the better that search engines could crawl and index AJAX, the more that developers could add richer features to their websites and still show up in search engines.
  
  Some of the goals that we wanted to achieve with this proposal were:
  

  • Minimal changes are required as the website grows
  
  
  • Users and search engines see the same content (no cloaking)
  
  
  • Search engines can send users directly to the AJAX URL (not to a static copy)
  
  
  • Site owners have a way of verifying that their AJAX website is rendered correctly and thus that the crawler has access to all the content

  
  
  Here's how search engines would crawl and index AJAX in our initial proposal:
  

  • Slightly modify the URL fragments for stateful AJAX pages
    Stateful AJAX pages display the same content whenever accessed directly. These are pages that could be referred to in search results. Instead of a URL like http://example.com/page?query#state we would like to propose adding a token to make it possible to recognize these URLs: http://example.com/page?query#[FRAGMENTTOKEN]state . Based on a review of current URLs on the web, we propose using "!" (an exclamation point) as the token for this. The proposed URL that could be shown in search results would then be: http://example.com/page?query#!state.
  
  
  • Use a headless browser that outputs an HTML snapshot on your web server
    The headless browser is used to access the AJAX page and generates HTML code based on the final state in the browser. Only specially tagged URLs are passed to the headless browser for processing. By doing this on the server side, the website owner is in control of the HTML code that is generated and can easily verify that all JavaScript is executed correctly. An example of such a browser is HtmlUnit, an open-sourced "GUI-less browser for Java programs.
  
  
  • Allow search engine crawlers to access these URLs by escaping the state
    As URL fragments are never sent with requests to servers, it's necessary to slightly modify the URL used to access the page. At the same time, this tells the server to use the headless browser to generate HTML code instead of returning a page with JavaScript. Other, existing URLs - such as those used by the user - would be processed normally, bypassing the headless browser. We propose escaping the state information and adding it to the query parameters with a token. Using the previous example, one such URL would be http://example.com/page?query&[QUERYTOKEN]=state . Based on our analysis of current URLs on the web, we propose using "_escaped_fragment_" as the token. The proposed URL would then become http://example.com/page?query&_escaped_fragment_=state .
  
  
  • Show the original URL to users in the search results
    To improve the user experience, it makes sense to refer users directly to the AJAX-based pages. This can be achieved by showing the original URL (such as http://example.com/page?query#!state from our example above) in the search results. Search engines can check that the indexable text returned to Googlebot is the same or a subset of the text that is returned to users.

  
  
  
  (Graphic by Katharina Probst)
  
  In summary, starting with a stateful URL such as
  http://example.com/dictionary.html#AJAX , it could be available to both crawlers and users as
  http://example.com/dictionary.html#!AJAX which could be crawled as
  http://example.com/dictionary.html?_escaped_fragment_=AJAX which in turn would be shown to users and accessed as
  http://example.com/dictionary.html#!AJAX
  
  -->View the presentation
  
  We're currently working on a proposal and a prototype implementation. Feedback is very welcome — please add your comments below or in our Webmaster Help Forum. Thank you for your interest in making the AJAX-based web accessible and useful through search engines!
  
  Proposal by Katharina Probst, Bruce Johnson, Arup Mukherjee, Erik van der Poel and Li Xiao, Google
  Blog post by John Mueller, Webmaster Trends Analyst, Google Zürich
• 

  Reunifying duplicate content on your website

  Posted: October 6th, 2009, 5:14pm CEST by John Mueller

  Tags:  crawling and indexing  

  Handling duplicate content within your own website can be a big challenge. Websites grow; features get added, changed and removed; content comes—content goes. Over time, many websites collect systematic cruft in the form of multiple URLs that return the same contents. Having duplicate content on your website is generally not problematic, though it can make it harder for search engines to crawl and index the content. Also, PageRank and similar information found via incoming links can get diffused across pages we aren't currently recognizing as duplicates, potentially making your preferred version of the page rank lower in Google.
  
  Steps for dealing with duplicate content within your website
  

  1. Recognize duplicate content on your website.
     The first and most important step is to recognize duplicate content on your website. A simple way to do this is to take a unique text snippet from a page and to search for it, limiting the results to pages from your own website by using a site:query in Google. Multiple results for the same content show duplication you can investigate.
     
  
  
  2. Determine your preferred URLs.
     Before fixing duplicate content issues, you'll have to determine your preferred URL structure. Which URL would you prefer to use for that piece of content?
     
  
  
  3. Be consistent within your website.
     Once you've chosen your preferred URLs, make sure to use them in all possible locations within your website (including in your Sitemap file).
     
  
  
  4. Apply 301 permanent redirects where necessary and possible.
     If you can, redirect duplicate URLs to your preferred URLs using a 301 response code. This helps users and search engines find your preferred URLs should they visit the duplicate URLs. If your site is available on several domain names, pick one and use the 301 redirect appropriately from the others, making sure to forward to the right specific page, not just the root of the domain. If you support both www and non-www host names, pick one, use the preferred domain setting in Webmaster Tools, and redirect appropriately.
     
  
  
  5. Implement the rel="canonical" link element on your pages where you can.
     Where 301 redirects are not possible, the rel="canonical" link element can give us a better understanding of your site and of your preferred URLs. The use of this link element is also supported by major search engines such as Ask.com, Bing and Yahoo!.
     
  
  
  6. Use the URL parameter handling tool in Google Webmaster Tools where possible.
     If some or all of your website's duplicate content comes from URLs with query parameters, this tool can help you to notify us of important and irrelevant parameters within your URLs. More information about this tool can be found in our announcement blog post.
     

  
  What about the robots.txt file?
  
  One item which is missing from this list is disallowing crawling of duplicate content with your robots.txt file. We now recommend not blocking access to duplicate content on your website, whether with a robots.txt file or other methods. Instead, use the rel="canonical" link element, the URL parameter handling tool, or 301 redirects. If access to duplicate content is entirely blocked, search engines effectively have to treat those URLs as separate, unique pages since they cannot know that they're actually just different URLs for the same content. A better solution is to allow them to be crawled, but clearly mark them as duplicate using one of our recommended methods. If you allow us to crawl these URLs, Googlebot will learn rules to identify duplicates just by looking at the URL and should largely avoid unnecessary recrawls in any case. In cases where duplicate content still leads to us crawling too much of your website, you can also adjust the crawl rate setting in Webmaster Tools.
  
  We hope these methods will help you to master the duplicate content on your website! Information about duplicate content in general can also be found in our Help Center. Should you have any questions, feel free to join the discussion in our Webmaster Help Forum.
  
  Posted by John Mueller, Webmaster Trends Analyst, Google Zürich
• 

  Translate your website with Google: Expand your audience globally

  Posted: September 30th, 2009, 4:54pm CEST by Susan Moskwa

  Tags:  products and services  

  (This has been cross-posted from the Official Google Blog)
  
  How long would it take to translate all the world's web content into 50 languages? Even if all of the translators in the world worked around the clock, with the current growth rate of content being created online and the sheer amount of data on the web, it would take hundreds of years to make even a small dent.
  
  Today, we're happy to announce a new website translator gadget powered by Google Translate that enables you to make your site's content available in 51 languages. Now, when people visit your page, if their language (as determined by their browser settings) is different than the language of your page, they'll be prompted to automatically translate the page into their own language. If the visitor's language is the same as the language of your page, no translation banner will appear.
  
  
  After clicking the Translate button, the automatic translations are shown directly on your page.
  
  
  It's easy to install — all you have to do is cut and paste a short snippet into your webpage to increase the global reach of your blog or website.
  
  
  Automatic translation is convenient and helps people get a quick gist of the page. However, it's not a perfect substitute for the art of professional translation. Today happens to be International Translation Day, and we'd like to take the opportunity to celebrate the contributions of translators all over the world. These translators play an essential role in enabling global communication, and with the rapid growth and ease of access to digital content, the need for them is greater than ever. We hope that professional translators, along with translation tools such as Google Translator Toolkit and this Translate gadget, will continue to help make the world's content more accessible to everyone.
  
  Posted by Jeff Chin, Product Manager, Google Translate
• 

  Spanish Site Clinic now live

  Posted: September 17th, 2009, 8:09am CEST by Cristina, Equipo de Calidad de Búsqueda

  Tags:  feedback and communication  

  The Google Webmaster Central blog in Spanish has launched a Site Clinic especially for the Spanish-speaking market. We're offering to analyze a series of websites in order to share some best practices with our community using real web sites. The plan is to offer constructive advice on accessibility and improvements that can lead to better visibility in Google's search results.
  During this month, we will be receiving submissions from any legitimate website, but it must be primarily in Spanish. So before you submit your site, please visit the original post and if you want to participate fill out the form as soon as possible, because we will only be selecting 3-5 websites from the first 200 submitted for this Site Clinic, so don't miss out!
  
  Posted by Esperanza, Search Quality Team
• 

  Supporting Facebook Share and RDFa for videos

  Posted: September 14th, 2009, 1:30pm CEST by Michael Wyszomierski

  Tags:  crawling and indexing  

  Have you ever wondered how to increase the chances of your videos appearing in Google's results? Over the last year, the Video Search team has been working hard to improve our index of video on the Web. Today, we're beginning the first in a series of posts to explain some best practices for sites hosting video content.
  We previously talked about the importance of submitting a Video Sitemap or mRSS feed to Google and following Google's webmaster guidelines. However, we wanted to offer webmasters an additional tool, so today we're taking a page from the rich snippets playbook and announcing support for Facebook Share and Yahoo! SearchMonkey RDFa. Both of these markup formats allow you to specify information essential to video indexing, such as a video's title and description, within the HTML of a video page. While we've become smarter at discovering this information on our own, we'd certainly appreciate some hints directly from webmasters. Also, to maximize the chances that we find the markup on your video pages, you should make sure it appears in the HTML without the execution of JavaScript or Flash.
  So, check out Facebook Share and RDFa and help Google find your videos!
  
  Facebook Share:
  

  <meta name="title" content="Baroo? - cute puppies" />
  <meta name="description" content="The cutest canine head tilts on the Internet!" />
  <link rel="image_src" href="http://example.com/thumbnail_preview.jpg" />
  <link rel="video_src" href="http://example.com/video_object.swf?id=12345"/>
  <meta name="video_height" content="296" />
  <meta name="video_width" content="512" />
  <meta name="video_type" content="application/x-shockwave-flash" />

  RDFa (Yahoo! SearchMonkey):
  

  <object width="512" height="296" rel="media:video"
  resource="http://example.com/video_object.swf?id=12345"
  xmlns:media="http://search.yahoo.com/searchmonkey/media/"
  xmlns:dc="http://purl.org/dc/terms/">
  <param name="movie" value="http://example.com/video_object.swf?id=12345" />
  <embed src="http://example.com/video_object.swf?id=12345"
  type="application/x-shockwave-flash" width="512" height="296"></embed>
  <a rel="media:thumbnail" href="http://example.com/thumbnail_preview.jpg" />
  <a rel="dc:license" href="http://example.com/terms_of_service.html" />
  <span property="dc:description" content="Cute Overload defines Baroo? as: Dogspeak for 'Whut the...?'
  Frequently accompanied by the Canine Tilt and/or wrinkled brow for enhanced effect." />
  <span property="media:title" content="Baroo? - cute puppies" />
  <span property="media:width" content="512" />
  <span property="media:height" content="296" />
  <span property="media:type" content="application/x-shockwave-flash" />
  <span property="media:region" content="us" />
  <span property="media:region" content="uk" />
  <span property="media:duration" content="63" />
  </object>
  

  Posted by Michael Cohen, Product Manager, Video Search Team
• 

  New tools for Google Services for Websites

  Posted: August 11th, 2009, 1:05pm CEST by Maile Ohye

  Tags:  products and services  

  Webmaster Level: All
  (A nearly duplicate version :) cross-posted on the Official Google Blog)
  
  Earlier this year, we launched Google Services for Websites, a program that helps partners, e.g., web hoster and access providers, offer useful and powerful tools to their customers. By making services, such as Webmaster Tools, Custom Search, Site Search and AdSense, easily accessible via the hoster control panel, hosters can easily enable these services for their webmasters. The tools help website owners understand search performance, improve user retention and monetize their content — in other words, run more effective websites.
  
  Since we launched the program, several hosting platforms have enhanced their offerings by integrating with the appropriate APIs. Webmasters can configure accounts, submit Sitemaps with Webmaster Tools, create Custom Search Boxes for their sites and monetize their content with AdSense, all with a few clicks at their hoster control panel. More partners are in the process of implementing these enhancements.
  
  We've just added new tools to the suite:

  • Web Elements allows your customers to enhance their websites with the ease of cut-and-paste. Webmasters can provide maps, real-time news, calendars, presentations, spreadsheets and YouTube videos on their sites. With the Conversation Element, websites can create more engagement with their communities. The Custom Search Element provides inline search over your own site (or others you specify) without having to write any code and various options to customize further.
  • Page Speed allows webmasters to measure the performance of their websites. Snappier websites help users find things faster; the recommendations from these latency tools allow hosters and webmasters to optimize website speed. These techniques can help hosters reduce resource use and optimize network bandwidth.
  • The Tips for Hosters page offers a set of tips for hosters for creating a richer website hosting platform. Hosters can improve the convenience and accessibility of tools, while at the same time saving platform costs and earning referral fees. Tips include the use of analytics tools such as Google Analytics to help webmasters understand their traffic and linguistic tools such as Google Translate to help websites reach a broader audience.

  If you're a hoster and would like to participate in the Google Services for Websites program, please apply here. You'll have to integrate with the service APIs before these services can be made available to your customers, so the earlier you start that process, the better.
  
  And if your hosting service doesn't have Google Services for Websites yet, send them to this page. Once they become a partner, you can quickly configure the services you want at your hoster's control panel (without having to come to Google).
  
  As always, we'd love to get feedback on how the program is working for you, and what improvements you'd like to see.
  
  Posted by Rajat Mukherjee, Group Product Manager
• 

  Optimize your crawling & indexing

  Posted: August 10th, 2009, 12:40am CEST by Susan Moskwa

  Tags:  crawling and indexing  

  Webmaster Level: Intermediate to Advanced
  
  Many questions about website architecture, crawling and indexing, and even ranking issues can be boiled down to one central issue: How easy is it for search engines to crawl your site? We've spoken on this topic at a number of recent events, and below you'll find our presentation and some key takeaways on this topic.
  
  
  
  The Internet is a big place; new content is being created all the time. Google has a finite number of resources, so when faced with the nearly-infinite quantity of content that's available online, Googlebot is only able to find and crawl a percentage of that content. Then, of the content we've crawled, we're only able to index a portion.
  
  URLs are like the bridges between your website and a search engine's crawler: crawlers need to be able to find and cross those bridges (i.e., find and crawl your URLs) in order to get to your site's content. If your URLs are complicated or redundant, crawlers are going to spend time tracing and retracing their steps; if your URLs are organized and lead directly to distinct content, crawlers can spend their time accessing your content rather than crawling through empty pages, or crawling the same content over and over via different URLs.
  
  In the slides above you can see some examples of what not to do—real-life examples (though names have been changed to protect the innocent) of homegrown URL hacks and encodings, parameters masquerading as part of the URL path, infinite crawl spaces, and more. You'll also find some recommendations for straightening out that labyrinth of URLs and helping crawlers find more of your content faster, including:
  

  • Remove user-specific details from URLs.
    URL parameters that don't change the content of the page—like session IDs or sort order—can be removed from the URL and put into a cookie. By putting this information in a cookie and 301 redirecting to a "clean" URL, you retain the information and reduce the number of URLs pointing to that same content.
  • Rein in infinite spaces.
    Do you have a calendar that links to an infinite number of past or future dates (each with their own unique URL)? Do you have paginated data that returns a status code of 200 when you add &page=3563 to the URL, even if there aren't that many pages of data? If so, you have an infinite crawl space on your website, and crawlers could be wasting their (and your!) bandwidth trying to crawl it all. Consider these tips for reining in infinite spaces.
  • Disallow actions Googlebot can't perform.
    Using your robots.txt file, you can disallow crawling of login pages, contact forms, shopping carts, and other pages whose sole functionality is something that a crawler can't perform. (Crawlers are notoriously cheap and shy, so they don't usually "Add to cart" or "Contact us.") This lets crawlers spend more of their time crawling content that they can actually do something with.
  • One man, one vote. One URL, one set of content.
    In an ideal world, there's a one-to-one pairing between URL and content: each URL leads to a unique piece of content, and each piece of content can only be accessed via one URL. The closer you can get to this ideal, the more streamlined your site will be for crawling and indexing. If your CMS or current site setup makes this difficult, you can use the rel=canonical element to indicate the preferred URL for a particular piece of content.

  
  If you have further questions about optimizing your site for crawling and indexing, check out some of our previous writing on the subject, or stop by our Help Forum.
  
  Posted by Susan Moskwa, Webmaster Trends Analyst
• 

  Google Friend Connect - Now in 47 new languages

  Posted: July 14th, 2009, 1:05am CEST by Maile Ohye

  Tags:  products and services  

  
  Have you been holding off on using Google Friend Connect because your site isn't in English? Starting today, Friend Connect is available in 47 new languages, including French, Italian, German, Spanish, Chinese, Japanese, Hindi and Portuguese. Now you can easily add social features that match the language of other content on your site.
  
  Most Google-created gadgets (such as the members, comments, and recommendation gadgets) are now available in these new languages. Some developers have also created gadgets that support additional languages and we hope that there will be more to come in the future. To see a list of gadgets available in your language, visit the gadget gallery.
  
  When you add Friend Connect to a new site, it will default to your primary language. But if your site is in another language, simply select it on the site settings tab and Friend Connect will automatically render the gadgets in that language. And if have multiple sites in different languages, you can select a different language for each of your sites.
  
  To learn more or see the full list of languages, check out the Social Web Blog.
  
  Posted by Mendel Chuang, Product Marketing Manager, Google Friend Connect
• 

  Webmaster Central YouTube update for July 6th - 10th

  Posted: July 13th, 2009, 7:55pm CEST by Michael Wyszomierski

  Tags:  feedback and communication  

  Want to see what's new on the Webmaster Central YouTube channel? Check out the answers to the latest Grab Bag questions:

  • How do meta geo tags influence the search results?
  • Will Google find text in images someday?
  • Is Google doing away with use of the meta description?
  • "Query deserves freshness." Fact or fiction?
  • Which is more reliable for searching: Google or Twitter?

  Below is Matt's clarification about Google's use of the meta description tag:
  
  Feel free to leave comments letting us know how you liked the videos, and if you have any specific questions, ask the experts in the Webmaster Help Forum.
  Posted by Michael Wyszomierski, Search Quality Team
• 

  Google Maps gadget: Help customers find their way

  Posted: July 7th, 2009, 6:30pm CEST by Maile Ohye

  Tags:  products and services  

  Webmaster Level: All
  
  With the new directions gadget from Google Maps, any business can offer customized Google Maps directions to their business locations. With the directions gadget, you no longer need to type and update multiple sets of text directions. Let's face it: customers are only looking for directions from their specific location.
  
  Last week, I looked up directions to the hotel in Sacramento that I had booked for the 4th of July weekend. As I had never been to that part of the state before, I was puzzled by the limited directions offered by their website - I had no idea whether I was approaching from the North, South, East or West or where the major highways were. What I needed were step-by-step directions from my exact starting point to the hotel that I could easily print and go.
  
  Google has made this process easy for you. By copying and pasting a single line of code, any website can offer customized door-to-door directions powered by Google Maps to their users.
  
  
  The gadget allows you to pre-fill the "To" field with one or multiple addresses. Customers are able to print their directions with a single click. And for those who prefer not to drive, the gadget also provides walking and public transit directions.
  
  
  If someone enters a vague starting address, they have the option to specify a more exact address, which will then fly into the "From" field.
  
  
  Providing directions from Google Maps is very flexible; in addition to run-of-the-mill addresses, you can give your customers directions to everything from a generic ZIP code to a specific set of latitude-longitude coordinates for any of your locations.
  
  Example destinations:
  123 River St, Woods, MA
  90210
  42.06782° N, 71.756963° W
  
  You can also give your address an alias, or a name that everyone will understand. You can do this by putting the alias in parentheses just after the address:
  
  42.06782° N, 71.756963° W (Grandma's House)
  
  
  Take a look at how Legoland California, Emeril Lagasse, and Harvard University are using the gadget. And then test and create your own directions gadget here. For the many locations outside of the US, the gadget is available in 23 different languages.
  
  Written by Julie Zhou, Product Marketing Manager, Google Maps
• 

  Webmaster Central YouTube update for June 29th - July 3rd

  Posted: July 7th, 2009, 4:03pm CEST by Michael Wyszomierski

  Tags:  feedback and communication  

  Want to see what's new on the Webmaster Central YouTube channel? Check out the answers to the latest Grab Bag questions:

  • How can I optimize my site on a small budget?
  • When will the data in Webmaster Tools be improved?
  • Can my blogroll affect my blog's reputation in Google?
  • Will a link to a page disallowed in robots.txt transfer PageRank?

  Do you run a small business? Check out Matt's tips on how to attract visitors on a small budget:
  
  Feel free to leave comments letting us know how you liked the videos, and if you have any specific questions, ask the experts in the Webmaster Help Forum.
  Posted by Michael Wyszomierski, Search Quality Team
• 

  Webmaster Central YouTube update for June 22nd - 26th

  Posted: June 30th, 2009, 12:22pm CEST by Michael Wyszomierski

  Tags:  feedback and communication  

  Want to know what's new on the Webmaster Central YouTube channel? Here's what we've uploaded in the past week:
  As part of Google's goal to make the web faster, we uploaded several video tips about optimizing the speed of your website. Check out the tutorials page to view the tutorials and associated videos.
  Matt Cutts answered a new question each day from the Grab Bag:

  • What are some best practices for moving to a new CMS?
  • Is over-optimization bad for a website?
  • Will having my software listed in low-quality directories harm my website's ranking?
  • Can the geographic location of a web server affect SEO?
  • Will I be penalized if all of my Sitemap URLs have the same priority?

  And during Adam Lasnik's visit to India, he was interviewed by Webmaster Help Forum guide Jayan Tharayil about issues related to webmasters in India. We have the full three-part interview right here.
  We'll get you started on this batch of videos with Matt's tips for targeting your site to a specific region:
  
  Feel free to leave comments letting us know how you liked the videos, and if you have any specific questions, ask the experts in the Webmaster Help Forum.
  Posted by Michael Wyszomierski, Search Quality Team
• 

  Traffic drops and site architecture issues

  Posted: June 28th, 2009, 5:39am CEST by Luisella Mazza

  Tags:  crawling and indexing  

  Webmaster Level: Intermediate.
  
  We hear lots of questions about site architecture issues and traffic drops, so it was a pleasure to talk about it in greater detail at SMX London and I'd like to highlight some key concepts from my presentation here. First off, let's gain a better understanding of drops in traffic, and then we'll take a look at site design and architecture issues.
  
  Understanding drops in traffic
  
  As you know, fluctuations in search results happen all the time; the web is constantly evolving and so is our index. Improvements in our ability to understand our users' interests and queries also often lead to differences in how our algorithms select and rank pages. We realize, however, that such changes might be confusing and sometimes foster misconceptions, so we'd like to address a couple of these myths head-on.
  
  Myth number 1: Duplicate content causes drops in traffic!
  Webmasters often wonder if the duplicates on their site can have a negative effect on their site's traffic. As mentioned in our guidelines, unless this duplication is intended to manipulate Google and/or users, the duplication is not a violation of our Webmaster Guidelines. The second part of my presentation illustrates in greater detail how to deal with duplicate content using canonicalization.
  
  Myth number 2: Affiliate programs cause drops in traffic!
  Original and compelling content is crucial for a good user experience. If your website participates in affiliate programs, it's essential to consider whether the same content is available in many other places on the web. Affiliate sites with little or no original and compelling content are not likely to rank well in Google search results, but including affiliate links within the context of original and compelling content isn't in itself the sort of thing that leads to traffic drops.
  
  Having reviewed a few of the most common concerns, I'd like to highlight two important sections of the presentation. The first illustrates how malicious attacks -- such as an injection of hidden text and links -- might cause your site to be removed from Google's search results. On a happier note, it also covers how you can use the Google cache and Webmaster Tools to identify this issue. On a related note, if we've found a violation of the Webmaster Guidelines such as the use of hidden text or the presence of malware on your site, you will typically find a note regarding this in your Webmaster Tools Message center.
  You may also find your site's traffic decreased if your users are being redirected to another site...for example, due to a hacker-applied server- or page-level redirection triggered by referrals from search engines. A similar scenario -- but with different results -- is the case in which a hacker has instituted a redirection for crawlers only. While this will cause no immediate drop in traffic since users and their visits are not affected, it might lead to a decrease in pages indexed over time.
  
  
  
  
  
  Site design and architecture issues
  Now that we've seen how malicious changes might affect your site and its traffic, let's examine some design and architecture issues. Specifically, you want to ensure that your site is able to be both effectively crawled and indexed, which is the prerequisite to being shown in our search results. What should you consider?
  
  

  • First off, check that your robots.txt file has the correct status code and is not returning an error.
  • Keep in mind some best practices when moving to a new site and the new "Change of address" feature recently added to Webmaster Tools.
  • Review the settings of the robots.txt file to make sure no pages -- particularly those rewritten and/or dynamic -- are blocked inappropriately.
  • Finally, make good use of the rel="canonical" attribute to reduce the indexing of duplicate content on your domain. The example in the presentation shows how using this attribute helps Google understand that a duplicate can be clustered with the canonical and that the original, or canonical, page should be indexed.

  
  
  
  In conclusion, remember that fluctuations in search results are normal but there are steps that you can take to avoid malicious attacks or design and architecture issues that might cause your site to disappear or fluctuate unpredictably in search results. Start by learning more about attacks by hackers and spammers, make sure everything is running properly at crawling and indexing level by double-checking the HTML suggestions in Webmaster Tools, and finally, test your robots.txt file in case you are accidentally blocking Googlebot. And don't forget about those "robots.txt unreachable" errors!
  
  Written by Luisella Mazza, Search Quality Senior Analyst
• 

  Webmaster Central YouTube update for June 15th - 19th

  Posted: June 22nd, 2009, 5:36pm CEST by Michael Wyszomierski

  Tags:  feedback and communication  

  Want to know what's new on the Webmaster Central YouTube channel? Here's what we've uploaded in the past week:
  
  Maile Ohye gave a webmaster-focused presentation about Product Search.
  
  Matt Cutts answered a new question each day from the Grab Bag:
  

  • Do dates in URLs determine freshness?
  • What impact does "page bloat" have on Google rankings?
  • What types of directories are seen as sources of paid links?
  • Should I include my logo text using the 'alt' attribute or CSS?
  • What's the preferred way to check for links to my site?

  To get you started on watching this latest batch of videos, here's Matt's answer about directories and paid links:
  
  
  
  Feel free to leave comments letting us know how you liked the videos, and if you have any specific questions, ask the experts in the Webmaster Help Forum.
  
  Posted by Michael Wyszomierski, Search Quality Team
• 

  Flash indexing with external resource loading

  Posted: June 19th, 2009, 1:27am CEST by Maile Ohye

  Tags:  crawling and indexing  

  Webmaster Level: All
  We just added external resource loading to our Flash indexing capabilities. This means that when a SWF file loads content from some other file—whether it's text, HTML, XML, another SWF, etc.—we can index this external content too, and associate it with the parent SWF file and any documents that embed it.
  This new capability improves search quality by allowing relevant content contained in external resources to appear in response to users' queries. For example, this result currently comes up in response to the query [2002 VW Transporter 888]:
  
  
  Prior to this launch, this result did not appear, because all of the relevant content is contained in an XML file loaded by a SWF file.
  
  To date, when Google encounters SWF files on the web, we can:

  • Index textual content displayed as a user interacts with the file. We click buttons and enter input, just like a user would.
  • Discover links within Flash files.
  • Load external resources and associate the content with the parent file.
  • Support common JavaScript techniques for embedding Flash, such as SWFObject and SWFObject2.
  • Index sites scripted with AS1 and AS2, even if the ActionScript is obfuscated. Update on June 19, 2009: We index sites with AS3 as well. The ActionScript version isn't particularly relevant in our Indexing process, so we support older versions of AS in addition to the latest.

  If you don't want your SWF file or any of its external resources crawled by search engines, please use an appropriate robots.txt directive.
  
  Written by Janis Stipins, Software Engineer
• 

  Best practices for Product Search

  Posted: June 17th, 2009, 10:27am CEST by Maile Ohye

  Tags:  products and services  

  Webmaster Level: Beginner to Intermediate
  
  If you run an e-commerce site and you'd like your products to be eligible to be shown in Google search results, then check out our "Product Search for Webmasters" video. In addition to the basics on Product Search, I cover:

  • Attributes to include in your feed
  • FAQs
  • Will my products' rankings improve if I include custom attributes in my feed?
  • Do product listings expire after 30 days?
  • How often should I submit my feed?

  
  
  
  More information can be found in the Product Search Help Center.
  
  Written by Maile Ohye, Developer Programs Tech Lead
• 

  Best practices for Product Search

  Posted: June 17th, 2009, 10:27am CEST by Maile Ohye

  Tags:  products and services  

  Webmaster Level: Beginner to Intermediate
  
  If you run an e-commerce site and you'd like your products to be eligible to be shown in Google search results, then check out our "Product Search for Webmasters" video. In addition to the basics on Product Search, I cover:

  • Attributes to include in your feed
  • FAQs
  • Will my products' rankings improve if I include custom attributes in my feed?
  • Do product listings expire after 30 days?
  • How often should I submit my feed?

  
  
  
  More information can be found in the Product Search Help Center.
  
  Written by Maile Ohye, Developer Programs Tech Lead
• 

  Webmaster Central YouTube update for June 8th - 12th

  Posted: June 15th, 2009, 4:22pm CEST by Michael Wyszomierski

  Tags:  feedback and communication  

  Want to know what's new on the Webmaster Central YouTube channel? Here's what we've uploaded in the past week:
  
  Matt Cutts answered a few new questions from the Grab Bag:
  

  • What if a search for my business triggers "Did you mean?"
  • Why do big brands rank so well?
  • Are CSS-based layouts better than tables for SEO?

  Matt also went over a great example of whitehat linkbait:
  
  
  
  And if you've ever thought about hiding text, here's one technique that didn't fool Google:
  
  
  
  Feel free to leave comments letting us know how you liked the videos, and if you have any specific questions, ask the experts in the Webmaster Help Forum.
  
  Posted by Michael Wyszomierski, Search Quality Team
• 

  Webmaster Central YouTube update for June 1st - 5th

  Posted: June 8th, 2009, 4:33pm CEST by Michael Wyszomierski

  Tags:  feedback and communication  

  Want to know what's new on the Webmaster Central YouTube channel? Here's what we've uploaded in the past week:
  
  Matt Cutts answered a new question each day from the Grab Bag:
  

  • Are shortened URLs treated differently than other redirects?
  • How can I make sure Google reaches my deeper pages?
  • What impact does server location have on rankings?
  • Will a "coming soon" page negatively impact my site?
  • How many pages can Google index from a single site?

  In response to questions we've been getting about the videos themselves, we created a behind the scenes look at our setup. And just to throw in some more variety, Matt took a break from SEO questions for a few minutes to show off a fun use of a barcode scanner with Google Book Search. You can read more about it on the Inside Google Book Search blog.
  
  For your instant viewing pleasure, here's the answer to the question about URL shortners:
  
  
  And now a peek behind the scenes:
  
  
  Feel free to leave comments letting us know how you liked the videos, and if you have any specific questions, ask the experts in the Webmaster Help Forum.
  
  Posted by Michael Wyszomierski, Search Quality Team
• 

  SEO Starter Guide now available in 40 languages

  Posted: June 5th, 2009, 9:10am CEST by Maile Ohye

  Tags:  crawling and indexing  

  Webmaster Level: Beginner
  
  Since the release of our Search Engine Optimization Starter Guide last November, we've steadily been adding localized versions for our international users. Today, we're happy to announce that the guide is now available in 40 languages, which covers 98% of the global Internet audience. We hope that webmasters around the world can use the guide to improve their sites' crawlability and indexing in search engines.
  
  We'd also like to thank everyone for their continued feedback on the guide. We're taking notes on what you liked and what you thought was missing for use in future updates and possibly a version for advanced users.
  
  Here's a list of the languages available:

  • Arabic
  • Bulgarian
  • Catalan
  • Chinese (Simplified)
  • Chinese (Traditional)
  • Croatian
  • Czech
  • Danish
  • Dutch
  • English
  • English (GB)
  • Filipino
  • Finnish
  • French
  • German
  • Greek
  • Hebrew
  • Hindi
  • Hungarian
  • Indonesian
  • Italian
  • Japanese
  • Korean
  • Latvian
  • Lithuanian
  • Norwegian
  • Polish
  • Portuguese (BR)
  • Portuguese (PT)
  • Romanian
  • Russian
  • Serbian
  • Slovak
  • Slovenian
  • Spanish
  • Swedish
  • Thai
  • Turkish
  • Ukrainian
  • Vietnamese

  Written by Brandon Falls, Search Quality Team
• 

  Webmaster Central YouTube update for May 26-29

  Posted: June 1st, 2009, 5:18pm CEST by Michael Wyszomierski

  Tags:  feedback and communication  

  In February, we launched the Webmaster Central YouTube channel, and since then we've been busy keeping it fresh with new content. We're not yet uploading 20 hours every minute, but did you know that we've been releasing a new video almost every weekday? You can keep up with the latest updates by subscribing to our channel in YouTube, or by adding our feed to your RSS reader.
  
  If you're more of a weekly digest kind of person, we're here for you too. For every week that we have new videos available, we'll let you know here on the blog. In the videos uploaded in the past week, you'll find answers to these questions from Matt's Grab Bag:
  
  Can product descriptions be considered duplicate content?
  How can new pages get indexed quickly?
  What are your views on PageRank sculpting?
  Will you ever switch to a Mac?
  
  Feel free to leave comments letting us know how you liked the videos, and if you have any specific questions, ask the experts in the Webmaster Help Forum.
  
  Posted by Michael Wyszomierski, Search Quality Team
• 

  Google Webmaster Help Forums in more languages

  Posted: April 30th, 2009, 12:48pm CEST by Reid Yokoyama

  Tags:  feedback and communication  

  Traditionally when we launch a new communication channel, we also give the shareholders a chance to introduce themselves. We did so when we opened webmaster help communities for European webmasters almost two years ago, and also more than a year ago, when we were able to expand and add groups in three more languages. Last December we were very happy to announce the re-launch of two of our Help Forums in a new and cool look and feel.
  
  Today, we're happy to announce that we keep on increasing the global dialogue with webmasters, opening an Arabic and a Czech/Slovak Webmaster Help Forum. Furthermore, we would like to highlight the support we offer in Chinese, Japanese, and Korean. While we've offered support to Chinese webmasters for a little more than a year, the Japanese and Korean forums are only a few weeks old. Keeping with tradition, the guides monitoring our new forums would like to introduce themselves to the global webmaster family:
  
  Arabic Webmaster Help Forum
  
  مرحبا! My name is Adel and I'll be monitoring the Arabic Webmaster Help Forum. I'm originally from Beirut, Lebanon. After finishing computer science studies, I joined Google, some 18 months ago.
  
  Besides working on search quality in Arabic and building a community on our forum, I enjoy traveling and listening to really loud heavy metal music; sometimes I get to do both at the same time! ;-)
  
  I am looking forward to a lot of questions regarding Arabic Google Search and of course ranking and indexing issues on your sites to come. I hope I'll see you there soon!
  - Adel
  
  Czech/Slovak Webmaster Help Forum
  
  Zdravím! I am Marcel, the Google Guide on the Czech/Slovak Webmaster Help Forum. I am originally from Slovakia. After graduating in New Media and Industrial Design, it took me some time and traveling around the globe before moving to Dublin and eventually joining Google some 3 years ago.
  
  Ever since, I've been working in different teams. I was lucky to be part of the AdSense team where I participated in launching AdSense for Content for Czech and Slovak. Since my transition to Search Quality, I enjoy working on improving the quality of our natural search results in Czech, Slovak, and Polish.
  
  Besides my work I have a few more passions, such as listening to live music in Irish pubs, challenging my colleagues in occasional Soulcalibur skirmishes on Playstation and testing burger places all over the world :-) If you want to discuss any of these topics or maybe something about your sites, please join the community. I am looking forward to meeting you there :-)
  - Marcel
  Chinese Webmaster Help Forum
  
  你好! Hi from the Chinese Webmaster Help Forum team! The Chinese Webmaster Help Forum has received great support from webmasters since its launch in March 2008. In March 2009, the Chinese Webmaster Help Forum moved to a new system with many more user-friendly features for better information sharing. It has become a good platform for webmasters to share their knowledge of Google search and Webmaster Tools and to communicate with Google.
  
  The Chinese Forum now has 6 Google Guides: Xiang (降龙十巴掌), Eric (趙錢孫李), Marina (小馬過河), Chris (城镇), Hyson (草帽路飞), and Fa (法人戴表). We are from many different provinces of mainland China. When not spending time in the forum, we enjoy playing ping-pong and foosball in the office. A few of us are huge video game fans. You may learn more about us when you participate in the forum :)
  
  A big thank you to everyone who has taken part in forum discussions! We hope to see both familiar faces and newcomers join in the Chinese Webmaster Help Forum!
  - Xiang (降龙十巴掌), Eric (趙錢孫李), Marina (小馬過河), Chris (城镇), Hyson (草帽路飞), and Fa (法人戴表)
  
  Japanese Webmaster Help Forum
  
  こんにちは! Hello from the Japanese Webmaster Help Forum team! Our names are Nao ( なお ), Kaede ( 楓 ), Haru ( ハル ), and Kyotaro. We are the four guides working in Google Search Quality for Japanese. We've just launched our forum on March 6th.
  
  All of us were born in Japan and grew up here. Nao has also lived in Greece, the Netherlands, and New York. Haru is from the west side of Japan, which is known for its talkative culture and traditional Japanese comedy. Maybe you will read Haru's unique communication on our forum :)
  
  As for our interests, we love eating and drinking! Between posting on the forum, we enjoy Google's excellent lunches and sweets a lot. After working, of course, we sometimes go out for a drink with our team members :) Kaede knows all the nice bars in Tokyo.
  
  Nao and Kyotaro love Sumo wrestling. We've watched two tournaments this year with Googlers from other locations. Haru, of course, loves watching comedies!
  
  We are really excited and happy to see many users joining our forum and sharing tips with each other. Looking forward to seeing you there!
  - Nao ( なお ), Kaede ( 楓 ), Haru ( ハル ), and Kyotaro
  
  Korean Webmaster Help Forum
  
  안녕하세요! Hello everyone, my name is Joowon and I work in Google Search Quality for Korean. I was born in Germany and lived in Korea for a few years before moving to Hawaii, California and New York to attend high school and college. After all that traveling, I'm only fluent in Korean and English, with a bit of proficiency in Japanese. Some of the interests I've developed over the years are design, wine, cooking, yoga, and sustainability issues.
  
  Currently I'm back in Seoul and enjoying the dynamic atmosphere here, with lots of interesting people and great food. The Korean Webmaster Help Forum was launched only a few weeks ago, and I'm very much looking forward to talking to all of you. See you in the forum!
  - Joowon
  
  Hello world! ;) I am Andrew and I am part of the Search Quality team in Seoul. I grew up in a port city in the southern part of Korea. Ironically, I don't eat seafood because it looks scary to me :( Many of my friends and colleagues love to make jokes about that, but I still don't eat any seafood yet. Playing drums, traveling and photography are my main interests. Currently I'm a drummer of "Spring Fingers", the first band of Google Korea, and we'll have our first concert at the end of April!
  
  I love playing around with web technologies/APIs and find it very exciting to exchange information and ideas on the web. The Korean Webmaster Help Forum was recently launched and I hope to see you there!
  - Andrew
  
  If you're curious about our Webmaster Help Forums in other languages, please feel free to peak in. Here's a list of our currently monitored Webmaster Help Forums: Arabic, Chinese, Czech/Slovak, Dutch, Danish, English, Finnish, French, German, Hebrew, Hungarian, Italian, Japanese, Korean, Polish, Portuguese, Russian, Spanish, Swedish, and Turkish.
  
  Written by Joowon Ahn, Andrew Baek, Adel Saud, Kaspar Szymanski, Fa Wang, Reid Yokoyama, and Marcel Zavacky, Search Quality Team
• 

  Get up-to-date on Image Search

  Posted: March 10th, 2009, 6:13pm CET by Maile Ohye

  Tags:  crawling and indexing  

  Webmaster Level: All
  
  Recently at SMX West, I gave an Image Search presentation that I'd like to share with our broader webmaster community. The goal of the presentation was to provide insights into how image search is used, how it works, and how webmasters can optimize their pages for image searchers.
  
  You'll see more information about:

  • Some background on the reach of Image Search
  • Interesting findings on the behavior of image searchers
  • Our efforts at handling multiple image referrers
  • How to best feature images (image quality and placement, relevant surrounding text, etc.)

  Take a look and let us know your thoughts in the comments. We'd love to hear from you.
  
  
  
  Posted by Peter Linsley, Product Manager
• 

  Using stats from site: and Sitemap details

  Posted: March 4th, 2009, 5:40pm CET by Chark

  Tags:  crawling and indexing  

  Webmaster Level: Beginner to Intermediate
  
  Every now and then in the webmaster blogosphere and forums, this issue comes up: when a webmaster performs a [site:example.com] query on their website, the number of indexed results differs from what is displayed in their Sitemaps report in Webmaster Tools. Such a discrepancy may smell like a bug, but it's actually by design. Your Sitemap report only reflects the URLs you've submitted in your Sitemap file. The site operator, on the other hand, takes into account whatever Google has crawled, which may include URLs not included in your Sitemap, such as newly added URLs or other URLs discovered via links.
  
  Think of the site operator as a quick diagnosis of the general health of your site in Google's index. Site operator results can show you:
  

  • a rough estimate of how many pages have been indexed
  • one indication of if your site has been hacked
  • if you have duplicate titles or snippets

  Here is an example query using the site operator:
  
  
  
  Your Sitemap report provides more granular statistics about the URLs you submitted, such as the number of indexed URLs vs. the number submitted for crawling, and Sitemap-specific warnings or errors that may have occurred when Google tried to access your URLs.
  
  
  
  Feel free to check out our Help Center for more on the site: operator and Sitemaps. If you have further questions or issues, please post to our Webmaster Help Forum, where experienced webmasters and Googlers are happy to help.
  
  Posted by Charlene Perez, Search Quality Team
• 

  Canonical Link Element: presentation from SMX West

  Posted: February 25th, 2009, 7:50am CET by Michael Wyszomierski

  Tags:  crawling and indexing  

  A little while ago, Google and other search engines announced support for a canonical link element that can help site owners with duplicate content issues. I recreated my presentation from SMX West and you can watch it below:
  
  
  
  You can access the slides directly or follow along here:
  
  
  
  By the way, Ask just announced that they will support the canonical link element. Read all about it in the Ask.com blog entry.
  
  Thanks again to Wysz for turning this into a great video.
  In fact, you might not have seen it, but we recently created a webmaster videos channel on YouTube. If you're interested, you can watch the new webmaster channel. If you subscribe to that channel, you'll always find out about new webmaster-related videos from Google.
  Posted by Matt Cutts, Search Quality Team
• 

  Introducing the Google Webmaster Central YouTube Channel

  Posted: February 23rd, 2009, 1:30pm CET by Michael Wyszomierski

  Tags:  feedback and communication  

  In his State of the Index presentation, Matt Cutts said that one of the things to look for from Google in 2009 is continued communication with webmasters. On the Webmaster Central team, we've found that using video is a great way to reach people. We've shown step-by-step instructions on how to use features of Webmaster Tools, shared our presentations with folks who were unable to make it to conferences, and even taken you through a day in the life of our very own Maile Ohye as she meets with many Googlers involved in webmaster support.
  
  We plan on releasing more videos like these in the future, so we've opened up our own channel on YouTube to host webmaster-related videos. Our first video is already up, and we'll have more to share with you soon. If you want to be the first to know when we release something new, you can subscribe to us using your YouTube account, or grab this RSS feed if you'd like to keep track in your feed reader. Please let us know how you like the channel, and use the comments in this post to share your ideas for future videos.
  
  And while we'll all do our best to make sure Matt Cutts understands that Rick Rolling is so last year, be careful where you click on April 1st.
  
  Posted by Michael Wyszomierski, Search Quality Team
• 

  State of the Index: my presentation from PubCon Vegas

  Posted: February 18th, 2009, 5:15pm CET by Michael Wyszomierski

  Tags:  feedback and communication  

  It seems like people enjoyed when I recreated my Virtual Blight talk from the Web 2.0 Summit late last year, so we decided to post another video. This video recreates the "State of the Index" talk that I did at PubCon in Las Vegas late last year as well.
  
  Here's the video of the presentation:
  
  
  and if you'd like to follow along, here are the slides:
  
  
  
  You can also access the presentation directly. Thanks again to Wysz for recording this video and splicing the slides into the video.
  Posted by Matt Cutts, Search Quality Team
• 

  Specify your canonical

  Posted: February 12th, 2009, 1:30pm CET by Maile Ohye

  Tags:  crawling and indexing  

  Carpe diem on any duplicate content worries: we now support a format that allows you to publicly specify your preferred version of a URL. If your site has identical or vastly similar content that's accessible through multiple URLs, this format provides you with more control over the URL returned in search results. It also helps to make sure that properties such as link popularity are consolidated to your preferred version.
  
  Let's take our old example of a site selling Swedish fish. Imagine that your preferred version of the URL and its content looks like this:
  
  http://www.example.com/product.php?item=swedish-fish
  
  
  However, users (and Googlebot) can access Swedish fish through multiple (not as simple) URLs. Even if the key information on these URLs is the same as your preferred version, they may show slight content variations due to things like sort parameters or category navigation:
  
  http://www.example.com/product.php?item=swedish-fish&category=gummy-candy
  Or they have completely identical content, but with different URLs due to things such as a tracking parameters or a session ID:
  
  http://www.example.com/product.php?item=swedish-fish&trackingid=1234&sessionid=5678
  Now, you can simply add this <link> tag to specify your preferred version:
  
  <link rel="canonical" href="http://www.example.com/product.php?item=swedish-fish" />
  
  inside the <head> section of the duplicate content URLs:
  
  http://www.example.com/product.php?item=swedish-fish&category=gummy-candy
  http://www.example.com/product.php?item=swedish-fish&trackingid=1234&sessionid=5678
  
  and Google will understand that the duplicates all refer to the canonical URL: http://www.example.com/product.php?item=swedish-fish. Additional URL properties, like PageRank and related signals, are transferred as well.
  
  This standard can be adopted by any search engine when crawling and indexing your site.
  
  Of course you may have more questions. Joachim Kupke, an engineer from our Indexing Team, is here to provide us with the answers:
  
  Is rel="canonical" a hint or a directive?
  It's a hint that we honor strongly. We'll take your preference into account, in conjunction with other signals, when calculating the most relevant page to display in search results.
  
  Can I use a relative path to specify the canonical, such as <link rel="canonical" href="product.php?item=swedish-fish" />?
  Yes, relative paths are recognized as expected with the <link> tag. Also, if you include a <base> link in your document, relative paths will resolve according to the base URL.
  
  Is it okay if the canonical is not an exact duplicate of the content?
  We allow slight differences, e.g., in the sort order of a table of products. We also recognize that we may crawl the canonical and the duplicate pages at different points in time, so we may occasionally see different versions of your content. All of that is okay with us.
  
  What if the rel="canonical" returns a 404?
  We'll continue to index your content and use a heuristic to find a canonical, but we recommend that you specify existent URLs as canonicals.
  
  What if the rel="canonical" hasn't yet been indexed?
  Like all public content on the web, we strive to discover and crawl a designated canonical URL quickly. As soon as we index it, we'll immediately reconsider the rel="canonical" hint.
  
  Can rel="canonical" be a redirect?
  Yes, you can specify a URL that redirects as a canonical URL. Google will then process the redirect as usual and try to index it.
  
  What if I have contradictory rel="canonical" designations?
  Our algorithm is lenient: We can follow canonical chains, but we strongly recommend that you update links to point to a single canonical page to ensure optimal canonicalization results.
  
  Can this link tag be used to suggest a canonical URL on a completely different domain?
  No. To migrate to a completely different domain, permanent (301) redirects are more appropriate. Google currently will take canonicalization suggestions into account across subdomains (or within a domain), but not across domains. So site owners can suggest www.example.com vs. example.com vs. help.example.com, but not example.com vs. example-widgets.com.
  
  Sounds great—can I see a live example?
  Yes, wikia.com helped us as a trusted tester. For example, you'll notice that the source code on the URL http://starwars.wikia.com/wiki/Nelvana_Limited specifies its rel="canonical" as: http://starwars.wikia.com/wiki/Nelvana.
  
  The two URLs are nearly identical to each other, except that Nelvana_Limited, the first URL, contains a brief message near its heading. It's a good example of using this feature. With rel="canonical", properties of the two URLs are consolidated in our index and search results display wikia.com's intended version.
  
  Feel free to ask additional questions in our comments below. And if you're unable to implement a canonical designation link, no worries; we'll still do our best to select a preferred version of your duplicate content URLs, and transfer linking properties, just as we did before.
  
  Updated: This link-tag is currently also supported by Ask.com, Microsoft Live Search and Yahoo!.
  
  Written by Joachim Kupke, Senior Software Engineer, and Maile Ohye, Developer Programs Tech Lead
• 

  Help us help you

  Posted: February 11th, 2009, 3:32pm CET by Matt Dougherty

  Tags:  feedback and communication  

  You're a webmaster, right? Well, we love webmasters! To ensure we give you the best support possible, we've set up a survey to get your thoughts on Webmaster Central and our related support efforts. If you have a few extra minutes this week, please click here to give us your honest feedback.
  
  Thanks from all of us on the Webmaster Central Team.
  
  Posted by Matt Dougherty, Search Quality
  
• 

  A new Google Sitemap Generator for your website

  Posted: January 13th, 2009, 6:12am CET by John Mueller

  Tags:  crawling and indexing  

  It's been well over three years since we initially announced the Python Sitemap generator in June 2005. In this time, we've seen lots of people create great third-party Sitemap generators to help webmasters create better Sitemap files. While most Sitemap generators either crawl websites or list the files on a server, we have created a different kind of Sitemap generator that uses several ways to find URLs on your website and then allows you to automatically create and maintain different kinds of Sitemap files.
  
  
  
  About Google Sitemap Generator
  Our new open-source Google Sitemap Generator finds new and modified URLs based on your webserver's traffic, its log files, or the files found on the server. By combining these methods, Google Sitemap Generator can be very fast in finding these URLs and calculating relevant metadata, thereby making your Sitemap files as effective as possible. Once Google Sitemap Generator has collected the URLs, it can create the following Sitemap files for you:
  

  • XML Sitemaps for Web Search according to the sitemaps.org standard
  
  
  • Mobile Sitemaps for mobile-friendly websites
  
  
  • Code Search Sitemaps for source code that you make available to users

  
  In addition, Google Sitemap Generator can send a ping to Google Blog Search for all of your new or modified URLs. You can optionally include the URLs of the Sitemap files in your robots.txt file as well as "ping" the other search engines that support the sitemaps.org standard.
  
  Sending the URLs to the right Sitemap files is simple thanks to the web-based administration console. This console gives you access to various features that make administration a piece of cake while maintaining a high level of security by default.
  
  Getting started
  Google Sitemap Generator is a server plug-in that can be installed on both Linux/Apache and Microsoft IIS Windows-based servers. As with other server-side plug-ins, you will need to have administrative access to the server to install it. You can find detailed information for the installation in the Google Sitemap Generator documentation.
  
  We're excited to release Google Sitemap Generator with the source code and hope that this will encourage more web hosters to include this or similar tools in their hosting packages!
  
  Do you have any questions? Feel free to drop by our Help Group for Google Sitemap Generator or ask general Sitemaps question in our Webmaster Help Forum.
  
  Posted by John Mueller, Webmaster Trends Analyst, Google Zürich
• 

  Reintroducing your English Webmaster Help Google Guides

  Posted: December 7th, 2008, 9:30pm CET by Reid Yokoyama

  Tags:  feedback and communication  

  When we announced our latest round of Bionic Posters in the old Help Group, an astute webmaster noted that we had never fully introduced ourselves to the webmaster community. With our announcement of a new English Webmaster Help forum, I'm happy to (re)introduce you all to the Google Guides from around the world who will be hanging out in the English forum:
  
  Guides in Mountain View, California (from left to right): Adam, Maile, Matt Cutts, Chris, Wysz, Matt, Chark, and Adi.
  
  Also in Mountain View: Evan, Jessica, and Nate.
  
  Guides in Kirkland, Washington: Susan, Riona, and Jonathan.
  
  Guides in Zürich, Switzerland: John Mueller and Balázs.
  
  Guides in Hyderabad, India: Koti, Reid (visiting from Mountain View), and Jayan.
  
  Guide in Ottowa Ottawa :), Ontario: Oliver. All of us look forward to seeing you around the forum! Our Google Guides in non-English Webmaster Help Groups introduce themselves here.
  
  Written by Reid Yokoyama, Search Quality
• 

  Friend Connect now available in beta to everyone

  Posted: December 5th, 2008, 4:38am CET by Maile Ohye

  Tags:  products and services  

  If you've been looking for a way to grow traffic and make your site more interactive, check out Google Friend Connect -- now in beta and available to all webmasters. Remember that with Friend Connect, you can easily add social features to your site by just copying and pasting a few snippets of code, no programming necessary! Your visitors will be able to join your site, create or link to a profile, interact with other visitors, and invite friends to visit your site. Best of all, your visitors won't be required to create yet another username and password -- Friend Connect lets them sign in using an existing Google, Yahoo, AOL, or OpenID account. To learn more, watch the video below:
  
  
  So for any of you who were interested in Friend Connect after our first announcement, and also to all of the newer readers out there, go ahead and give Friend Connect a try.
  
  Written by Mendel Chuang, Product Marketing Manager
• 

  A new look for our Webmaster Help Group

  Posted: December 4th, 2008, 9:15am CET by Susan Moskwa

  Tags:  products and services  

  Googlers strongly believe in dogfooding our own products. We manage our work schedules with Google Calendar, publish our blogs on Blogger, and store scads of documentation on Google Sites. So, ever since we launched our first Webmaster Help Group, we've been using Google Groups to facilitate conversations about Webmaster Tools and web search issues.

  Today, however, I'm thrilled to announce that our English and Polish Help Groups are getting a makeover. And the changes are more than just skin-deep. Our new Help Forums should make it easier for you to find answers, share resources with others, and have your participation acknowledged.

  You can read more about the changes on the Official Google Blog, and then check it out for yourself: English, Polish.

  Q: What will happen to the old English and Polish Help Groups?
  A: While our old groups are now closed to new posts, they will still be available in read-only mode in case you want to reference any of your favorite posts from the good old days. Many of the most frequently-asked questions (and answers!) have already been transferred to our new Help Forums.

  Q: If I was a member of the old group, will I automatically be a member of the new forum?
  A: We won't be "transferring" membership from the old groups to the new, so even if you were a member of our Google Groups forum, you'll still need to join the new forum in order to participate. Nicknames and user profiles are also managed separately, so you're welcome to recreate your Google Groups profile in our new forum, or reinvent yourself.

  Q: What about the Webmaster Help Groups in other languages?
  A: They'll be moving to the new Help Forum format in 2009. Specific dates will be announced in each of the groups as they get closer to their moving date.

  Feel free to post any other questions about the new Help Forums in the comments below.

  
  Written by Susan Moskwa, Webmaster Trends Analyst
• 

  Date with Googlebot, Part II: HTTP status codes and If-Modified-Since

  Posted: November 27th, 2008, 1:12am CET by Gerald Hibbert

  Tags:  crawling and indexing  

  Our date with Googlebot was so wonderful, but it's hard to tell if we, the websites, said the right thing. We returned 301 permanent redirect, but should we have responded with 302 temporary redirect (so he knows we're playing hard to get)? If we sent a few new 404s, will he ever call our site again? Should we support the header "If-Modified-Since?" These questions can be confusing, just like young love. So without further ado, let's ask the expert, Googlebot, and find out how he judged our response (code).
  
  
  Supporting the "If-Modified-Since" header and returning 304 can save bandwidth.
  
  -----------
  Dearest Googlebot,
    Recently, I did some spring cleaning on my site and deleted a couple of old, orphaned pages. They now return the 404 "Page not found" code. Is this ok, or have I confused you?
  Frankie O'Fore
  
  Dear Frankie,
    404s are the standard way of telling me that a page no longer exists. I won't be upset—it's normal that old pages are pruned from websites, or updated to fresher content. Most websites will show a handful of 404s in the Crawl Diagnostics over at Webmaster Tools. It's really not a big deal. As long as you have good site architecture with links to all your indexable content, I'll be happy, because it means I can find everything I need.
  
    But don't forget, it's not just me who comes to your website—there may be humans seeing these pages too. If you've only got a very simple '404 page not found' message, visitors who aren't as savvy could be baffled. There are lots of ways to make your 404 page more friendly; a quick one is our 404 widget over at Webmaster Tools, which will help direct people to content which does exist. For more information, you can read the blog post. Most web hosting companies, big and small, will let you customise your 404 page (and other return codes too).
  
  Love and kisses,
  Googlebot
  
  
  Hey Googlebot,
    I was just reading your reply to Frankie above, and it raised a couple of questions.
  What if I have someone linking to a page that no longer exists? How can I make sure my visitors still find what they're after? Also, what if I just move some pages around? I'd like to better organise my site, but I'm worried you'll get confused. How can I help you?
  Yours hopefully,
  Little Jimmy
  
  
  Hello Jimmy,
     Let's pretend there are no anachronisms in your letter, and get to the meat of the matter. Firstly, let's look at links coming from other sites. Obviously, these can be a great source of traffic, and you don't want visitors presented with an unfriendly 'Page not found' message. So, you can harness the power of the mighty redirect.
  
     There are two types of redirect—301 and 302. Actually, there are lots more, but these are the two we'll concern ourselves with now. Just like 404, 301 and 302 are different types of responses codes you can send to users and search engine crawlers. They're both redirects, but a 301 is permanent and a 302 is temporary. A 301 redirect tells me that whatever this page used to be, now it lives somewhere else. This is perfect for when you're re-organising your site, and also helps with links from offsite. Whenever I see a 301, I'll update all references to that old page with the new one you've told me about. Isn't that easy?
  
     If you don't know where to begin with redirects, let me get you started. It depends on your webserver, but here are some searches that may be helpful:
  Apache: [www.google.com]
  IIS: [www.google.com]
  You can also check your manual, or the README files that came with your server.
  
     As an alternative to a redirect, you can email the webmaster of the site linking to you and ask them to update their link. Not sure what sites are linking to you? Don't despair - my human co-workers have made that easy to figure out. In the "Links" portion of Webmaster Tools, you can enter a specific URL on your site to determine who's linking to it.
  
    My human co-workers also just released a tool which shows URLs linking to non-existent pages on your site. You can read more about that here.
  
  Yours informationally,
  Googlebot
  
  
  Darling Googlebot,
     I have a problem—I live in a very dynamic part of the web, and I keep changing my mind about things. When you ask me questions, I never respond the same way twice—my top threads change every hour, and I get new content all the time! You seem like a straightforward guy who wants straightforward answers. How can I tell you when things change without confusing you?
  Temp O'Rary
  
  
  Dear Temp,
     I just told little Jimmy that 301's are the best way to tell a Googlebot about your new address, but what you're looking for is a 302.
     Once you're indexed, it's the polite way to tell your visitors that your address is still the right one, but that the content can temporarily be found elsewhere. In these situations, a 302 (or the rarer '307 Temporary Redirect') would be better. For example, orkut redirects from http://orkut.com to http://google.com/accounts/login?service=orkut, which isn't a page that humans would find particularly useful when searching for Orkut***.
  It's on a different domain, for starters. So, a 302 has been used to tell me that all the content and linking properties of the URL shouldn't be updated to the target - it's just a temporary page.
  
    That's why when you search for orkut, you see orkut.com and not that longer URL.
  
    Remember: simple communication is the key to any relationship.
  
  Your friend,
  Googlebot
  
  *** Please note, I simplified the URL to make it easier to read. It's actually much more complex than that.
  
  Captain Googlebot,
     I am the kind of site who likes to reinvent herself. I noticed that the links to me on my friends' sites are all to URLs I got rid of several redesigns ago! I had set up 301s to my new URLs for those pages, but after that I 301'ed the newer URLs to my next version. Now I'm afraid that if you follow their directions when you come to crawl, you'll end up following a string of 301s so long that by the end you won't come calling any more.
  Ethel Binky
  
  
  Dear Ethel,
     It sounds like you have set up some URLs that redirect to more redirects to... well, goodness! In small amounts, these "repeat redirects" are understandable, but it may be worth considering why you're using them in the first place. If you remove the 301s in the middle and send me straight to the final destination on all of them, you'll save the both of us a bunch of time and HTTP requests. But don't just think of us. Other people get tired of seeing that same old 'contacting.... loading ... contacting...' game in their status bar.
  
     Put yourself in their shoes—if your string of redirects starts to look rather long, users might fear that you have set them off into an infinite loop! Bots and humans alike can get scared by that kind of "eternal commitment." Instead, try to get rid of those chained redirects, or at least keep 'em short. Think of the humans!
  
  Yours thoughtfully,
  Googlebot
  
  Dear Googlebot,
     I know you must like me—you even ask me for unmodified files, like my college thesis that hasn't changed in 10 years. It's starting to be a real hassle! Is there anything I can do to prevent your taking up my lovely bandwidth?
  
  Janet Crinklenose
  
  
  Janet, Janet, Janet,
     It sounds like you might want to learn a new phrase—'304 Not Modified'. If I've seen a URL before, I insert an 'If-Modified-Since' in my request's header. This line also includes an HTTP-formatted date string. If you don't want to send me yet another copy of that file, stand up for yourself and send back a normal HTTP header with the status '304 Not Modified'! I like information, and this qualifies too. When you do that, there's no need to send me a copy of the file—which means you don't waste your bandwidth, and I don't feel like you're palming me off with the same old stuff.
  
     You'll probably notice that a lot of browsers and proxies will say 'If-Modified-Since' in their headers, too. You can be well on your way to curbing that pesky bandwidth bill.
  
  Now go out there and save some bandwidth!
  Good ol' Googlebot
  -----------
  
  Googlebot has been so helpful! Now we know how to best respond to users and search engines. The next time we get together, though, it's time to sit down for a good long heart-to-heart with the guy (Date with Googlebot: Part III, is coming soon!).
  
  As told to Chewy [Trewhella], Developer Advocate
  
  UPDATE: Added a missing link. Thanks to Boris for pointing that out.
• 

  Google's SEO Starter Guide

  Posted: November 12th, 2008, 12:55pm CET by Maile Ohye

  Tags:  crawling and indexing  

  Webmasters often ask us at conferences or in the Webmaster Help Group, "What are some simple ways that I can improve my website's performance in Google?" There are lots of possible answers to this question, and a wealth of search engine optimization information on the web, so much that it can be intimidating for newer webmasters or those unfamiliar with the topic. We thought it'd be useful to create a compact guide that lists some best practices that teams within Google and external webmasters alike can follow that could improve their sites' crawlability and indexing.
  
  Our Search Engine Optimization Starter Guide covers around a dozen common areas that webmasters might consider optimizing. We felt that these areas (like improving title and description meta tags, URL structure, site navigation, content creation, anchor text, and more) would apply to webmasters of all experience levels and sites of all sizes and types. Throughout the guide, we also worked in many illustrations, pitfalls to avoid, and links to other resources that help expand our explanation of the topics. We plan on updating the guide at regular intervals with new optimization suggestions and to keep the technical advice current.
  
  So, the next time we get the question, "I'm new to SEO, how do I improve my site?", we can say, "Well, here's a list of best practices that we use inside Google that you might want to check out."
  
  Update: The guide is now available in Chinese, German, Portuguese (Brazil, Portugal), Spanish, and Turkish.
  
  Written by Brandon Falls, Search Quality Team
• 

  Helping you break the language barrier

  Posted: October 14th, 2008, 1:03pm CEST by Maile Ohye

  Tags:  products and services  

  When webmasters put content out on the web it's there for the world to see. Unfortunately, most content on the web is only published in a single language, understandable by only a fraction of the world's population.
  
  In a continued effort to make the world's information universally accessible, Google Translate has a number of tools for you to automatically translate your content into the languages of the world.
  
  
  Users may already be translating your webpage using Google Translate, but you can make it even easier by including our "Translate My Page" gadget, available at [translate.google.com].
  
  The gadget will be rendered in the user's language, so if they come to your page and can't understand anything else, they'll be able to read the gadget, and translate your page into their language.
  
  Sometimes there may be some content on your page that you don't want us to translate. You can now add class=notranslate to any HTML element to prevent that element from being translated. For example, you may want to do something like:
  

  Email us at <span class="notranslate">sales at mydomain dot com</span>

  And if you have an entire page that should not be translated, you can add:
  

  <meta name="google" value="notranslate">

  to the <head> of your page and we won't translate any of the content on that page.
  
  Update on 12/15/2008: We also support:

  <meta name="google" content="notranslate">

  Thanks to chaoskaizer for pointing this out in the comments. :)
  
  Lastly, if you want to do some fancier automatic translation integrated directly into your page, check out the AJAX Language API we launched last March.
  
  With these tools we hope you can more easily make your content available in all the languages we support, including Arabic, Bulgarian, Catalan, Chinese, Croatian, Czech, Danish, Dutch, English, Filipino, Finnish, French, German, Greek, Hebrew, Hindi, Indonesian, Italian, Japanese, Korean, Latvian, Lithuanian, Norwegian, Polish, Portuguese, Romanian, Russian, Serbian, Slovak, Slovenian, Spanish, Swedish, Ukrainian, and Vietnamese.
  
  Written by Josh Estelle, Google Translate
• 

  Webmaster Tools shows Crawl error sources

  Posted: October 13th, 2008, 4:03pm CEST by Jonathan Simon

  Tags:  crawling and indexing  

  Ever since we released the crawl errors feature in Webmaster Tools, webmasters have asked for the sources of the URLs causing the errors. Well, we're listening! We know it was difficult for those of you who wanted to identify the cause of a particular "Not found" error, in order to prevent it in the future or even to request a correction, without knowing the source URL. Now, Crawl error sources makes the process of tracking down the causes of "Not found" errors a piece of cake. This helps you improve the user experience on your site and gives you a jump start for links week (check out our updated post on "Good times with inbound links" to get the scoop).
  
  In our "Not Found" and "Errors for URLs in Sitemaps" reports, we've added the "Linked From" column. For every error in these reports, the "Linked From" column now lists the number of pages that link to a specific "Not found" URL.
  
  
  
  Clicking on an item in the "Linked From" column opens a separate dialog box which lists each page that linked to this URL along with the date it was discovered. The source URL for the 404 can be within or external to your site.
  
  
  
  
  
  For those of you who just want the data, we've also added the ability to download all your crawl error sources at once. Just click the "Download all sources of errors on this site" link to download all your site's crawl error sources.
  
  
  
  Again, if we report crawl errors for your website, you can use crawl error sources to quickly determine if the cause is from your site or someone else's. You'll have the information you need to contact them to get it fixed, and if needed, you can still put in place redirects on your own site to the appropriate URL. Just sign in to Webmaster Tools and check it out for your verified site. You can help people visiting your site—from anywhere on the web—find what they're looking for.
  
  Written by Jonathan Simon, Webmaster Trends Analyst and Michael Williamson, Webmaster Tools Intern
• 

  Good times with inbound links

  Posted: October 10th, 2008, 1:17am CEST by Maile Ohye

  Tags:  crawling and indexing  

  Inbound links are links from pages on external sites linking back to your site. Inbound links can bring new users to your site, and when the links are merit-based and freely-volunteered as an editorial choice, they're also one of the positive signals to Google about your site's importance. Other signals include things like our analysis of your site's content, its relevance to a geographic location, etc. As many of you know, relevant, quality inbound links can affect your PageRank (one of many factors in our ranking algorithm). And quality links often come naturally to sites with compelling content or offering a unique service.
  
  How do these signals factor into ranking?
  
  Let's say I have a site, example.com, that offers users a variety of unique website templates and design tips. One of the strongest ranking factors is my site's content. Additionally, perhaps my site is also linked from three sources -- however, one inbound link is from a spammy site. As far as Google is concerned, we want only the two quality inbound links to contribute to the PageRank signal in our ranking.
  
  Given the user's query, over 200 signals (including the analysis of the site's content and inbound links as mentioned above) are applied to return the most relevant results to the user.
  
  
  So how can you engage more users and potentially increase merit-based inbound links?
  
  Many webmasters have written about their success in growing their audience. We've compiled several ideas and resources that can improve the web for all users.
  

  Create unique and compelling content on your site and the web in general

  • Start a blog: make videos, do original research, and post interesting stuff on a regular basis. If you're passionate about your site's topic, there are lots of great avenues to engage more users.
    
    If you're interested in blogging, see our Help Center for specific tips for bloggers.
  
  
  • Teach readers new things, uncover new news, be entertaining or insightful, show your expertise, interview different personalities in your industry and highlight their interesting side. Make your site worthwhile.
  
  
  • Participate thoughtfully in blogs and user reviews related to your topic of interest. Offer your knowledgeable perspective to the community.
  
  
  • Provide a useful product or service. If visitors to your site get value from what you provide, they're more likely to link to you.
  
  For more actionable ideas, see one of my favorite interviews with Matt Cutts for no-cost tips to help increase your traffic. It's a great primer for webmasters. (Even before this post, I forwarded the URL to many of my friends. :)

  Pursue business development opportunities

  Use Webmaster Tools for "Links > Pages with external links" to learn about others interested in your site. Expand the web community by figuring out who links to you and how they're linking. You may have new audiences or demographics you didn't realize were interested in your niche. For instance, if the webmasters for example.com noticed external links coming from art schools, they may start to engage with the art community -- receiving new feedback and promoting their site and ideas.
  
  Of course, be responsible when pursuing possible opportunities in this space. Don't engage in mass link-begging; no one likes form letters, and few webmasters of quality sites are likely to respond positively to such solicitations. In general, many of the business development techniques that are successful in human relationships can also be reflected online for your site.

  Now that you've read more information about internal links, outbound links, and inbound links (today's post :), we'll see you in the blog comments! Thanks for joining us for links week.
  
  Update -- Here's one more business development opportunity:
  Investigate your "Diagnostics > Web/mobile crawl > Crawl error sources" to not only correct broken links, but also to cultivate relationships with external webmasters who share an interest in your site. (And while you're chatting, see if they'll correct the broken link. :) This is a fantastic way to turn broken links into free links to important parts of your site.
  
  In addition to contacting these webmasters, you may also wish to use 301 redirects to redirect incoming traffic from old pages to their new locations. This is good for users who may still have bookmarks with links to your old pages... and you'll be happy to know that Google appropriately flows PageRank and related signals through these redirects.
  
  Written by Maile Ohye, Developer Programs Tech Lead
• 

  Importance of link architecture

  Posted: October 7th, 2008, 12:51am CEST by Maile Ohye

  Tags:  crawling and indexing  

  In Day 2 of links week, we'd like to discuss the importance of link architecture and answer more advanced questions on the topic. Link architecture—the method of internal linking on your site—is a crucial step in site design if you want your site indexed by search engines. It plays a critical role in Googlebot's ability to find your site's pages and ensures that your visitors can navigate and enjoy your site.
  
  Keep important pages within several clicks from the homepage
  
  Although you may believe that users prefer a search box on your site rather than category navigation, it's uncommon for search engine crawlers to type into search boxes or navigate via pulldown menus. So make sure your important pages are clickable from the homepage and for easy for Googlebot to find throughout your site. It's best to create a link architecture that's intuitive for users and crawlable for search engines. Here are more ideas to get started:

  Intuitive navigation for users
  
  Create common user scenarios, get "in character," then try working through your site. For example, if your site is about basketball, imagine being a visitor (in this case a "baller" :) trying to learn the best dribbling technique.
  

  • Starting at the homepage, if the user doesn't use the search box on your site or a pulldown menu, can they easily find the desired information (ball handling like a superstar) from the navigation links?
  
  
  • Let's say a user found your site through an external link, but they didn't land on the homepage. Starting from any (sub-/child) page on your site, make sure they can easily find their way to the homepage and/or other relevant sections. In other words, make sure users aren't trapped or stuck. Was the "best dribbling technique" easy for your imaginary user to find? Often breadcrumbs such as "Home > Techniques > Dribbling" help users to understand where they are.

  Crawlable links for search engines
  

  • Text links are easily discovered by search engines and are often the safest bet if your priority is having your content crawled. While you're welcome to try the latest technologies, keep-in-mind that when text-based links are available and easily navigable for users, chances are that search engines can crawl your site as well.
    
    This <a href="new-page.html">text link</a> is easy for search engines to find.
  
  
  • Sitemap submission is also helpful for major search engines, though it shouldn't be a substitute for crawlable link architecture. If your site utilizes newer techniques, such as AJAX, see "Verify that Googlebot finds your internal links" below.

  Use descriptive anchor text
  
  Writing descriptive anchor text, the clickable words in a link, is a useful signal to help search engines and users alike to better understand your content. The more Google knows about your site—through your content, page titles, anchor text, etc.—the more relevant results we can return for users (and your potential search visitors). For example, if you run a basketball site and you have videos to accompany the textual content, a not-very-optimal way of linking would be:
  
  To see all our basketball videos, <a href="videos.html">click here</a> for the entire listing.
  
  However, instead of the generic "click here," you could rewrite the anchor text more descriptively as:
  
  Feel free to browse all of our <a href="videos.html">basketball videos</a>.
  
  Verify that Googlebot finds your internal links
  
  For verified site owners, Webmaster Tools has the feature "Links > Pages with internal links" that's great for verifying that Googlebot finds most of the links you'd expect. This is especially useful if your site uses navigation involving JavaScript (which Googlebot doesn't always execute)—you'll want to make sure that Googlebot is finding other internal links as expected.
  
  Here's an abridged snapshot of our internal links to the introductory post for "404 week at Webmaster Central." Our internal links are discovered as we had hoped.
  
  
  Feel free to ask more internal linking questions
  Here are some to get you started...
  
  Q: What about using rel="nofollow" for maximizing PageRank flow in my internal link architecture (such as PageRank sculpting, or PageRank siloing)?
  

  A: It's not something we, as webmasters who also work at Google, would really spend time or energy on. In other words, if your site already has strong link architecture, it's far more productive to work on keeping users happy with fresh and compelling content rather than to worry about PageRank sculpting.
  
  Matt Cutts answered more questions about "appropriate uses of nofollow" in our webmaster discussion group.

  Q: Let's say my website is about my favorite hobbies: biking and camping. Should I keep my internal linking architecture "themed" and not cross-link between the two?

  A: We haven't found a case where a webmaster would benefit by intentionally "theming" their link architecture for search engines. And, keep-in-mind, if a visitor to one part of your site can't easily reach other parts of your site, that may be a problem for search engines as well.

  Perhaps it's cliche, but at the end of the day, and at the end of this post, :) it's best to create solid link architecture (making navigation intuitive for users and crawlable for search engines)—implementing what makes sense for your users and their experience on your site.
  
  Thanks for your time today! Information about outbound links will soon be available in Day 3 of links week. And, if you have helpful tips about internal links or questions for our team, please share them in the comments below.
  
  Written by Maile Ohye, Developer Programs Tech Lead
• 

  Links information straight from the source

  Posted: October 6th, 2008, 4:41am CEST by Maile Ohye

  Tags:  crawling and indexing  

  We hope that you're able to focus on helping users (and improving the web) by creating great content or providing a great service on your site. In between creating content and working on your site, you may have read some of the (often conflicting) link discussions circling the web. If you're asking, "What's going on -- what do I need to know about links?" then welcome to the first day of links week!
  
  Day 2: Internal links (links within your site)

  Internal linking is your homepage linking to your "Contact us" page, or your "Contact us" page linking to your "About me" page. Internal linking (also known as link architecture) is important because it's a major factor in how easily visitors can navigate your site. Additionally, internal linking contributes to your site's "crawlability" -- how easily a spider can reach your pages. More in Day 2 of links week.

  Day 3: Outbound links (sites you link to)

  Outbound links are external sites that you're linking to. For example, www.google.com/webmasters links to the domain googlewebmastercentral.blogspot.com (our lovely blog!). Outbound links allow us to surf the web -- they're a big reason why the web is so exciting and collaborative. Without outbound links, your site can seem isolated from the community because each page becomes "brochure-ware." Most sites include outbound links naturally and it shouldn't be a big concern. If you still have questions, we'll be covering outbound linking in more detail on Day 3.

  Day 4: Inbound links (sites linking to you)

  Inbound links are external sites linking to you. There are many webmasters who (rightfully) aren't preoccupied by the subject of inbound links. So why do some webmasters care? It's likely because merit-based or volunteered inbound links may seem like a quick way to increase rankings and traffic. Answers to your questions like, "Are there no-cost methods to maximize my merit-based links?" are provided on Day 4.

  Update: Included references to blog posts as they were published throughout links week.
  
  Written by Maile Ohye, Developer Programs Tech Lead
• 

  More webmaster questions - Answered!

  Posted: September 23rd, 2008, 3:24pm CEST by Maile Ohye

  Tags:  crawling and indexing  

  When it comes to answering your webmaster related questions, we just can't get enough. I wanted to follow-up and answer some additional questions that webmasters asked in our latest installment of Popular Picks. In case you missed it, you can find our answers to image search ranking, sitelinks, reconsideration requests, redirects, and our communication with webmasters in this blog post.
  
  
  
  Check out these resources for additional details on questions answered in the video:

  • Duplicate Content: check out Susan's blog post, Demystifying the "duplicate content penalty."
  • Website Design: we've written a thorough set of Help Center articles on creating a Google-friendly site.
  • Meta tags: you can read our Help Center article on how to block or remove pages using meta tags.
  • Local Search: Some tips to rank well for local searches include signing up with Google's Local Business Center, registering the appropriate top level domain or using the geographic target tool in Webmaster Tools.
  • Problems with Webmaster Tools: check the Webmaster Tools subgroup in our Help Group. If you're experiencing errors, other webmasters may be experiencing the same. In the past, we've pinned a message at the top of the Webmaster Tools subgroup to let users know we're aware of a problem and working to fix it.

  Video Transcript:
  
  Hi everyone, I'm Reid from the Search Quality team. Today I'd like to answer some of the unanswered questions from our latest round of popular picks.
  
  Searchmaster had a question about duplicate content. Understandably, this is a popular concern from webmasters. You should check out the Google Webmaster Central Blogwhere my colleague Susan Moskwa recently posted "Demystifying the 'duplicate content penalty," which answers many questions and concerns about duplicate content.
  
  Jay is the Boss wanted to know if e-commerce websites suffer if they have two or more different themes. For example, you could have a site that sells auto parts, but also sightseeing guides. In general, I'd encourage webmasters to create a website that they feel is relevant for users. If it makes sense to sell auto parts and sightseeing guides, then go for it. Those are the sites that perform well, because users want to visit those sites, and they'll link to them as well.
  
  emma2 wanted to know if Google will follow links on a page using the "noindex" attribute in the "robots" meta tag. To answer this question, Googlebot will follow links on a page which uses the meta "noindex" tag, but that page will not appear in our search results. As a reminder, if you would like to prevent Googlebot from crawling any links on a page, use the "nofollow" attribute in the "robots" meta tag.
  
  Aaron Pratt wanted to know about some ways a webmaster can rank well for local searches. A quick recommendation is to add your business to the Local Business Center. There, you can add contact information as well as store operating hours and coupons as well. Another example, or a tip, is to take advantage and purchase a country-specific top-level domain, or use the geotargeting feature in Webmaster Tools.
  
  jdeb901 said it would be helpful if we could let webmasters know if we are having problems with Webmaster Tools. This is an excellent point, and we're always thinking about better ways to communicate with webmasters. If you're having problems with Webmaster Tools, chances are someone else is as well, and they've posted to the Google Webmaster Help Group about this. In the past, if we've experienced problems with Webmaster Tools, we've also created a "sticky" post to let users know that we know about these issues with Webmaster Tools, and we're working to find a solution.
  
  Well, that about wraps it up with our Popular Picks. Thanks again for all of your questions, and I look forward to seeing you around the group.
  
  Written by Reid Yokoyama, Search Quality
• 

  Dynamic URLs vs. static URLs

  Posted: September 22nd, 2008, 5:20pm CEST by Maile Ohye

  Tags:  crawling and indexing  

  Chatting with webmasters often reveals widespread beliefs that might have been accurate in the past, but are not necessarily up-to-date any more. This was the case when we recently talked to a couple of friends about the structure of a URL. One friend was concerned about using dynamic URLs, since (as she told us) "search engines can't cope with these." Another friend thought that dynamic URLs weren't a problem at all for search engines and that these issues were a thing of the past. One even admitted that he never understood the fuss about dynamic URLs in comparison to static URLs. For us, that was the moment we decided to read up on the topic of dynamic and static URLs. First, let's clarify what we're talking about:
  
  What is a static URL?
  A static URL is one that does not change, so it typically does not contain any url parameters. It can look like this: http://www.example.com/archive/january.htm. You can search for static URLs on Google by typing filetype:htm in the search field. Updating these kinds of pages can be time consuming, especially if the amount of information grows quickly, since every single page has to be hard-coded. This is why webmasters who deal with large, frequently updated sites like online shops, forum communities, blogs or content management systems may use dynamic URLs.
  
  What is a dynamic URL?
  If the content of a site is stored in a database and pulled for display on pages on demand, dynamic URLs maybe used. In that case the site serves basically as a template for the content. Usually, a dynamic URL would look something like this: http://code.google.com/p/google-checkout-php-sample-code/issues/detail?id=31. You can spot dynamic URLs by looking for characters like: ? = &. Dynamic URLs have the disadvantage that different URLs can have the same content. So different users might link to URLs with different parameters which have the same content. That's one reason why webmasters sometimes want to rewrite their URLs to static ones.
  
  Should I try to make my dynamic URLs look static?
  Following are some key points you should keep in mind while dealing with dynamic URLs:
  

  1. It's quite hard to correctly create and maintain rewrites that change dynamic URLs to static-looking URLs.
  2. It's much safer to serve us the original dynamic URL and let us handle the problem of detecting and avoiding problematic parameters.
  3. If you want to rewrite your URL, please remove unnecessary parameters while maintaining a dynamic-looking URL.
  4. If you want to serve a static URL instead of a dynamic URL you should create a static equivalent of your content.

  Which can Googlebot read better, static or dynamic URLs?
  We've come across many webmasters who, like our friend, believed that static or static-looking URLs were an advantage for indexing and ranking their sites. This is based on the presumption that search engines have issues with crawling and analyzing URLs that include session IDs or source trackers. However, as a matter of fact, we at Google have made some progress in both areas. While static URLs might have a slight advantage in terms of clickthrough rates because users can easily read the urls, the decision to use database-driven websites does not imply a significant disadvantage in terms of indexing and ranking. Providing search engines with dynamic URLs should be favored over hiding parameters to make them look static.
  
  Let's now look at some of the widespread beliefs concerning dynamic URLs and correct some of the assumptions which spook webmasters. :)
  
  Myth: "Dynamic URLs cannot be crawled."
  Fact: We can crawl dynamic URLs and interpret the different parameters. We might have problems crawling and ranking your dynamic URLs if you try to make your urls look static and in the process hide parameters which offer the Googlebot valuable information. One recommendation is to avoid reformatting a dynamic URL to make it look static. It's always advisable to use static content with static URLs as much as possible, but in cases where you decide to use dynamic content, you should give us the possibility to analyze your URL structure and not remove information by hiding parameters and making them look static.
  
  Myth: "Dynamic URLs are okay if you use fewer than three parameters."
  Fact: There is no limit on the number of parameters, but a good rule of thumb would be to keep your URLs short (this applies to all URLs, whether static or dynamic). You may be able to remove some parameters which aren't essential for Googlebot and offer your users a nice looking dynamic URL. If you are not able to figure out which parameters to remove, we'd advise you to serve us all the parameters in your dynamic URL and our system will figure out which ones do not matter. Hiding your parameters keeps us from analyzing your URLs properly and we won't be able to recognize the parameters as such, which could cause a loss of valuable information.
  
  Following are some questions we thought you might have at this point.
  
  Does that mean I should avoid rewriting dynamic URLs at all?
  That's our recommendation, unless your rewrites are limited to removing unnecessary parameters, or you are very diligent in removing all parameters that could cause problems. If you transform your dynamic URL to make it look static you should be aware that we might not be able to interpret the information correctly in all cases. If you want to serve a static equivalent of your site, you might want to consider transforming the underlying content by serving a replacement which is truly static. One example would be to generate files for all the paths and make them accessible somewhere on your site. However, if you're using URL rewriting (rather than making a copy of the content) to produce static-looking URLs from a dynamic site, you could be doing harm rather than good. Feel free to serve us your standard dynamic URL and we will automatically find the parameters which are unnecessary.
  
  Can you give me an example?
  If you have a dynamic URL which is in the standard format like foo?key1=value&key2=value2 we recommend that you leave the url unchanged, and Google will determine which parameters can be removed; or you could remove uncessary parameters for your users. Be careful that you only remove parameters which do not matter. Here's an example of a URL with a couple of parameters:
  
  www.example.com/article/bin/answer.foo?language=en&answer=3&sid=98971298178906&query=URL
  

  • language=en - indicates the language of the article
  • answer=3 - the article has the number 3
  • sid=8971298178906 - the session ID number is 8971298178906
  • query=URL - the query with which the article was found is [URL]

  Not all of these parameters offer additional information. So rewriting the URL to www.example.com/article/bin/answer.foo?language=en&answer=3 probably would not cause any problems as all irrelevant parameters are removed.
  
  The following are some examples of static-looking URLs which may cause more crawling problems than serving the dynamic URL without rewriting:

  • www.example.com/article/bin/answer.foo/en/3/98971298178906/URL
  • www.example.com/article/bin/answer.foo/language=en/answer=3/
    sid=98971298178906/query=URL
  • www.example.com/article/bin/answer.foo/language/en/answer/3/
    sid/98971298178906/query/URL
  • www.example.com/article/bin/answer.foo/en,3,98971298178906,URL

  Rewriting your dynamic URL to one of these examples could cause us to crawl the same piece of content needlessly via many different URLs with varying values for session IDs (sid) and query. These forms make it difficult for us to understand that URL and 98971298178906 have nothing to do with the actual content which is returned via this URL. However, here's an example of a rewrite where all irrelevant parameters have been removed:

  • www.example.com/article/bin/answer.foo/en/3

  Although we are able to process this URL correctly, we would still discourage you from using this rewrite as it is hard to maintain and needs to be updated as soon as a new parameter is added to the original dynamic URL. Failure to do this would again result in a static looking URL which is hiding parameters. So the best solution is often to keep your dynamic URLs as they are. Or, if you remove irrelevant parameters, bear in mind to leave the URL dynamic as the above example of a rewritten URL shows:

  • www.example.com/article/bin/answer.foo?language=en&answer=3

  We hope this article is helpful to you and our friends to shed some light on the various assumptions around dynamic URLs. Please feel free to join our discussion group if you have any further questions.
  
  Written by Juliane Stiller and Kaspar Szymanski, Search Quality Team
• 

  Demystifying the "duplicate content penalty"

  Posted: September 12th, 2008, 10:30am CEST by Susan Moskwa

  Tags:  crawling and indexing  

  Duplicate content. There's just something about it. We keep writing about it, and people keep asking about it. In particular, I still hear a lot of webmasters worrying about whether they may have a "duplicate content penalty."

  Let's put this to bed once and for all, folks: There's no such thing as a "duplicate content penalty." At least, not in the way most people mean when they say that.

  There are some penalties that are related to the idea of having the same content as another site—for example, if you're scraping content from other sites and republishing it, or if you republish content without adding any additional value. These tactics are clearly outlined (and discouraged) in our Webmaster Guidelines:

  • Don't create multiple pages, subdomains, or domains with substantially duplicate content.
  • Avoid... "cookie cutter" approaches such as affiliate programs with little or no original content.
  • If your site participates in an affiliate program, make sure that your site adds value. Provide unique and relevant content that gives users a reason to visit your site first.

  (Note that while scraping content from others is discouraged, having others scrape you is a different story; check out this post if you're worried about being scraped.)

  But most site owners whom I hear worrying about duplicate content aren't talking about scraping or domain farms; they're talking about things like having multiple URLs on the same domain that point to the same content. Like www.example.com/skates.asp?color=black&brand=riedell and www.example.com/skates.asp?brand=riedell&color=black. Having this type of duplicate content on your site can potentially affect your site's performance, but it doesn't cause penalties. From our article on duplicate content:

  Duplicate content on a site is not grounds for action on that site unless it appears that the intent of the duplicate content is to be deceptive and manipulate search engine results. If your site suffers from duplicate content issues, and you don't follow the advice listed above, we do a good job of choosing a version of the content to show in our search results.

  This type of non-malicious duplication is fairly common, especially since many CMSs don't handle this well by default. So when people say that having this type of duplicate content can affect your site, it's not because you're likely to be penalized; it's simply due to the way that web sites and search engines work.

  Most search engines strive for a certain level of variety; they want to show you ten different results on a search results page, not ten different URLs that all have the same content. To this end, Google tries to filter out duplicate documents so that users experience less redundancy. You can find details in this blog post, which states:

  1. When we detect duplicate content, such as through variations caused by URL parameters, we group the duplicate URLs into one cluster.
  2. We select what we think is the "best" URL to represent the cluster in search results.
  3. We then consolidate properties of the URLs in the cluster, such as link popularity, to the representative URL.

  Here's how this could affect you as a webmaster:

  • In step 2, Google's idea of what the "best" URL is might not be the same as your idea. If you want to have control over whether www.example.com/skates.asp?color=black&brand=riedell or www.example.com/skates.asp?brand=riedell&color=black gets shown in our search results, you may want to take action to mitigate your duplication. One way of letting us know which URL you prefer is by including the preferred URL in your Sitemap.
  • In step 3, if we aren't able to detect all the duplicates of a particular page, we won't be able to consolidate all of their properties. This may dilute the strength of that content's ranking signals by splitting them across multiple URLs.

  In most cases Google does a good job of handling this type of duplication. However, you may also want to consider content that's being duplicated across domains. In particular, deciding to build a site whose purpose inherently involves content duplication is something you should think twice about if your business model is going to rely on search traffic, unless you can add a lot of additional value for users. For example, we sometimes hear from Amazon.com affiliates who are having a hard time ranking for content that originates solely from Amazon. Is this because Google wants to stop them from trying to sell Everyone Poops? No; it's because how the heck are they going to outrank Amazon if they're providing the exact same listing? Amazon has a lot of online business authority (most likely more than a typical Amazon affiliate site does), and the average Google search user probably wants the original information on Amazon, unless the affiliate site has added a significant amount of additional value.

  Lastly, consider the effect that duplication can have on your site's bandwidth. Duplicated content can lead to inefficient crawling: when Googlebot discovers ten URLs on your site, it has to crawl each of those URLs before it knows whether they contain the same content (and thus before we can group them as described above). The more time and resources that Googlebot spends crawling duplicate content across multiple URLs, the less time it has to get to the rest of your content.

  In summary: Having duplicate content can affect your site in a variety of ways; but unless you've been duplicating deliberately, it's unlikely that one of those ways will be a penalty. This means that:

  • You typically don't need to submit a reconsideration request when you're cleaning up innocently duplicated content.
  • If you're a webmaster of beginner-to-intermediate savviness, you probably don't need to put too much energy into worrying about duplicate content, since most search engines have ways of handling it.
  • You can help your fellow webmasters by not perpetuating the myth of duplicate content penalties! The remedies for duplicate content are entirely within your control. Here are some good places to start.

  Posted by Susan Moskwa, Webmaster Trends Analyst

• 

  More on 404

  Posted: August 15th, 2008, 4:52pm CEST by Maile Ohye

  Tags:  crawling and indexing  

  Now that we've bid farewell to soft 404s, in this post for 404 week we'll answer your burning 404 questions.
  
  How do you treat the response code 410 "Gone"?
  Just like a 404.
  
  Do you index content or follow links from a page with a 404 response code?
  We aim to understand as much as possible about your site and its content. So while we wouldn't want to show a hard 404 to users in search results, we may utilize a 404's content or links if it's detected as a signal to help us better understand your site.
  
  Keep in mind that if you want links crawled or content indexed, it's far more beneficial to include them in a non-404 page.
  
  What about 404s with a 10-second meta refresh?
  Yahoo! currently utilizes this method on their 404s. They respond with a 404, but the 404 content also shows:
  
  <meta http-equiv="refresh" content="10;url=http://www.yahoo.com/?xxx">
  
  We feel this technique is fine because it reduces confusion by giving users 10 seconds to make a new selection, only offering the homepage after 10 seconds without the user's input.
  
  Should I 301-redirect misspelled 404s to the correct URL?
  Redirecting/301-ing 404s is a good idea when it's helpful to users (i.e. not confusing like soft 404s). For instance, if you notice that the Crawl Errors of Webmaster Tools shows a 404 for a misspelled version of your URL, feel free to 301 the misspelled version of the URL to the correct version.
  
  For example, if we saw this 404 in Crawl Errors:
  http://www.google.com/webmsters  <-- typo for "webmasters"
  
  we may first correct the typo if it exists on our own site, then 301 the URL to the correct version (as the broken link may occur elsewhere on the web):
  http://www.google.com/webmasters
  
  Have you guys seen any good 404s?
  Yes, we have! (Confession: no one asked us this question, but few things are as fun to discuss as response codes. :) We've put together a list of some of our favorite 404 pages. If you have more 404-related questions, let us know, and thanks for joining us for 404 week!

  [www.metrokitchen.com]
  "If you're looking for an item that's no longer stocked (as I was), this makes it really easy to find an alternative."
  -Riona, domestigeek
  [www.comedycentral.com]
  "Blame the robot monkeys"
  -Reid, tells really bad jokes
  [www.splicemusic.com]
  "Boost your 'Time on site' metrics with a 404 page like this."
  -Susan, dabbler in music and Analytics
  [www.treachery.net]
  "It's not reassuring, but it's definitive."
  -Jonathan, has trained actual spiders to build websites, ants handle the 404s
  [www.apple.com]
  "Good with respect to usability."
  [thcnet.net]
  "At least there's a mailbox."
  -JohnMu, adventurous
  [lookitsme.co.uk]
  "It's pretty cute. :)"
  -Jessica, likes cute things
  [www.orangecoat.com]
  "Flow charts rule."
  -Sahala, internet traveller
  [icanhascheezburger.com]
  "I can has useful links and even e-mail address for questions! But they could have added 'OH NOES! IZ MISSING PAGE! MAYBE TIPO OR BROKN LINKZ?' so folks'd know what's up."
  -Adam, lindy hop geek

  
  Written by Maile Ohye, Developer Programs Tech Lead
• 

  Farewell to soft 404s

  Posted: August 12th, 2008, 4:54pm CEST by Maile Ohye

  Tags:  crawling and indexing  

  We see two kinds of 404 ("File not found") responses on the web: "hard 404s" and "soft 404s." We discourage the use of so-called "soft 404s" because they can be a confusing experience for users and search engines. Instead of returning a 404 response code for a non-existent URL, websites that serve "soft 404s" return a 200 response code. The content of the 200 response is often the homepage of the site, or an error page.
  
  How does a soft 404 look to the user? Here's a mockup of a soft 404: This site returns a 200 response code and the site's homepage for URLs that don't exist.
  
  
  
  As exemplified above, soft 404s are confusing for users, and furthermore search engines may spend much of their time crawling and indexing non-existent, often duplicative URLs on your site. This can negatively impact your site's crawl coverage—because of the time Googlebot spends on non-existent pages, your unique URLs may not be discovered as quickly or visited as frequently.
  
  What should you do instead of returning a soft 404?
  It's much better to return a 404 response code and clearly explain to users that the file wasn't found. This makes search engines and many users happy.
  
  Return 404 response code
  
  
  Return clear message to users
  
  
  Can your webserver return 404, but send a helpful "Not found" message to the user?
  Of course! More info as "404 week" continues!
  
  Written by Maile Ohye, Developer Programs Tech Lead
• 

  It's 404 week at Webmaster Central

  Posted: August 11th, 2008, 3:40pm CEST by Maile Ohye

  Tags:  crawling and indexing  

  This week we're publishing several blog posts dedicated to helping you with one response code: 404.
  
  Response codes are a numeric status (like 200 for "OK", 301 for "Moved Permanently") that a webserver returns in response to a request for a URL. The 404 response code should be returned for a file "Not Found".
  
  When a user sends a request for your webpage, your webserver looks for the corresponding file for the URL. If a file exists, your webserver likely responds with a 200 response code along with a message (often the content of the page, such as the HTML).
  
  
  
  So what's a 404? Let's say that in the link to "Visit Google Apps" above, the link is broken because of a typing error when coding the page. Now when a user clicks "Visit Google Apps", the particular webpage/file isn't located by the webserver. The webserver should return a 404 response code, meaning "Not Found".
  
  
  
  Now that we're all on board with the basics of 404s, stay tuned 4 even more information on making 404s good 4 users and 4 search engines.
  
  Written by Maile Ohye, Developer Programs Tech Lead
• 

  To infinity and beyond? No!

  Posted: August 5th, 2008, 3:27pm CEST by Maile Ohye

  Tags:  crawling and indexing  

  When Googlebot crawls the web, it often finds what we call an "infinite space". These are very large numbers of links that usually provide little or no new content for Googlebot to index. If this happens on your site, crawling those URLs may use unnecessary bandwidth, and could result in Googlebot failing to completely index the real content on your site.
  
  Recently, we started notifying site owners when we discover this problem on their web sites. Like most messages we send, you'll find them in Webmaster Tools in the Message Center. You'll probably want to know right away if Googlebot has this problem - or other problems - crawling your sites. So verify your site with Webmaster Tools, and check the Message Center every now and then.
  
  
  
  Examples of an infinite space
  
  The classic example of an "infinite space" is a calendar with a "Next Month" link. It may be possible to keep following those "Next Month" links forever! Of course, that's not what you want Googlebot to do. Googlebot is smart enough to figure out some of those on its own, but there are a lot of ways to create an infinite space and we may not detect all of them.
  
  
  Another common scenario is websites which provide for filtering a set of search results in many ways. A shopping site might allow for finding clothing items by filtering on category, price, color, brand, style, etc. The number of possible combinations of filters can grow exponentially. This can produce thousands of URLs, all finding some subset of the items sold. This may be convenient for your users, but is not so helpful for the Googlebot, which just wants to find everything - once!
  
  Correcting infinite space issues
  
  Our Webmaster Tools Help article describes more ways infinite spaces can arise, and provides recommendations on how to avoid the problem. One fix is to eliminate whole categories of dynamically generated links using your robots.txt file. The Help Center has lots of information on how to use robots.txt. If you do that, don't forget to verify that Googlebot can find all your content some other way. Another option is to block those problematic links with a "nofollow" link attribute. If you'd like more information on "nofollow" links, check out the Webmaster Help Center.
  
  Written by Torrey Hoffman, Webmaster Tools team
• 

  A day in the life of webmaster support

  Posted: July 30th, 2008, 7:08pm CEST by Maile Ohye

  Tags:  feedback and communication  

  It was an ordinary Wednesday morning when I decided to film my day at the Googleplex...
  
  
  
  A recap of the day's events
  
  10AM Meeting with the webmaster support team

  Our team shares a doc containing our current agenda and the previous meetings' agenda, minutes, and action items. In this meeting, we discussed:

  • Feedback from blog post on Duplicate content due to scrapers. Some webmasters suggested that we could improve our detection. In order to improve quality, it would help to get feedback with specific examples. Susan Moskwa, one of our Webmaster Trends Analysts based in Kirkland, Washington, volunteered to post a blog comment to solicit more information.
  
  
  • Recent and upcoming releases
  • Webmaster Tools API on schedule
  • "Skip intro" in search results
  
  
  • JuneTune online chat agenda
  
  
  • Two recent spam techniques mentioned in the blogosphere. Brian White, who leads one of the Webspam-fighting groups at Google, explained that one technique is new twist on old idea, both are already handled.

  11AM Meeting with Matt Cutts

  Matt provided feedback on:

  • Proposal to write follow-up blog comment on duplicate content caused by scrapers to solicit specific examples. Approved.
  
  
  • "URLs and case sensitivity basics" presentation for online chat

  1PM Lunch with Shyam, a Crawl engineer, and Jason, and AdSense engineer
  2PM Meeting with Wysz to review slides for Google Trifecta
  5PM Little "drive-by" to catch Reid, Evan, Charlene, Jessica, and Wysz while they're monitoring the discussion group
  7PM Dinner with Matthias
  p.s. A huge thanks to Wysz for his film editing skillz.
  
  Written by Maile Ohye, Developer Programs Tech Lead
• 

  Google Trifecta recording now available

  Posted: July 17th, 2008, 5:18pm CEST by Michael Wyszomierski

  Tags:  products and services  

  Last Tuesday, Google hosted an online presentation about three free tools to help analyze and improve your sites. Dubbed "The Google Trifecta," this presentation explains the key features of Google Webmaster Tools, Google Analytics, and Google Website Optimizer, and how you can use them together.
  
  To make the information from this presentation accessible to those who were unable to listen in last week or would like to review it again, we've put a recording of the session on YouTube for your viewing pleasure:
  
  
  
  If you have any followup questions about the topics covered in the video, we welcome you to ask! Each of these products has a discussion group available for you to discuss the product with other users and Googlers. Please post in the appropriate group, so your question is seen by the right people:
  
  Google Webmaster Help (Webmaster Tools)
  Analytics Help
  Website Optimizer Forum
  
  Posted by Michael Wyszomierski, Search Quality Team
• 

  Improved Flash indexing

  Posted: June 30th, 2008, 11:31pm CEST by A Googler

  Tags:  crawling and indexing  

  Written by Ron Adler, Janis Stipins, and Maile Ohye
  
  We've received numerous requests to improve our indexing of Adobe Flash files. Today, Ron Adler and Janis Stipins—software engineers on our indexing team—will provide us with more in-depth information about our recent announcement that we've greatly improved our ability to index Flash.
  
  Q: Which Flash files can Google better index now?
  We've improved our ability to index textual content in SWF files of all kinds. This includes Flash "gadgets" such as buttons or menus, self-contained Flash websites, and everything in between.
  
  Q: What content can Google better index from these Flash files?
  All of the text that users can see as they interact with your Flash file. If your website contains Flash, the textual content in your Flash files can be used when Google generates a snippet for your website. Also, the words that appear in your Flash files can be used to match query terms in Google searches.
  
  In addition to finding and indexing the textual content in Flash files, we're also discovering URLs that appear in Flash files, and feeding them into our crawling pipeline—just like we do with URLs that appear in non-Flash webpages. For example, if your Flash application contains links to pages inside your website, Google may now be better able to discover and crawl more of your website.
  
  Q: What about non-textual content, such as images?
  At present, we are only discovering and indexing textual content in Flash files. If your Flash files only include images, we will not recognize or index any text that may appear in those images. Similarly, we do not generate any anchor text for Flash buttons which target some URL, but which have no associated text.
  
  Also note that we do not index FLV files, such as the videos that play on YouTube, because these files contain no text elements.
  
  Q: How does Google "see" the contents of a Flash file?
  We've developed an algorithm that explores Flash files in the same way that a person would, by clicking buttons, entering input, and so on. Our algorithm remembers all of the text that it encounters along the way, and that content is then available to be indexed. We can't tell you all of the proprietary details, but we can tell you that the algorithm's effectiveness was improved by utilizing Adobe's new Searchable SWF library.
  
  Q: What do I need to do to get Google to index the text in my Flash files?
  Basically, you don't need to do anything. The improvements that we have made do not require any special action on the part of web designers or webmasters. If you have Flash content on your website, we will automatically begin to index it, up to the limits of our current technical ability (see next question).
  
  That said, you should be aware that Google is now able to see the text that appears to visitors of your website. If you prefer Google to ignore your less informative content, such as a "copyright" or "loading" message, consider replacing the text within an image, which will make it effectively invisible to us.
  
  Q: What are the current technical limitations of Google's ability to index Flash?
  There are three main limitations at present, and we are already working on resolving them:
  
  1. Googlebot does not execute some types of JavaScript. So if your web page loads a Flash file via JavaScript, Google may not be aware of that Flash file, in which case it will not be indexed.
  2. We currently do not attach content from external resources that are loaded by your Flash files. If your Flash file loads an HTML file, an XML file, another SWF file, etc., Google will separately index that resource, but it will not yet be considered to be part of the content in your Flash file.
  3. While we are able to index Flash in almost all of the languages found on the web, currently there are difficulties with Flash content written in bidirectional languages. Until this is fixed, we will be unable to index Hebrew language or Arabic language content from Flash files.
  
  We're already making progress on these issues, so stay tuned!
  
  Update: Everyone, thanks for your great questions and feedback. Our focus is to improve search quality for all users, and with better Flash indexing we create more meaningful search results. Listed below, we’ve also answered some of the most prevalent questions. Thanks again!
  
  Flash site in search results before improvements
  
  
  Flash site after improved indexing, querying [nasa deep impact animation]
  
  
  Helping us access and index your Flash files
  

  @fintan: We verified with Adobe that the textual content from legacy sites, such as those scripted with AS1 and AS2, can be indexed by our new algorithm.
  
  @andrew, jonny m, erichazann, mike, ledge, stu, rex, blog, dis: For our July 1st launch, we didn't enable Flash indexing for Flash files embedded via SWFObject. We're now rolling out an update that enables support for common JavaScript techniques for embedding Flash, including SWFObject and SWFObject2.
  
  @mike: At this time, content loaded dynamically from resource files is not indexed. We’ve noted this feature request from several webmasters -- look for this in a near future update.

  
  Interaction of HTML pages and Flash
  

  @captain cuisine: The text found in Flash files is treated similarly to text found in other files, such as HTML, PDFs, etc. If the Flash file is embedded in HTML (as many of the Flash files we find are), its content is associated with the parent URL and indexed as single entity.
  
  @jeroen: Serving the same content in Flash and an alternate HTML version could cause us to find duplicate content. This won't cause a penalty -- we don’t lower a site in ranking because of duplicate content. Be aware, though, that search results will most likely only show one version, not both.
  
  @All: We’re trying to serve users the most relevant results possible regardless of the file type. This means that standalone Flash, HTML with embedded Flash, HTML only, PDFs, etc., can all have the potential to be returned in search results.

  
  Indexing large Flash files
  

  @dsfdgsg: We’ve heard requests for deep linking (linking to specific content inside file) not just for Flash results, but also for other large documents and presentations. In the case of Flash, the ability to deep link will require additional functionality in Flash with which we integrate.
  
  @All: The majority of the existing Flash files on the web are fine in regard to filesize. It shouldn’t be too much of a concern.

  
  More details about our Flash indexing algorithm
  

  @brian, marcos, bharath: Regarding ActionScript, we’re able to find new links loaded through ActionScript. We explore Flash like a website visitor does, we do not decompile the SWF file. Unless you're making ActionScript visible to users, Google will not expose ActionScript code.
  
  @dlocks: We respect rel="nofollow" wherever we encounter it in HTML.

• 

  What are your SEO recommendations?

  Posted: June 30th, 2008, 2:31pm CEST by Susan Moskwa

  Tags:  feedback and communication  

  Posted by the Webmaster Trends Analysts (John, Jonathan, Susan)

  You may have noticed that we recently rewrote our article on What is an SEO? Does Google recommend them? Previously, the article had focused on warning people about common SEO scams to look out for, but didn't mention many of the valuable services that a helpful SEO can provide.

  The article now notes some of the benefits of search engine optimization, and provides some guidance to site owners who are considering hiring an SEO. We'd also like to get your perspective: how would you define SEO? What questions would you ask a prospective SEO? What advice would you give to an inexperienced webmaster who's considering whether to contract an SEO? We'd like to hear your thoughts and incorporate your feedback if there's important advice that we should add.

• 

  One year of monitored European Webmaster Help Groups

  Posted: June 24th, 2008, 7:27am CEST by John Mueller

  Tags:  feedback and communication  

  Written by Kaspar Szymanski and the European Guides Team, Dublin
  
  A year ago Google Guides from the Search Quality Team started monitoring Google Webmaster Help Groups in European languages. It has been an interesting time and a great, rewarding experience for all those involved :o). We have enjoyed building and growing our communities with webmasters across Europe and we hope you have as well. The feedback we have been receiving since the beginning has been encouraging, with many webmasters responding positively to the opportunity to be able to solve their indexing issues in their native language. Still, we are aware that we have a lot of work ahead of us. I would like to thank everyone who has been active in our communities and invite those who have not joined yet to do so. Sign up, participate and help us make a difference.
  
  Taking advantage of this anniversary, I would like the European Google Guides to have a chance to say a few words about their communities.
  
  Polish Webmaster Help Group
  Guglarz, the Polish Google Guide
  Our group grew very fast, especially in the last few months and we are well on the way to hitting 600 users! But what I appreciate even more then our group's success, is the open, friendly spirit we have been cultivating in the last twelve months :o). I feel like this is a community that is able to help with indexing issues and beyond!
  
  I already pointed out some exceptional community members in the last blog post about communication with webmasters in a dozen languages. Since then, a few more have joined the list of tremendously committed, group-driving contributors. Bigu has been jumping in every time I've been traveling or very busy, which is the reason why I’d like to thank him a lot.
  
  Also I want to acknowledge Maciej Gluszek for escalating language issues we used to struggle with. You guys help to improve the Google experience for everyone.
  
  Lastly, I want to invite all group members to the chit chat section in order to introduce yourself and share some more information about who you are and what you do. I am really curious to find out more about the people I work with! I hope to see you in the group soon :o).
  
  ---
  Группа помощи Google для веб-мастеров
  Oxana, Tilek, Vitali, Mariya
  Our Russian Webmaster Help group is growing by the day and just reached an important milestone - a whole kilobyte of members: 1024 (-:! Thanks to all our users for the great questions, the wisdom, and the humour which they bring to our community. Special thanks go to Web-Master, Crazy LionHeart, Andrey Morozov and Lmd for their spot-on comments and prolific posts - we really appreciate your dedication in helping others. We're very happy to be part of such a savvy and lively community, we look forward to more interesting questions and discussions in the future, and to reaching one megabyte of members (-:! С днем рождения!
  
  ---
  Google Webmester Súgó Csoport
  Tibor, János
  Although the Hungarian Webmaster Help Group is one of our youngest, it already counts 300+ members, and has a pretty high number of lively discussions. The atmosphere is really great and loads of issues are resolved thanks largely to our dedicated and wise users. I would especially like to thank snomag for helping greatly in building the group up in the beginning (where have you disappeared to? We miss you! :)
  
  ---
  Google Site Yöneticisi Yardım Grubu
  Bahar
  The Turkish Webmaster Help Group has grown into a dynamic and effective community of fantastic webmasters in just four months. I want to thank to everyone in the group who has been eager to share his/her knowledge to help others. Special thanks to Erkan (man_blood) who has written a huge amount of posts in the last three months; Merve who has helped the group with her technical skills as well as calm spirit, especially when I was on vacation; Salih (SesVer) who used to help a lot when the group was first launched; and many more who are not mentioned by name here. It has been exciting and a pleasure to be a part of this rapidly growing family! Desteğiniz için teşekkürler :-)
  
  ---
  קבוצת העזרה למנהלי ומנהלות אתרים
  Alon, your Google Guide
  Our Hebrew group is maturing. To date we have more than 200 users and we are gaining visibility while more users are using the well-informed discussions we produce.
  
  It is a pleasure to see so many webmasters stopping by to ask, learn and help others. I would like to take the opportunity to thank all of you who make the group a place for discussion and collaboration. My special thanks go to Shoshanna, AliaG, Tomer and Seosos, for their great contribution.
  
  By browsing through the “tell us about yourself ” thread you can learn more about our community where many members introduce themselves. You're also welcome to watch my short video giving some specific tips for Israeli Webmasters.
  
  ---
  Google diskussionsgruppe for webmasters
  GoogleGuide (Jonas)
  The Danish group has been slowly gaining more subscribers since its launch last year, and we still hope to attract more people. We've grown almost 100% in the past year, and we hope to keep up the steam! (: This growth wouldn't have been possible without the help of Anders who contributes equal amounts of webmaster goodness in the form of sharing his tips on 301's and moving content to a new domain (which has been the hottest and most debated topic in the group), to issues on getting your site better indexed and crawled. So a big thanks to Anders, and all the others who have made this group useful for everybody.
  
  ---
  Ajuda a Webmasters do Google
  Pedro
  The Portuguese Webmaster Help Group is now above 1200 members. The success of the group is a reflection on the great community built by every single contributor. Special thanks to Carlos Lavieri, Rodrigo Soares, Leandro Leite e João Carlos (Orquiza) who used to help so much and recently Flávio Raimundo (M&S) and Ruben Zevallos Jr.
  
  Recently I had the chance of giving a few public interviews to Portuguese speaking webmasters and recognizing the group and its success publicly. I'd also like to take the opportunity to thank my interviewers for their great questions; if you speak Portuguese or you are a Google Translator fan you can read them: Entrevista com Pedro Dias (by Paulo Teixeira) and Entrevista a Pedro Dias (by Kazulo Webmarketing). It is a great pleasure to be a part of such an awesome community.
  
  ---
  Googlen Keskusteluryhmä Verkkovastaaville
  Anu
  The Finnish Webmaster Help Group will be celebrating its first birthday soon. During the first year, we've built a small but dynamic community and covered various topics from general webmaster issues to language-related toughies. A huge thank you to everyone who has taken part in discussions! I'm looking forward to starting our second year as a group, hope to see both familiar faces and newcomers join in our community!
  
  ---
  Groupe Google d'entraide pour les webmasters
  Alec, Nathalie
  
  The French Webmaster Help Group was launched one year ago and we've had a tremendous growth thanks to all our fantastic members. Special thanks to Paul-Arnaud and Cyrille who used to help so much and to Thierry who is now our bionic poster :-)
  
  Recently I had the chance to discuss the Webmaster Help Group with a French SEO I met at SES Paris '08. It was so great to connect again on the Internet and help him with his issues.
  
  ---
  Swedish Webmaster Help Group
  Hessam, the Swedish Google Guide
  The small Swedish webmaster forum, with currently almost 200 users, has almost tripled its size since its launch last July. During the past year we have seen a steady growth in user participation and a number of questions from our Swedish webmasters. I also had the chance to attend Search Marketing Expo (SMX) Stockholm 2007 which was definitely the highlight of the year. There I was able to meet a number of Swedish webmasters to talk about Google, our Webmaster Tools as well as the cold Swedish winter. During the coming year we are determined to further grow the community to better support our users. Lastly, a big thanks goes to everyone who has supported the group, especially Mattias Nordin and Widar Nord who have contributed with their knowledge and time. I am looking forward to hearing more from our regulars and would like to extend an invite to new members.
  
  ---
  Google Diskussionsforum für Webmaster
  Uli
  With more than 2700 members and almost a thousand posts per month, the German Webmaster Help Group has developed into a vibrant community. You will find many very savvy webmasters there who are always prepared to discuss all webmaster-related issues and offer advice to both beginners and advanced webmasters.
  
  I would like to take this opportunity to thank everyone who has contributed to this forum. Some webmasters like luzie, alpaka, beejees, Garzosch have been a crucial part of our group since this group began, and with the likes of Zombie joining in October, Mikki in January and Sistrix in March after having the pleasure of meeting him at SMX in Munich, we have an ever-growing pool of wisdom :-)
  
  ---
  Foro de Google para Webmasters
  Rebecca, Alvar
  The Spanish webmaster group now has almost 2500 members. We have some wonderful webmasters that offer great advice and support, people like Ricardo Antonio, Enrique de Mesa, Jesús Cáceres (chupi), Jose Antonio Nobile (nobilesoft), Carles Pastor, and many others who help make this forum work as well as it does. Recently we had one member visit whose site had been hacked with a cloaking technique and the community was able to help point out how to detect the hidden links and clear the hacked content. It’s great to see his website clean again and up and running.
  
  If you want to know who is behind this forum, have a look at the video where we introduce the forum and Webmaster Tools. Additionally, please give any feedback you may have in the thread where we present the video. By the way, it was great meeting some of you guys face to face at the SMX Madrid conference.
  
  ---
  Google Discussiegroep voor Webmaster
  Andre, Jos
  Our Dutch Webmaster Help Group now has well over 700 members. Some of our regular posters, like Marketsharer, Janus and Joop are doing a great job helping a lot of our less experienced (as well as more experienced!) webmasters with their questions. Recently, some Dutch webmasters have been dealing with the fact that their sites have been hacked. We've added a sticky post to the group to help webmasters clean up their sites after getting hacked, so if you are dealing with this issue, do have a look at the group! At this moment, the group is still steadily growing and we are really looking forward to reaching the 1000 members mark!
  
  ---
  Google Assistenza Webmaster
  Lella, Alessandro, Stefano
  The Italian webmaster group is about to turn one year old pretty soon and has been a great success. We have a bunch of fabulous witty power posters like Angelo Palma (Angelo), Marco (Customsoft) and others that keep on massively contributing to the success of the group day by day. Over the last year we achieved excellent results both in terms of traffic increase and new users posting frequently. Coordinating the localization of the post on paid links - later translated into 14 languages - was a turning point for the Webmaster Help team. So was Lella's blog post in English, German and Italian about the Search Engine Strategies Conference in London, where she represented Google as a speaker.
  
  ---
  
  Webmaster Help Group Guides and Domo-Kun, the team mascot
  
  Speaking on behalf of all our Google Guides, we are looking forward to continued collaboration with webmasters all over the world. Surely the upcoming 12 months are going to be exciting for the Google Help Groups as the communities continue to grow. Thanks again to all the great webmasters out there and see you soon in the discussions!
• 

  A new layer to Google Trends

  Posted: June 20th, 2008, 11:13am CEST by Maile Ohye

  Tags:  products and services  

  Written by R.J. Pittman, Director of Product Management - Consumer Search Properties
  
  Two years ago, we launched Google Trends, a tool that lets anyone see what the world is searching for, and compare the world's interest in your favorite topics. Last year, we added Hot Trends, which shows what people are searching for right now - the fastest rising search queries on Google, updated every hour. And just last week, we introduced normalized search volume numbers available for export in Google Trends.
  
  Today, we add a new layer to Trends with Google Trends for Websites, a fun tool that gives you a view of how popular your favorite websites are, including your own! It also compares and ranks site visitation across geographies, and related websites and searches.
  
  Let's take a look at one example, the release of Radiohead's In Rainbows album. As part of our annual Zeitgeist, we post the fastest rising search terms, and this past year, radiohead took the crown as the fastest rising search term in the last quarter of 2007.
  
  Using Google Trends, we can see the search volume for radiohead compared to in rainbows over the last 12 months.
  
  
  radiohead vs. in rainbows
  
  With Hot Trends, we can see that on October 10th (the release date of In Rainbows), people were most interested in downloading In Rainbows and reviews of the new release.
  
  Now, using Trends for Websites, this story can be viewed from another perspective: we can see how the number of unique visitors that visit radiohead.com and inrainbows.com has changed over the last 12 months, the countries where the sites are most popular, the top related sites and search terms.
  
  
  radiohead.com vs. inrainbows.com
  
  We can see that in October 2007, radiohead.com (the blue line) saw a huge surge in popularity. The release of In Rainbows clearly drove many people to visit the band's website. The album's website, inrainbows.com (the red line), saw an even more dramatic jump; probably because that is where you could actually download the new album. And, it looks like site traffic to radiohead.com has increased overall, even as inrainbows.com traffic declined.
  
  Keep in mind that Trends for Websites is a Google Labs product and that we are experimenting with ways to improve the quality of the data. Because data is estimated and aggregated over a variety of sources, it may not match the other data sources you rely on for web traffic information. For more information, be sure to check out our Website Owners FAQs.
  
  To start using Trends for Websites, head over to Google Trends.
• 

  Join us for another live chat - June 19, 2008

  Posted: June 16th, 2008, 9:46pm CEST by Adam Lasnik

  Tags:  feedback and communication  

  Written by Adam Lasnik, Search Evangelist
  When it comes to talking with webmasters, we just can't get enough.
  This past March we had the pleasure of connecting with hundreds of you online in our first-ever Webmaster Help Group online live chat, which included a presentation on Images in Google Search by Maile, lots of feedback from you on webmaster issues, a site clinic, and dozens of questions answered by folks from our Google Webmaster Central team.
  Given the success of this previous chat, we've decided to do it again.  We're hosting another free live chat (dubbed JuneTune), and we'd love to have you attend!Here's what you'll need:

  • About an hour of free time.
  • A phone, or ability to make a phone call online (e.g., with Skype or Gizmo5, etc.)
  • An internet-connected computer, and these additional specifications.
  • A URL for the chat, which you can only get when you register for the event  (you can likely do this in under 60 seconds)

  What will our JuneTune chat include?  

  • INTRO:  A quick hello from some of your favorite Help Group Guides
  • PRESO:  A presentation on Personalization in Google Search by our own John Mueller.
  • FAQs:  We're calling this "Three for Three," and we'll have three different Googlers tackling three different issues we've seen come up in the Group recently.  What will they be?  You'll just have to attend the chat to find out!
  • And lots of Q&A!  You'll have a chance to type questions during the entire session, and we'll pick as many as we can to answer in writing and in speaking during the chat.

  When and how can you join in?

  • Mark the date on your calendar now: Thursday, June 19, for about one hour starting at 2:00pm PDT / 5:00pm EDT / 21:00 UTC / 23:00 CET 
  • Register right now for this event. Please note that you'll need to click on the "register" link on the lefthand side.
  • Using the link e-mailed to you by WebEx (the service hosting the event), log in 5-10 minutes prior to 2pm PDT.

  
  We hope you can stop by, and look forward to chatting with you!  In the meantime, if you have any questions, feel free to post a note in this Groups thread.
• 

  Duplicate content due to scrapers

  Posted: June 9th, 2008, 5:40am CEST by Search Quality Team

  Tags:  crawling and indexing  

  Written by Sven Naumann, Search Quality Team
  
  Since duplicate content is a hot topic among webmasters, we thought it might be a good time to address common questions we get asked regularly at conferences and on the Google Webmaster Help Group.
  
  Before diving in, I'd like to briefly touch on a concern webmasters often voice: in most cases a webmaster has no influence on third parties that scrape and redistribute content without the webmaster's consent. We realize that this is not the fault of the affected webmaster, which in turn means that identical content showing up on several sites in itself is not inherently regarded as a violation of our webmaster guidelines. This simply leads to further processes with the intent of determining the original source of the content—something Google is quite good at, as in most cases the original content can be correctly identified, resulting in no negative effects for the site that originated the content.
  
  Generally, we can differentiate between two major scenarios for issues related to duplicate content:
  

  • Within-your-domain-duplicate-content, i.e. identical content which (often unintentionally) appears in more than one place on your site
    
  
  
  • Cross-domain-duplicate-content, i.e. identical content of your site which appears (again, often unintentionally) on different external sites
    

  With the first scenario, you can take matters into your own hands to avoid Google indexing duplicate content on your site. Check out Adam Lasnik's post Deftly dealing with duplicate content and Vanessa Fox's Duplicate content summit at SMX Advanced, both of which give you some great tips on how to resolve duplicate content issues within your site. Here's one additional tip to help avoid content on your site being crawled as duplicate: include the preferred version of your URLs in your Sitemap file. When encountering different pages with the same content, this may help raise the likelihood of us serving the version you prefer. Some additional information on duplicate content can also be found in our comprehensive Help Center article discussing this topic.
  
  In the second scenario, you might have the case of someone scraping your content to put it on a different site, often to try to monetize it. It's also common for many web proxies to index parts of sites which have been accessed through the proxy. When encountering such duplicate content on different sites, we look at various signals to determine which site is the original one, which usually works very well. This also means that you shouldn't be very concerned about seeing negative effects on your site's presence on Google if you notice someone scraping your content.
  
  In cases when you are syndicating your content but also want to make sure your site is identified as the original source, it's useful to ask your syndication partners to include a link back to your original content. You can find some additional tips on dealing with syndicated content in a recent post by Vanessa Fox, Ranking as the original source for content you syndicate.
  
  Some webmasters have asked what could cause scraped content to rank higher than the original source. That should be a rare case, but if you do find yourself in this situation:
  

  • Check if your content is still accessible to our crawlers. You might unintentionally have blocked access to parts of your content in your robots.txt file.
  
  
  • You can look in your Sitemap file to see if you made changes for the particular content which has been scraped.
  
  
  • Check if your site is in line with our webmaster guidelines.
    

  To conclude, I'd like to point out that in the majority of cases, having duplicate content does not have negative effects on your site's presence in the Google index. It simply gets filtered out. If you check out some of the tips mentioned in the resources above, you'll basically learn how to have greater control about what exactly we're crawling and indexing and which versions are more likely to appear in the index. Only when there are signals pointing to deliberate and malicious intent, occurrences of duplicate content might be considered a violation of the webmaster guidelines.
  
  If you would like to further discuss this topic, feel free to visit our Webmaster Help Group.
  
  For the German version of this post, go to "Duplicate Content aufgrund von Scraper-Sites".
• 

  Improving on Robots Exclusion Protocol

  Posted: June 3rd, 2008, 11:00am CEST by koppula

  Tags:  crawling and indexing  

  Written by Prashanth Koppula, Product Manager
  
  Web publishers often ask us how they can maximize their visibility on the web. Much of this has to do with search engine optimization -- making sure a publisher's content shows up on all the search engines.
  
  However, there are some cases in which publishers need to communicate more information to search engines -- like the fact that they don't want certain content to appear in search results. And for that they use something called the Robots Exclusion Protocol (REP), which lets publishers control how search engines access their site: whether it's controlling the visibility of their content across their site (via robots.txt) or down to a much more granular level for individual pages (via META tags).
  
  Since it was introduced in the early '90s, REP has become the de facto standard by which web publishers specify which parts of their site they want public and which parts they want to keep private. Today, millions of publishers use REP as an easy and efficient way to communicate with search engines. Its strength lies in its flexibility to evolve in parallel with the web, its universal implementation across major search engines and all major robots, and in the way it works for any publisher, no matter how large or small.
  
  While REP is observed by virtually all search engines, we've never come together to detail how we each interpret different tags. Over the last couple of years, we have worked with Microsoft and Yahoo! to bring forward standards such as Sitemaps and offer additional tools for webmasters. Since the original announcement, we have, and will continue to, deliver further improvements based on what we are hearing from the community.
  
  Today, in that same spirit of making the lives of webmasters simpler, we're releasing detailed documentation about how we implement REP. This will provide a common implementation for webmasters and make it easier for any publisher to know how their REP directives will be handled by three major search providers -- making REP more intuitive and friendly to even more publishers on the web.
  
  So, without further ado...
  
  Common REP Directives
  The following list are all the major REP features currently implemented by Google, Microsoft, and Yahoo!. With each feature, you'll see what it does and how you should communicate it.
  
  Each of these directives can be specified to be applicable for all crawlers or for specific crawlers by targeting them to specific user-agents, which is how any crawler identifies itself. Apart from the identification by user-agent, each of our crawlers also supports Reverse DNS based authentication to allow you to verify the identity of the crawler.
  
  1. Robots.txt Directives
  

  DIRECTIVE	IMPACT	USE CASES
  Disallow	Tells a crawler not to index your site -- your site's robots.txt file still needs to be crawled to find this directive, however disallowed pages will not be crawled	'No Crawl' page from a site. This directive in the default syntax prevents specific path(s) of a site from being crawled.
  Allow	Tells a crawler the specific pages on your site you want indexed so you can use this in combination with Disallow	This is useful in particular in conjunction with Disallow clauses, where a large section of a site is disallowed except for a small section within it
  $ Wildcard Support	Tells a crawler to match everything from the end of a URL -- large number of directories without specifying specific pages	'No Crawl' files with specific patterns, for example, files with certain filetypes that always have a certain extension, say pdf
  * Wildcard Support	Tells a crawler to match a sequence of characters	'No Crawl' URLs with certain patterns, for example, disallow URLs with session ids or other extraneous parameters
  Sitemaps Location	Tells a crawler where it can find your Sitemaps	Point to other locations where feeds exist to help crawlers find URLs on a site

  
  2. HTML META Directives
  

  DIRECTIVE	IMPACT	USE CASES
  NOINDEX META Tag	Tells a crawler not to index a given page	Don't index the page. This allows pages that are crawled to be kept out of the index.
  NOFOLLOW META Tag	Tells a crawler not to follow a link to other content on a given page	Prevent publicly writeable areas to be abused by spammers looking for link credit. By using NOFOLLOW you let the robot know that you are discounting all outgoing links from this page.
  NOSNIPPET META Tag	Tells a crawler not to display snippets in the search results for a given page	Present no snippet for the page on Search Results
  NOARCHIVE META Tag	Tells a search engine not to show a "cached" link for a given page	Do not make available to users a copy of the page from the Search Engine cache
  NOODP META Tag	Tells a crawler not to use a title and snippet from the Open Directory Project for a given page	Do not use the ODP (Open Directory Project) title and snippet for this page

  
  
  These directives are applicable for all forms of content. They can be placed in either the HTML of a page or in the HTTP header for non-HTML content, e.g., PDF, video, etc. using an X-Robots-Tag. You can read more about it here:X-Robots-Tag Post or in our series of posts about using robots and Meta Tags.
  
  Other REP Directives
  The directives listed above are used by Microsoft, Google and Yahoo!, but may not be implemented by all other search engines. In addition, the following directives are supported by Google, but are not supported by all three as are those above:
  
  UNAVAILABLE_AFTER Meta Tag - Tells a crawler when a page should "expire", i.e., after which date it should not show up in search results.
  
  NOIMAGEINDEX Meta Tag - Tells a crawler not to index images for a given page in search results.
  
  NOTRANSLATE Meta Tag - Tells a crawler not to translate the content on a page into different languages for search results.
  
  
  Going forward, we plan to continue to work together to ensure that as new uses of REP arise, we're able to make it as easy as possible for webmasters to use them. So stay tuned for more!
  
  Learn more
  You can find out more about robots.txt at http://www.robotstxt.org and at Google's Webmaster help center, which contains lots of helpful information, including:

  • How to create a robots.txt file
  • Descriptions of each user-agent that Google uses
  • How to use pattern matching
  • How often we recrawl your robots.txt file

  We've also done several posts in our webmaster blog about robots.txt that you may find useful, such as:

  • Using robots.txt files
  • All about Googlebot

  There is also a useful list of the bots used by the major search engines.
  
  To see what our colleagues have to say, you can also check out the blog posts published by Yahoo! and Microsoft.
• 

  How Google defines IP delivery, geolocation, and cloaking

  Posted: June 2nd, 2008, 1:06am CEST by Maile Ohye

  Tags:  crawling and indexing  

  Written by Maile Ohye
  
  Many of you have asked for more information regarding webserving techniques (especially related to Googlebot), so we made a short glossary of some of the more unusual methods.

  • Geolocation: Serving targeted/different content to users based on their location. As a webmaster, you may be able to determine a user's location from preferences you've stored in their cookie, information pertaining to their login, or their IP address. For example, if your site is about baseball, you may use geolocation techniques to highlight the Yankees to your users in New York.
    
    The key is to treat Googlebot as you would a typical user from a similar location, IP range, etc. (i.e. don't treat Googlebot as if it came from its own separate country—that's cloaking).
  
  
  • IP delivery: Serving targeted/different content to users based on their IP address, often because the IP address provides geographic information. Because IP delivery can be viewed as a specific type of geolocation, similar rules apply. Googlebot should see the same content a typical user from the same IP address would see.
    
    (Author's warning: This 7.5-minute video may cause drowsiness. Even if you're really interested in IP delivery or multi-language sites, it's a bit uneventful.)
  
  
  • Cloaking: Serving different content to users than to Googlebot. This is a violation of our webmaster guidelines. If the file that Googlebot sees is not identical to the file that a typical user sees, then you're in a high-risk category. A program such as md5sum or diff can compute a hash to verify that two different files are identical.
  
  
  • First click free: Implementing Google News' First click free policy for your content allows you to include your premium or subscription-based content in Google's websearch index without violating our quality guidelines. You allow all users who find your page using Google search to see the full text of the document, even if they have not registered or subscribed. The user's first click to your content area is free. However, you can block the user with a login or payment request when he clicks away from that page to another section of your site.
    
    If you're using First click free, the page displayed to users who visit from Google must be identical to the content that is shown to the Googlebot.

  Still have questions?  We'll see you at the related thread in our Webmaster Help Group.
• 

  Best practices when moving your site

  Posted: April 16th, 2008, 3:55pm CEST by Ríona MacNamara

  Tags:  crawling and indexing  

  Posted by Ríona MacNamara, Webmaster Tools Team
  

  Planning on moving your site to a new domain? Lots of webmasters find this a scary process. How do you do it without hurting your site's performance in Google search results?

  
  
  Your aim is to make the transition invisible and seamless to the user, and to make sure that Google knows that your new pages should get the same quality signals as the pages on your own site. When you're moving your site, pesky 404 (File Not Found) errors can harm the user experience and negatively impact your site's performance in Google search results.
  

  Let's cover moving your site to a new domain (for instance, changing from www.example.com to www.example.org). This is different from moving to a new IP address; read this post for more information on that.

  Here are the main points:

  • Test the move process by moving the contents of one directory or subdomain first. Then use a 301 Redirect to permanently redirect those pages on your old site to your new site. This tells Google and other search engines that your site has permanently moved.

  • Once this is complete, check to see that the pages on your new site are appearing in Google's search results. When you're satisfied that the move is working correctly, you can move your entire site. Don't do a blanket redirect directing all traffic from your old site to your new home page. This will avoid 404 errors, but it's not a good user experience. A page-to-page redirect (where each page on the old site gets redirected to the corresponding page on the new site) is more work, but gives your users a consistent and transparent experience. If there won't be a 1:1 match between pages on your old and new site, try to make sure that every page on your old site is at least redirected to a new page with similar content.

  • If you're changing your domain because of site rebranding or redesign, you might want to think about doing this in two phases: first, move your site; and second, launch your redesign. This manages the amount of change your users see at any stage in the process, and can make the process seem smoother. Keeping the variables to a minimum also makes it easier to troubleshoot unexpected behavior.

  • Check both external and internal links to pages on your site. Ideally, you should contact the webmaster of each site that links to yours and ask them to update the links to point to the page on your new domain. If this isn't practical, make sure that all pages with incoming links are redirected to your new site. You should also check internal links within your old site, and update them to point to your new domain. Once your content is in place on your new server, use a link checker like Xenu to make sure you don't have broken legacy links on your site. This is especially important if your original content included absolute links (like www.example.com/cooking/recipes/chocolatecake.html) instead of relative links (like .../recipes/chocolatecake.html).

  • To prevent confusion, it's best to make sure you retain control of your old site domain for at least 180 days.

  • Add your new site to your Webmaster Tools account, and verify your ownership of it. Then create and submit a Sitemap listing the URLs on your new site. This tells Google that your content is now available on your new site, and that we should go and crawl it.

  • Finally, keep both your new and old site verified in Webmaster Tools, and review crawl errors regularly to make sure that the 301s from the old site are working properly, and that the new site isn't showing unwanted 404 errors.

  We'll admit it, moving is never easy - but these steps should help ensure that none of your good web reputation falls off the truck in the process.
• 

  Crawling through HTML forms

  Posted: April 11th, 2008, 12:50pm CEST by Maile Ohye

  Tags:  crawling and indexing  

  Written by Jayant Madhavan and Alon Halevy, Crawling and Indexing Team
  
  Google is constantly trying new ideas to improve our coverage of the web. We already do some pretty smart things like scanning JavaScript and Flash to discover links to new web pages, and today, we would like to talk about another new technology we've started experimenting with recently.
  
  In the past few months we have been exploring some HTML forms to try to discover new web pages and URLs that we otherwise couldn't find and index for users who search on Google. Specifically, when we encounter a <FORM> element on a high-quality site, we might choose to do a small number of queries using the form. For text boxes, our computers automatically choose words from the site that has the form; for select menus, check boxes, and radio buttons on the form, we choose from among the values of the HTML. Having chosen the values for each input, we generate and then try to crawl URLs that correspond to a possible query a user may have made. If we ascertain that the web page resulting from our query is valid, interesting, and includes content not in our index, we may include it in our index much as we would include any other web page.
  
  Needless to say, this experiment follows good Internet citizenry practices. Only a small number of particularly useful sites receive this treatment, and our crawl agent, the ever-friendly Googlebot, always adheres to robots.txt, nofollow, and noindex directives. That means that if a search form is forbidden in robots.txt, we won't crawl any of the URLs that a form would generate.  Similarly, we only retrieve GET forms and avoid forms that require any kind of user information. For example, we omit any forms that have a password input or that use terms commonly associated with personal information such as logins, userids, contacts, etc. We are also mindful of the impact we can have on web sites and limit ourselves to a very small number of fetches for a given site.
  
  The web pages we discover in our enhanced crawl do not come at the expense of regular web pages that are already part of the crawl, so this change doesn't reduce PageRank for your other pages. As such it should only increase the exposure of your site in Google. This change also does not affect the crawling, ranking, or selection of other web pages in any significant way.
  
  This experiment is part of Google's broader effort to increase its coverage of the web. In fact, HTML forms have long been thought to be the gateway to large volumes of data beyond the normal scope of search engines. The terms Deep Web, Hidden Web, or Invisible Web have been used collectively to refer to such content that has so far been invisible to search engine users. By crawling using HTML forms (and abiding by robots.txt), we are able to lead search engine users to documents that would otherwise not be easily found in search engines, and provide webmasters and users alike with a better and more comprehensive search experience.
• 

  Speaking the language of robots

  Posted: March 27th, 2008, 2:27pm CET by Sean Harding

  Tags:  indexing and crawling  

  Written by Sean Harding, Webmaster Tools Team
  
  We all know how friendly Googlebot is. And like all benevolent robots, he listens to us and respects our wishes about parts of our site that we don't want crawled. We can just give him a robots.txt file explaining what we want, and he'll happily comply. But what if you're intimidated by the idea of communicating directly with Googlebot? After all, not all of us are fluent in the language of robots.txt. This is why we're pleased to introduce you to your personal robot translator: the Robots.txt Generator in Webmaster Tools. It's designed to give you an easy and interactive way to build a robots.txt file. It can be as simple as entering the files and directories you don't want crawled by any robots.
  
  Or, if you need to, you can create fine-grained rules for specific robots and areas of your site. Once you're finished with the generator, feel free to test the effects of your new robots.txt file with our robots.txt analysis tool. When you're done, just save the generated file to the top level (root) directory of your site, and you're good to go. There are a couple of important things to keep in mind about robots.txt files:

  • Not every search engine will support every extension to robots.txt files

  The Robots.txt Generator creates files that Googlebot will understand, and most other major robots will understand them too. But it's possible that some robots won't understand all of the robots.txt features that the generator uses.

  • Robots.txt is simply a request

  Although it's highly unlikely from a major search engine, there are some unscrupulous robots that may ignore the contents of robots.txt and crawl blocked areas anyway. If you have sensitive content that you need to protect completely, you should put it behind password protection rather than relying on robots.txt.
  
  We hope this new tool helps you communicate your wishes to Googlebot and other robots that visit your site. If you want to learn more about robots.txt files, check out our Help Center. And if you'd like to discuss robots.txt and robots with other webmasters, visit our Google Webmaster Help Group.