SEO News » Tags » and

Feeds

5016 items (4491 unread) in 3 feeds

Google (304 unread)

• Official Google Webmaster Central Blog (304 unread)

SEO (4187 unread)

• Emily Chang (334 unread)
• Search Engine Roundtable (3853 unread)

116 items tagged "and"

Related tags: indexing [+], crawling [+], feedback [+], communication [+], services [+], products [+]

Official Google Webmaster Central Blog

• 

  More Options for Google+ Badges

  Posted: January 26th, 2012, 2:53pm CET by Maile Ohye

  Tags:  products and services  

  Webmaster Level: All
  
  Update on February 2, 2012: The new Google+ badge is now out of preview and available to all users on all sites.
  
  When we launched Google+ pages in November, we also released Google+ badges to promote your Google+ presence right on your site. Starting today in developer preview (and soon available to all your users), we're adding more options for integrating the Google+ badge into your website. You can configure a badge with a width that fits your site design and choose a version that works better on darker sites. You'll also see that Google+ badges now include the unified +1 and circle count that we added to Pages last month.
  
  
  If you’re still considering whether to add a Google+ badge on your website, consider this: We recently looked at top sites using the badge and found that, on average, the badge accounted for an additional 38% of followers. When you add the badge visitors to your website can discover your Google+ page and connect in a variety of ways: they can follow your Google+ page, +1 your site, share your site with their circles, see which of their friends have +1’d your site, and click through to visit your Google+ page.
  
  The Google+ Badge makes it easy for your fans to find and follow you on Google+. With these additional options, we hope it's even easier to create a badge that fits your website.
  
  Follow the conversation on Google+.
  
  Posted by Lucy Hadden, Software Engineer, Google+
• 

  Making form-filling faster, easier and smarter

  Posted: January 25th, 2012, 11:00am CET by Maile Ohye

  Tags:  products and services  

  Webmaster Level: Intermediate
  
  One of the biggest bottlenecks on any conversion funnel is filling out an online form – shopping and registration flows all rely on forms as a crucial and demanding step in accomplishing the goals of your site. For many users, online forms mean repeatedly typing common information like our names and addresses on different sites across the web – a tedious task that causes many to give up and abandon the flow entirely.
  
  Chrome’s Autofill and other form-filling providers help to break down this barrier by remembering common profile information and pre-populating the form with those values. Unfortunately, up to now it has been difficult for webmasters to ensure that Chrome and other form-filling providers can parse their form correctly. Some standards exist; but they put onerous burdens on the implementation of the website, so they’re not used much in practice.
  
  Today we’re pleased to announce support in Chrome for an experimental new “autocomplete type” attribute for form fields that allows web developers to unambiguously label text and select fields with common data types such as ‘full-name’ or ‘street-address’. With this attribute, web developers can drive conversions on their sites by marking their forms for auto-completion without changing the user interface or the backend.
  
  
  Just add an attribute to the input element, for example an email address field might look like:
  
  <input type=”text” name=”field1” x-autocompletetype=”email” />
  
  We’ve been working on this design in collaboration with several other autofill vendors. Like any early stage proposal we expect this will change and evolve as the web standards community provides feedback, but we believe this will serve as a good starting point for the discussion on how to best support autofillable forms in the HTML5 spec. For now, this new attribute is implemented in Chrome as x-autocompletetype to indicate that this is still experimental and not yet a standard, similar to the webkitspeech attribute we released last summer.
  
  For more information, you can read the full text of the proposed specification, ask questions on the Webmaster help forum, or you can share your feedback in the standardization discussion!
  
  Posted by Ilya Sherman, Software Engineer
• 

  Grow your audience with Google+

  Posted: November 7th, 2011, 11:40am CET by Maile Ohye

  Tags:  products and services  

  Webmaster Level: All
  
  At Google, we help grow your audience by connecting you with new users. We introduced the +1 button so your site would stand out on search and your users could easily share your content on Google+. But, sometimes you want to join the conversation and post content directly to where people are sharing.
  
  Today we’re introducing Google+ for Business, a collection of tools and products that help you grow your audience. At the core of this is Google+ Pages, your site’s identity on Google+.
  
  Google+ Pages: Have real conversations with the right people
  To get your site on Google+, you first need to create a Google+ Page. On your page, you can engage in conversations with your visitors, direct readers back to your site for the latest updates, send tailored messages to specific groups of people, and see how many +1’s you have across the web. Google+ Pages will help you build relationships with your users, encouraging them to spend more time engaging with your content.
  
  Google+ Pages are at the heart of Google+ for Business
  Hangouts
  Sometimes you might want to chat with your users face-to-face.  For example, if you run a food blog, you may want to invite a chef to talk about her favorite recipe, or if you manage a fashion review site, beauty specialists might want to hold how-to sessions with makeup tips. Hangouts make this easy, by letting you have high-quality video chats with nine people with a single click. You can use Hangouts to hold live forums, break news or simply get to know people better, all in real time.
  
  Hangouts let you meet your customers, face-to-face
  Circles
  Circles allow you to group followers of your Page into smaller audiences. You can then share specific messages with specific groups. For example, you could create a Circle containing your most loyal readers and offer them exclusive content.
  The Google+ badge: Grow your audience on Google+
  
  To help your users find your page and start sharing, there are two buttons you can add to your site by visiting our Google+ badge configuration tool:
  
  The Google+ icon, a small icon that directly links to your Page.
  The Google+ badge, which we’re introducing in the coming days. This badge lets people add your page to their circles without leaving your site, and allows them to get updates from your site via Google+.
  
   
  
  Extend the power of +1, stand out in Google search
  You can also link your site to your Google+ page so that all your +1s -- from your Page, your website, and search results -- will get tallied together and appear as a single total. Potential visitors will be more likely to see the recommendations your site has received, whether they’re looking at a search result, your website, or your Page, meaning your +1’s will reach not only the 40 million users of Google+, but all the people who come to Google every day. You can link your site to your Page either using the Google+ badge or with a  piece of code. To set this up, visit our Google+ badge configuration tool.
  
  Bringing Google+ to the rest of Google
  Our ultimate vision for Google+ is to transform the overall Google experience -- weaving identity and sharing into all of our products. Beginning today, we’re rolling out a new experimental feature to a small group of eligible publishers, Google+ Direct Connect -- an easy way for your audience to find your Google+ Page on Google search.  If you’ve linked your Page to your site and you qualify, when someone searches for your website’s name with the ‘+’ sign before it Direct Connect will send them directly to your Page. For example, try searching for ‘+YouTube’ on Google. Users will also be prompted to automatically add Pages they find through Direct Connect to their circles. 
  
  Direct Connect suggestions start populating as you type on Google.com
  Just the beginning
  We want to help you get your site on Google+ as soon as possible, so we’re opening the field trial for Google+ Pages to everyone today. Creating a Google+ Page only takes a few minutes. To get started, you’ll need a personal Google+ profile. If you don’t have a Google account, it’s very quick and easy to join. And if you’re looking for inspiration, check out some of the sites that are already starting to set up their Pages:
  
  
  To learn more about how Google+ works for your site, check out the Google+ Your Business site. We’re just getting started, and have many more features planned for the coming weeks and months. To keep up to date on the latest news and tips, add the Google+ Your Business page to your circles. If you have ideas on how we can improve Google+ for your site, we’d love to hear them.
  
  Posted by Dennis Troper, Product Management Director, Google+ Pages
• 

  GET, POST, and safely surfacing more of the web

  Posted: November 1st, 2011, 4:53pm CET by Maile Ohye

  Tags:  crawling and indexing  

  Webmaster Level: Intermediate to Advanced
  
  As the web evolves, Google’s crawling and indexing capabilities also need to progress. We improved our indexing of Flash, built a more robust infrastructure called Caffeine, and we even started crawling forms where it makes sense. Now, especially with the growing popularity of JavaScript and, with it, AJAX, we’re finding more web pages requiring POST requests -- either for the entire content of the page or because the pages are missing information and/or look completely broken without the resources returned from POST. For Google Search this is less than ideal, because when we’re not properly discovering and indexing content, searchers may not have access to the most comprehensive and relevant results.
  
  We generally advise to use GET for fetching resources a page needs, and this is by far our preferred method of crawling. We’ve started experiments to rewrite POST requests to GET, and while this remains a valid strategy in some cases, often the contents returned by a web server for GET vs. POST are completely different. Additionally, there are legitimate reasons to use POST (e.g., you can attach more data to a POST request than a GET). So, while GET requests remain far more common, to surface more content on the web, Googlebot may now perform POST requests when we believe it’s safe and appropriate.
  
  We take precautions to avoid performing any task on a site that could result in executing an unintended user action. Our POSTs are primarily for crawling resources that a page requests automatically, mimicking what a typical user would see when they open the URL in their browser. This will evolve over time as we find better heuristics, but that’s our current approach.
  
  Let’s run through a few POSTs request scenarios that demonstrate how we’re improving our crawling and indexing to evolve with the web.
  
  Examples of Googlebot’s POST requests
  

  • Crawling a page via a POST redirect
    <html>
      <body onload="document.foo.submit();">
        <form name="foo" action="request.php" method="post">       <input type="hidden" name="bar" value="234"/>
        </form>
      </body>
    </html>
  • Crawling a resource via a POST XMLHttpRequest
    In this step-by-step example, we improve both the indexing of a page and its Instant Preview by following the automatic XMLHttpRequest generated as the page renders.
    
    
    1. Google crawls the URL, yummy-sundae.html.
    2. Google begins indexing yummy-sundae.html and, as a part of this process, decides to attempt to render the page to better understand its content and/or generate the Instant Preview.
    3. During the render, yummy-sundae.html automatically sends an XMLHttpRequest for a resource, hot-fudge-info.html, using the POST method.
       <html>
         <head>
           <title>Yummy Sundae</title>
           <script src="jquery.js"></script>
         </head>
         <body>
           This page is about a yummy sundae.
           <div id="content"></div>
           <script type="text/javascript">
             $(document).ready(function() {
               $.post('hot-fudge-info.html', function(data)
                 {$('#content').html(data);});
             });
           </script>
         </body>
       </html>
    4. The URL requested through POST, hot-fudge-info.html, along with its data payload, is added to Googlebot’s crawl queue.
    5. Googlebot performs a POST request to crawl hot-fudge-info.html.
    6. Google now has an accurate representation of yummy-sundae.html for Instant Previews. In certain cases, we may also incorporate the contents of hot-fudge-info.html into yummy-sundae.html.
    7. Google completes the indexing of yummy-sundae.html.
    8. User searches for [hot fudge sundae].
    9. Google’s algorithms can now better determine how yummy-sundae.html is relevant for this query, and we can properly display a snapshot of the page for Instant Previews.

  Improving your site’s crawlability and indexability
  
  General advice for creating crawlable sites is found in our Help Center. For webmasters who want to help Google crawl and index their content and/or generate the Instant Preview, here are a few simple reminders:

  • Prefer GET for fetching resources, unless there’s a specific reason to use POST.
  • Verify that we're allowed to crawl the resources needed to render your page. In the example above, if hot-fudge-info.html is disallowed by robots.txt, Googlebot won't fetch it. More subtly, if the JavaScript code that issues the XMLHttpRequest is located in an external .js file disallowed by robots.txt, we won't see the connection between yummy-sundae.html and hot-fudge-info.html, so even if the latter is not disallowed itself, that may not help us much. We've seen even more complicated chains of dependencies in the wild. To help Google better understand your site it's almost always better to allow Googlebot to crawl all resources.
    
    You can test whether resources are blocked through Webmaster Tools “Labs -> Instant Previews.”
  • Make sure to return the same content to Googlebot as is returned to users’ web browsers. Cloaking (sending different content to Googlebot than to users) is a violation of our Webmaster Guidelines because, among other things, it may cause us to provide a searcher with an irrelevant result -- the content the user views in their browser may be a complete mismatch from what we crawled and indexed. We’ve seen numerous POST-request examples where a webmaster non-maliciously cloaked (which is still a violation), and their cloaking -- on even the smallest of changes -- then caused JavaScript errors that prevented accurate indexing and completely defeated their reason for cloaking in the first place. Summarizing, if you want your site to be search-friendly, cloaking is an all-around sticky situation that’s best to avoid.
    
    To verify that you're not accidentally cloaking, you can use Instant Previews within Webmaster Tools, or try setting the User-Agent string in your browser to something like:
    
    Mozilla/5.0 (compatible; Googlebot/2.1;
      +http://www.google.com/bot.html)
    Your site shouldn't look any different after such a change. If you see a blank page, a JavaScript error, or if parts of the page are missing or different, that means that something's wrong.
  • Remember to include important content (i.e., the content you’d like indexed) as text, visible directly on the page and without requiring user-action to display. Most search engines are text-based and generally work best with text-based content. We’re always improving our ability to crawl and index content published in a variety of ways, but it remains a good practice to use text for important information.

  Controlling your content
  
  If you’d like to prevent content from being crawled or indexed for Google Web Search, traditional robots.txt directives remain the best method. To prevent the Instant Preview for your page(s), please see our Instant Previews FAQ which describes the “Google Web Preview” User-Agent and the nosnippet meta tag.
  
  Moving forward
  
  We’ll continue striving to increase the comprehensiveness of our index so searchers can find more relevant information. And we expect our crawling and indexing capability to improve and evolve over time, just like the web itself. Please let us know if you have questions or concerns.
  
  Written by Pawel Aleksander Fedorynski, Software Engineer, Indexing Team, and Maile Ohye, Developer Programs Tech Lead
• 

  Pagination with rel=“next” and rel=“prev”

  Posted: September 15th, 2011, 7:00am CEST by Maile Ohye

  Tags:  crawling and indexing  

  Webmaster level: Intermediate to Advanced
  
  Much like rel=”canonical” acts a strong hint for duplicate content, you can now use the HTML link elements rel=”next” and rel=”prev” to indicate the relationship between component URLs in a paginated series. Throughout the web, a paginated series of content may take many shapes—it can be an article divided into several component pages, or a product category with items spread across several pages, or a forum thread divided into a sequence of URLs. Now, if you choose to include rel=”next” and rel=”prev” markup on the component pages within a series, you’re giving Google a strong hint that you’d like us to:
  

  • Consolidate indexing properties, such as links, from the component pages/URLs to the series as a whole (i.e., links should not remain dispersed between page-1.html, page-2.html, etc., but be grouped with the sequence).
  • Send users to the most relevant page/URL—typically the first page of the series.

  
  
  The relationship between component URLs in a series can now be indicated to Google through rel=”next” and rel=”prev”.
  There’s an exception to the rel=”prev” and rel=”next” implementation: If, alongside your series of content, you also offer users a view-all page, or if you’re considering a view-all page, please see our post on View-all in search results for more information. Because view-all pages are most commonly preferred by searchers, we do our best to surface this version when appropriate in results rather than a component page (component pages are more likely to surface with rel=”next” and rel=”prev”).
  
  If you don’t have a view-all page or you’d like to override Google returning a view-all page, you can use rel="next" and rel="prev" as described in this post.
  
  For information on paginated configurations that include a view-all page, please see our post on View-all in search results.
  Outlining your options
  
  Here are three options for a series:
  

  1. Leave whatever you have exactly as-is. Paginated content exists throughout the web and we’ll continue to strive to give searchers the best result, regardless of the page’s rel=”next”/rel=”prev” HTML markup—or lack thereof.
  2. If you have a view-all page, or are considering a view-all page, see our post on View-all in search results.
  3. Hint to Google the relationship between the component URLs of your series with rel=”next” and rel=”prev”. This helps us more accurately index your content and serve to users the most relevant page (commonly the first page). Implementation details below.

  
  Implementing rel=”next” and rel=”prev”
  
  If you prefer option 3 (above) for your site, let’s get started! Let’s say you have content paginated into the URLs:
  
  http://www.example.com/article?story=abc&page=1
  http://www.example.com/article?story=abc&page=2
  http://www.example.com/article?story=abc&page=3
  http://www.example.com/article?story=abc&page=4
  
  On the first page, http://www.example.com/article?story=abc&page=1, you’d include in the <head> section:
  <link rel="next" href="http://www.example.com/article?story=abc&page=2" />
  
  On the second page, http://www.example.com/article?story=abc&page=2:
  <link rel="prev" href="http://www.example.com/article?story=abc&page=1" />
<link rel="next" href="http://www.example.com/article?story=abc&page=3" />
  
  On the third page, http://www.example.com/article?story=abc&page=3:
  <link rel="prev" href="http://www.example.com/article?story=abc&page=2" />
<link rel="next" href="http://www.example.com/article?story=abc&page=4" />
  
  And on the last page, http://www.example.com/article?story=abc&page=4:
  <link rel="prev" href="http://www.example.com/article?story=abc&page=3" />
  
  A few points to mention:
  

  • The first page only contains rel=”next” and no rel=”prev” markup.
  • Pages two to the second-to-last page should be doubly-linked with both rel=”next” and rel=”prev” markup.
  • The last page only contains markup for rel=”prev”, not rel=”next”.
  • rel=”next” and rel=”prev” values can be either relative or absolute URLs (as allowed by the <link> tag). And, if you include a <base> link in your document, relative paths will resolve according to the base URL.
  • rel=”next” and rel=”prev” only need to be declared within the <head> section, not within the document <body>.
  • We allow rel=”previous” as a syntactic variant of rel=”prev” links.
  • rel="next" and rel="previous" on the one hand and rel="canonical" on the other constitute independent concepts. Both declarations can be included in the same page. For example, http://www.example.com/article?story=abc&page=2&sessionid=123 may contain:
    
    <link rel="canonical" href="http://www.example.com/article?story=abc&page=2”/>
<link rel="prev" href="http://www.example.com/article?story=abc&page=1&sessionid=123" />
<link rel="next" href="http://www.example.com/article?story=abc&page=3&sessionid=123" />
    
    
  • rel=”prev” and rel=”next” act as hints to Google, not absolute directives.
  • When implemented incorrectly, such as omitting an expected rel="prev" or rel="next" designation in the series, we'll continue to index the page(s), and rely on our own heuristics to understand your content.

  
  Questions?
  More information can be found in our Help Center, or join the conversation in our Webmaster Help Forum!
  
  Written by Benjia Li & Joachim Kupke, Software Engineers, Indexing Team
• 

  View-all in search results

  Posted: September 15th, 2011, 7:00am CEST by Maile Ohye

  Tags:  crawling and indexing  

  Webmaster level: Intermediate to Advanced
  
  User testing has taught us that searchers much prefer the view-all, single-page version of content over a component page containing only a portion of the same information with arbitrary page breaks (which cause the user to click “next” and load another URL).
  
  
  Searchers often prefer the view-all vs. paginated content with arbitrary page breaks and worse latency.
  Therefore, to improve the user experience, when we detect that a content series (e.g. page-1.html, page-2.html, etc.) also contains a single-page version (e.g. page-all.html), we’re now making a larger effort to return the single-page version in search results. If your site has a view-all option, there’s nothing you need to do; we’ll work to do it on your behalf. Also, indexing properties, like links, will be consolidated from the component pages in the series to the view-all page.
  
  However, high latency can make the view-all less preferred
  
  Interestingly, the cases when users didn’t prefer the view-all page were correlated with high latency (e.g., when the view-all page took a while to load, say, because it contained many images). This makes sense because we know users are less satisfied with slow results. So while a view-all page is commonly desired, as a webmaster it’s important to balance this preference with the page’s load time and overall user experience.
  
  Best practices for a series of content
  

  1. If your site includes view-all pages
     
     We aim to detect the view-all version of your content and, if available, its associated component pages. There’s nothing more you need to do! However, if you’d like to make it more explicit to us, you can include rel=”canonical” from your component pages to your view-all to increase the likelihood that we detect your series of pages appropriately.
     
     
     rel=”canonical” can specify the superset of content (i.e. the view-all page, in this case page-all.html) from the same information in a series of URLs.
     Why does this work?

     In the diagram, page-2.html of a series may specify the canonical target as page-all.html because page-all.html is a superset of page-2.html's content. When a user searches for a query term and page-all.html is selected in search results, even if the query most related to page-2.html, we know the user will still see page-2.html’s relevant information within page-all.html.

     
     

     On the other hand, page-2.html shouldn’t designate page-1.html as the canonical because page-2.html’s content isn’t included on page-1.html. It’s possible that a user’s search query is relevant to content on page-2.html, but if page-2.html’s canonical is set to page-1.html, the user could then select page-1.html in search results and find herself in a position where she has to further navigate to a different page to arrive at the desired information. That’s a poor experience for the user, a suboptimal result from us, and it could also bring poorly targeted traffic to your site.

     
     However, if you strongly desire your view-all page not to appear in search results: 1) make sure the component pages in the series don’t include rel=”canonical” to the view-all page, and 2) mark the view-all page as “noindex” using any of the standard methods.
  2. If you’d like to surface individual, component pages (or there’s no view-all available)
     
     It may be the case that one or both of the situations below apply to your site:
     
     
     • The view-all page is undesirable as a search result (e.g., load time too high or too difficult for users to navigate).
     • Your users prefer the multi-page experience and to be directed to a component page in search results, rather than the view-all page.
     
     If so, you can use standard HTML rel=”next” and rel=”prev” elements to specify a relationship between the component pages in your series of content. If done correctly, Google will generally strive to:
     
     
     • Consolidate indexing properties, such as links, between the component pages/URLs.
     • Send users to the most relevant page/URL from the component pages. Typically, the most relevant page is the first page of your content, but our algorithms may point users to one of the component pages in the series.

  
  It’s not uncommon for webmasters to incorrectly use rel=”canonical” from component pages to the first page of their series (e.g. page-2.html with rel=”canonical” to page-1.html). We recommend against this implementation because the component pages don’t actually contain duplicate content. Using rel=”next” and rel=”prev” is far more appropriate.
  
  Summary
  
  Because users generally prefer the view-all option in search results, we’re making more of an effort to properly detect and serve this version to searchers. If you have a series of content, there’s nothing more you need to do. If you’d like to hint more to Google how best to serve users your information:
  

  1. To better optimize your view-all page, you can use rel=”canonical” from component pages to the single-page version; otherwise,
  2. If a view-all page doesn’t provide a good user experience for your site, you can use the rel=”next” and rel=”prev” attributes as a strong hint for Google to identify the series of pages and still surface a component page in results.

  
  Questions?
  
  As always, feel free to ask in our Webmaster Help Forum.
  
  Written by Benjia Li & Joachim Kupke, Software Engineers, Indexing Team
• 

  PDFs in Google search results

  Posted: September 1st, 2011, 9:23am CEST by Gary Illyes

  Tags:  crawling and indexing  

  Webmaster level: All
  
  Our mission is to organize the world’s information and make it universally accessible and useful. During this ambitious quest, we sometimes encounter non-HTML files such as PDFs, spreadsheets, and presentations. Our algorithms don’t let different filetypes slow them down; we work hard to extract the relevant content and to index it appropriately for our search results. But how do we actually index these filetypes, and—since they often differ so much from standard HTML—what guidelines apply to these files? What if a webmaster doesn’t want us to index them?
  
  
  
  Google first started indexing PDF files in 2001 and currently has hundreds of millions of PDF files indexed. We’ve collected the most often-asked questions about PDF indexing; here are the answers:
  
  Q: Can Google index any type of PDF file?
  A: Generally we can index textual content (written in any language) from PDF files that use various kinds of character encodings, provided they’re not password protected or encrypted. If the text is embedded as images, we may process the images with OCR algorithms to extract the text. The general rule of the thumb is that if you can copy and paste the text from a PDF document into a standard text document, we should be able to index that text.
  
  Q: What happens with the images in PDF files?
  A: Currently the images are not indexed. In order for us to index your images, you should create HTML pages for them. To increase the likelihood of us returning your images in our search results, please read the tips in our Help Center.
  
  Q: How are links treated in PDF documents?
  A: Generally links in PDF files are treated similarly to links in HTML: they can pass PageRank and other indexing signals, and we may follow them after we have crawled the PDF file. It’s currently not possible to "nofollow" links within a PDF document.
  
  Q: How can I prevent my PDF files from appearing in search results; or if they already do, how can I remove them?
  A: The simplest way to prevent PDF documents from appearing in search results is to add an X-Robots-Tag: noindex in the HTTP header used to serve the file. If they’re already indexed, they’ll drop out over time if you use the X-Robot-Tag with the noindex directive. For faster removals, you can use the URL removal tool in Google Webmaster Tools.
  
  Q: Can PDF files rank highly in the search results?
  A: Sure! They’ll generally rank similarly to other webpages. For example, at the time of this post, [mortgage market review], [irs form 2011] or [paracetamol expert report] all return PDF documents that manage to rank highly in our search results, thanks to their content and the way they’re embedded and linked from other webpages.
  
  Q: Is it considered duplicate content if I have a copy of my pages in both HTML and PDF?
  A: Whenever possible, we recommend serving a single copy of your content. If this isn’t possible, make sure you indicate your preferred version by, for example, including the preferred URL in your Sitemap or by specifying the canonical version in the HTML or in the HTTP headers of the PDF resource. For more tips, read our Help Center article about canonicalization.
  
  Q: How can I influence the title shown in search results for my PDF document?
  A: We use two main elements to determine the title shown: the title metadata within the file, and the anchor text of links pointing to the PDF file. To give our algorithms a strong signal about the proper title to use, we recommend updating both.
  
  If you want to learn more, watch Matt Cutt’s video about PDF files’ optimization for search, and visit our Help Center for information about the content types we’re able to index. If you have feedback or suggestions, please let us know in the Webmaster Help Forum.
  
  
  Posted by Gary Illyes, Webmaster Trends Analyst
• 

  Google News now crawling with Googlebot

  Posted: August 25th, 2011, 4:00pm CEST by John Mueller

  Tags:  crawling and indexing  

  Webmaster Level: intermediate
  
  (Cross-posted on the Google News Blog)
  
  Google News recently updated our infrastructure to crawl with Google’s primary user-agent, Googlebot. What does this mean? Very little to most publishers. Any news organizations that wish to opt out of Google News can continue to do so: Google News will still respect the robots.txt entry for Googlebot-News, our former user-agent, if it is more restrictive than the robots.txt entry for Googlebot.
  
  Our Help Center provides detailed guidance on using the robots exclusion protocol for Google News, and publishers can contact the Google News Support Team if they have any questions, but we wanted to first clarify the following:
  

  • Although you’ll now only see the Googlebot user-agent in your site’s logs, no need to worry: the appearance of Googlebot instead of Googlebot-News is independent of our inclusion policies. (You can always check whether your site is included in Google News by searching with the “site:” operator. For instance, enter “site:yournewssite.com” in the search field for Google News, and if you see results then we are currently indexing your news site.)
  • Your analytics tool will still be able to differentiate user traffic coming to your website from Google Search and traffic coming from Google News, so you should see no changes there. The main difference is that you will no longer see occasional automated visits to your site from the Googlebot-news crawler.
  • If you’re currently respecting our guidelines for Googlebot, you will not need to make any code changes to your site. Sites that have implemented subscriptions using a metered model or who have implemented First Click Free will not experience any changes. For sites which require registration, payment or login prior to reading any full article, Google News will only be able to crawl and index the title and snippet that you show all users who visit your page. Our Webmaster Guidelines provide additional information about “cloaking” (i.e., showing a bot a different version than what users experience). Learn more about Google News and subscription publishers in this Help Center article.
  • Rest assured, your Sitemap will still be crawled. This change does not affect how we crawl News Sitemaps. If you are a News publisher who hasn’t yet set up a News Sitemap and are interested in getting started, please follow this link.
  • For any publishers that wish to opt out of Google News and stay in Google Search, you can simply disallow Googlebot-news and allow Googlebot. For more information on how to do this, consult our Help Center.

  
  As with any website, from time to time we need to make updates to our infrastructure. At the same time, we want to continue to provide as much control as possible to news web sites. We hope we have answered any questions you might have about this update. If you have additional questions, please check out our Help Center.
  
  Posted by David Smydra, Google News Product Specialist
• 

  Making the most of improvements to the +1 button

  Posted: August 24th, 2011, 12:22pm CEST by Maile Ohye

  Tags:  products and services  

  Webmaster level: All
  
  For the past few months, you might have used +1 buttons to help visitors recommend your content on Google Search and on their Google Profiles. We’ve just announced a few changes that make +1 even more useful.
  
  First, the +1 button now lets visitors share links to your pages on Google+. If someone wants to start a conversation about your content, it’s easy for them to do so. Second, you can use +Snippets to customize the name, image and description that appear when your content is shared. Finally, new inline annotations help increase engagement after users see a friend’s recommendation right on your page.
  
  Here are a couple of tips to help you take full advantage of these improvements:
  
  +Snippets
  The +1 button opens up your site to a valuable new source of traffic with sharing on Google+. +Snippets let you put your best face forward by customizing exactly what appears when your content is shared.
  
  For example, if you’re running a movie review site, you might want visitors to share posts containing the title, movie poster, and a brief synopsis:
  
  
  
  You may already be using this markup to build rich annotations for your pages on Google Search. If not, marking up your pages is simple. Just add the correct schema.org attributes to the data already present on your pages. You’ll set a name, image, and description in your code:
  
  <body itemscope itemtype="http://schema.org/Article">
<h1 itemprop="name">This is the article name</h1>
<img itemprop="image" src="thumbnail.jpg" />
<p itemprop="description">This is the description of the article.</p>
</body>Example code containing each of the +Snippet attributes
  
  For more details on alternate markup types, please see our technical documentation.
  
  Inline annotations
  Now, when a person visits a page that someone they know has +1’d, they can see a name and face reminding them to pay special attention to your content. Here’s how it looks:
  
  
  Inline annotations let people see which of their friends +1’d your content
  
  To add inline annotations, you need to update your +1 button code. Visit the configuration tool, select ‘inline’ from the ‘Annotation’ menu, and grab a new snippet of code.
  
  Both sharing from +1 and inline annotations are rolling out fully over the next few days. To test these improvements right now, join our Platform Preview group.
  
  Update later the same day, August 24, 2011: If you have any thoughts or feedback you'd like to share, continue the conversation on Google+.
  
  Posted by Daniel Dulitz, Group Product Manager
• 

  Submit URLs to Google with Fetch as Googlebot

  Posted: August 3rd, 2011, 10:22am CEST by Jonathan Simon

  Tags:  crawling and indexing  

  Webmaster Level: All
  
  The Fetch as Googlebot feature in Webmaster Tools now provides a way to submit new and updated URLs to Google for indexing. After you fetch a URL as Googlebot, if the fetch is successful, you’ll now see the option to submit that URL to our index. When you submit a URL in this way Googlebot will crawl the URL, usually within a day. We’ll then consider it for inclusion in our index. Note that we don’t guarantee that every URL submitted in this way will be indexed; we’ll still use our regular processes—the same ones we use on URLs discovered in any other way—to evaluate whether a URL belongs in our index.
  
  This new functionality may help you in several situations: if you’ve just launched a new site, or added some key new pages, you can ask Googlebot to find and crawl them immediately rather than waiting for us to discover them naturally. You can also submit URLs that are already indexed in order to refresh them, say if you’ve updated some key content for the event you’re hosting this weekend and want to make sure we see it in time. It could also help if you’ve accidentally published information that you didn’t mean to, and want to update our cached version after you’ve removed the information from your site.
  
  How to submit a URL
  First, use Diagnostics > Fetch As Googlebot to fetch the URL you want to submit to Google. If the URL is successfully fetched you’ll see a new “Submit to index” link appear next to the fetched URL.
  Once you click “Submit to index” you’ll see a dialog box that allows you to choose whether you want to submit only the one URL, or that URL and all its linked pages.
  When submitting individual URLs, we have a maximum limit of 50 submissions per week; when submitting URLs with all linked pages, the limit is 10 submissions per month. You can see how many submissions you have left on the Fetch as Googlebot page. Any URL submitted should point to content that would be suitable for Google Web Search, so if you're trying to submit images or videos you should use Sitemaps instead.
  
  Submit URLs to Google without verifying
  In conjunction with this update to Fetch as Googlebot, we've also updated the public "Add your URL to Google" form. It's now the Crawl URL form. It has the same quota limits for submitting pages to the index as the Fetch as Googlebot feature but doesn't require verifying ownership of the site in question, so you can submit any URLs that you want crawled and indexed.
  
  Note that Googlebot is already pretty good about finding and crawling new content in a timely fashion, so don’t feel obligated to use this tool for every change or update on your site. But if you’ve got a URL whose crawling or indexing you want to speed up, consider submitting it using the Crawl URL form or the updated Fetch as Googlebot feature in Webmaster Tools. Feel free to comment here or visit our Webmaster Help Forum if you have more detailed questions.
  
  Written by Jonathan Simon & Susan Moskwa, Webmaster Trends Analysts
• 

  Improved handling of URLs with parameters

  Posted: July 22nd, 2011, 10:15am CEST by Jonathan Simon

  Tags:  crawling and indexing  

  Webmaster level: Advanced
  
  You may have noticed that the Parameter Handling feature disappeared from the Site configuration > Settings section of Webmaster Tools. Fear not; you can now find it under its new name, URL Parameters! Along with renaming it, we refreshed and improved the feature. We hope you’ll find it even more useful. Configuration of URL parameters made in the old version of the feature will be automatically visible in the new version. Before we reveal all the cool things you can do with URL parameters now, let us remind you (or introduce, if you are new to this feature) of the purpose of this feature and when it may come in handy.
  
  When to use
  URL Parameters helps you control which URLs on your site should be crawled by Googlebot, depending on the parameters that appear in these URLs. This functionality provides a simple way to prevent crawling duplicate content on your site. Now, your site can be crawled more effectively, reducing your bandwidth usage and likely allowing more unique content from your site to be indexed. If you suspect that Googlebot's crawl coverage of the content on your site could be improved, using this feature can be a good idea. But with great power comes great responsibility! You should only use this feature if you're sure about the behavior of URL parameters on your site. Otherwise you might mistakenly prevent some URLs from being crawled, making their content no longer accessible to Googlebot.
  
  A lot more to do
  Okay, let’s talk about what’s new and improved. To begin with, in addition to assigning a crawl action to an individual parameter, you can now also describe the behavior of the parameter. You start by telling us whether or not the parameter changes the content of the page. If the parameter doesn’t affect the page’s content then your work is done; Googlebot will choose URLs with a representative value of this parameter and will crawl the URLs with this value. Since the parameter doesn’t change the content, any value chosen is equally good. However, if the parameter does change the content of a page, you can now assign one of four possible ways for Google to crawl URLs with this parameter:

  • Let Googlebot decide
  • Every URL
  • Only crawl URLs with value=x
  • No URLs

  We also added the ability to provide your own specific value to be used, with the “Only URLs with value=x” option; you’re no longer restricted to the list of values that we provide. Optionally, you can also tell us exactly what the parameter does--whether it sorts, paginates, determines content, etc. One last improvement is that for every parameter, we’ll try to show you a sample of example URLs from your site that Googlebot crawled which contain that particular parameter.
  
  Of the four crawl options listed above, “No URLs” is new and deserves special attention. This option is the most restrictive and, for any given URL, takes precedence over settings of other parameters in that URL. This means that if the URL contains a parameter that is set to the “No URLs” option, this URL will never be crawled, even if other parameters in the URL are set to “Every URL.” You should be careful when using this option. The second most restrictive setting is “Only URLs with value=x.”
  
  Feature in use
  Now let’s do something fun and exercise our brains on an example.
  - - -Once upon a time there was an online store, fairyclothes.example.com. The store’s website used parameters in its URLs, and the same content could be reached through multiple URLs. One day the store owner noticed, that too many redundant URLs could be preventing Googlebot from crawling the site thoroughly. So he sent his assistant CuriousQuestionAsker to The GreatWebWizard to get advice on using the URL parameters feature to reduce the duplicate content crawled by Googlebot. The Great WebWizard was famous for his wisdom. He looked at the URL parameters and proposed the following configuration:
  
  

  Parameter name	Effect on content?	What should Googlebot crawl?
  trackingId	None	One representative URL
  sortOrder	Sorts	Only URLs with value = ‘lowToHigh’
  sortBy	Sorts	Only URLs with value = ‘price’
  filterByColor	Narrows	No URLs
  itemId	Specifies	Every URL
  page	Paginates	Every URL

  
  The CuriousQuestionAsker couldn’t avoid his nature and started asking questions:
  
  CuriousQuestionAsker: You’ve instructed Googlebot to choose a representative URL for trackingId (value to be chosen by Googlebot). Why not select the Only URLs with value=x option and choose the value myself?
  Great WebWizard: While crawling the web Googlebot encountered the following URLs that link to your site:

  1. fairyclothes.example.com/skirts/?trackingId=aaa123
  2. fairyclothes.example.com/skirts/?trackingId=aaa124
  3. fairyclothes.example.com/trousers/?trackingId=aaa125

  Imagine that you were to tell Googebot to only crawl URLs where “trackingId=aaa125”. In that case Googlebot would not crawl URLs 1 and 2 as neither of them has the value aaa125 for trackingId. Their content would neither be crawled nor indexed and none of your inventory of fine skirts would show up in Google’s search results. No, for this case choosing a representative URL is the way to go. Why? Because that tells Googlebot that when it encounters two URLs on the web that differ only in this parameter (as URLs 1 and 2 above do) then it only needs to crawl one of them (either will do) and it will still get all the content. In the example above two URLs will be crawled; either 1 & 3, or 2 & 3. Not a single skirt or trouser will be lost.
  
  CuriousQuestionAsker: What about the sortOrder parameter? I don’t care if the items are listed in ascending or descending order. Why not let Google select a representative value?
  Great WebWizard: As Googlebot continues to crawl it may find the following URLs:
  

  1. fairyclothes.example.com/skirts/?page=1&sortBy=price&sortOrder=’lowToHigh’
  2. fairyclothes.example.com/skirts/?page=1&sortBy=price&sortOrder=’highToLow’
  3. fairyclothes.example.com/skirts/?page=2&sortBy=price&sortOrder=’lowToHigh’
  4. fairyclothes.example.com/skirts/?page=2&sortBy=price&sortOrder=’ highToLow’

  Notice how the first pair of URLs (1 & 2) differs only in the value of the sortOrder parameter as do URLs in the second pair (3 & 4). However, URLs 1 and 2 will produce different content: the first showing the least expensive of your skirts while the second showing the priciest. That should be your first hint that using a single representative value is not a good choice for this situation. Moreover, if you let Googlebot choose a single representative from among a set of URLs that differ only in their sortOrder parameter it might choose a different value each time. In the example above, from the first pair of URLs, URL 1 might be chosen (sortOrder=’lowToHigh’). Whereas from the second pair URL 4 might be picked (sortOrder=’ highToLow’). If that were to happen Googlebot would crawl only the least expensive skirts (twice):

  • fairyclothes.example.com/skirts/?page=1&sortBy=price&sortOrder=’lowToHigh’
  • fairyclothes.example.com/skirts/?page=2&sortBy=price&sortOrder=’ highToLow’

  Your most expensive skirts would not be crawled at all! When dealing with sorting parameters consistency is key. Always sort the same way.
  
  CuriousQuestionAsker: How about the sortBy value?
  Great WebWizard: This is very similar to the sortOrder attribute. You want the crawled URLs of your listing to be sorted consistently throughout all the pages, otherwise some of the items may not be visible to Googlebot. However, you should be careful which value you choose. If you sell books as well as shoes in your store, it would be better not to select the value ‘title’ since URLs pointing to shoes never contain ‘sortBy=title’, so they will not be crawled. Likewise setting ‘sortBy=size’ works well for crawling shoes, but not for crawling books. Keep in mind that parameters configuration has influence throughout the whole site.
  
  CuriousQuestionAsker: Why not crawl URLs with parameter filterByColor?
  Great WebWizard: Imagine that you have a three-page list of skirts. Some of the skirts are blue, some of them are red and others are green.
  

  • fairyclothes.example.com/skirts/?page=1
  • fairyclothes.example.com/skirts/?page=2
  • fairyclothes.example.com/skirts/?page=3

  This list is filterable. When a user selects a color, she gets two pages of blue skirts:

  • fairyclothes.example.com/skirts/?page=1&flterByColor=blue
  • fairyclothes.example.com/skirts/?page=2&flterByColor=blue

  They seem like new pages (the set of items are different from all other pages), but there is actually no new content on them, since all the blue skirts were already included in the original three pages. There’s no need to crawl URLs that narrow the content by color, since the content served on those URLs was already crawled. There is one important thing to notice here: before you disallow some URLs from being crawled by selecting the “No URLs” option, make sure that Googlebot can access the content in another way. Considering our example, Googlebot needs to be able to find the first three links on your site, and there should be no settings that prevent crawling them.
  - - -
  If your site has URL parameters that are potentially creating duplicate content issues then you should check out the new URL Parameters feature in Webmaster Tools. Let us know what you think or if you have any questions post them to the Webmaster Help Forum.
  
  Written by Kamila Primke, Software Engineer, Webmaster Tools Team
• 

  Authorship markup and web search

  Posted: June 7th, 2011, 2:49pm CEST by Maile Ohye

  Tags:  crawling and indexing  

  Webmaster level: Intermediate
  
  Today we're beginning to support authorship markup—a way to connect authors with their content on the web. We're experimenting with using this data to help people find content from great authors in our search results.
  
  We now support markup that enables websites to publicly link within their site from content to author pages. For example, if an author at The New York Times has written dozens of articles, using this markup, the webmaster can connect these articles with a New York Times author page. An author page describes and identifies the author, and can include things like the author’s bio, photo, articles and other links.
  
  If you run a website with authored content, you’ll want to learn about authorship markup in our Help Center. The markup uses existing standards such as HTML5 (rel=”author”) and XFN (rel=”me”) to enable search engines and other web services to identify works by the same author across the web. If you're already doing structured data markup using microdata from schema.org, we'll interpret that authorship information as well.
  
  We wanted to make sure the markup was as easy to implement as possible. To that end, we’ve already worked with several sites to markup their pages, including The New York Times, The Washington Post, CNET, Entertainment Weekly, The New Yorker and others. In addition, we’ve taken the extra step to add this markup to everything hosted by YouTube and Blogger. In the future, both platforms will automatically include this markup when you publish content.
  
  We know that great content comes from great authors, and we’re looking closely at ways this markup could help us highlight authors and rank search results.
  
  Posted by Othar Hansson, Software Engineer
• 

  Easier URL removals for site owners

  Posted: May 17th, 2011, 11:00am CEST by Jonathan Simon

  Tags:  crawling and indexing  

  Webmaster Level: All
  
  We recently made a change to the Remove URL tool in Webmaster Tools to eliminate the requirement that the webpage's URL must first be blocked by a site owner before the page can be removed from Google's search results. Because you've already verified ownership of the site, we can eliminate this requirement to make it easier for you, as the site owner, to remove unwanted pages (e.g. pages accidentally made public) from Google's search results.
  
  Removals persist for at least 90 days
  When a page’s URL is requested for removal, the request is temporary and persists for at least 90 days. We may continue to crawl the page during the 90-day period but we will not display it in the search results. You can still revoke the removal request at any time during those 90 days. After the 90-day period, the page can reappear in our search results, assuming you haven’t made any other changes that could impact the page’s availability.
  
  Permanent removal
  In order to permanently remove a URL, you must ensure that one of the following page blocking methods is implemented for the URL of the page that you want removed:
  

  • indicate that the page no longer exists by returning a 404 or 410 HTTP status code
  • block the page from crawling via a robots.txt file
  • block the page from indexing via a noindex meta tag

  This will ensure that the page is permanently removed from Google's search results for as long as the page is blocked. If at any time in the future you remove the previously implemented page blocking method, we may potentially re-crawl and index the page. For immediate and permanent removal, you can request that a page be removed using the Remove URL tool and then permanently block the page’s URL before the 90-day expiration of the removal request.
  
  
  
  For more information about URL removals, see our “URL removal explained” blog series covering this topic. If you still have questions about this change or about URL removal requests in general, please post in our Webmaster Help Forum.
  
  Written by Jonathan Simon, Webmaster Trends Analyst
• 

  Do 404s hurt my site?

  Posted: May 2nd, 2011, 11:00am CEST by Susan Moskwa

  Tags:  crawling and indexing  

  Webmaster level: Beginner/Intermediate
  
  So there you are, minding your own business, using Webmaster Tools to check out how awesome your site is... but, wait! The Crawl errors page is full of 404 (Not found) errors! Is disaster imminent??
  
  
  Fear not, my young padawan. Let’s take a look at 404s and how they do (or do not) affect your site:
  
  Q: Do the 404 errors reported in Webmaster Tools affect my site’s ranking?
  A: 404s are a perfectly normal part of the web; the Internet is always changing, new content is born, old content dies, and when it dies it (ideally) returns a 404 HTTP response code. Search engines are aware of this; we have 404 errors on our own sites, as you can see above, and we find them all over the web. In fact, we actually prefer that, when you get rid of a page on your site, you make sure that it returns a proper 404 or 410 response code (rather than a “soft 404”). Keep in mind that in order for our crawler to see the HTTP response code of a URL, it has to be able to crawl that URL—if the URL is blocked by your robots.txt file we won’t be able to crawl it and see its response code. The fact that some URLs on your site no longer exist / return 404s does not affect how your site’s other URLs (the ones that return 200 (Successful)) perform in our search results.
  
  Q: So 404s don’t hurt my website at all?
  A: If some URLs on your site 404, this fact alone does not hurt you or count against you in Google’s search results. However, there may be other reasons that you’d want to address certain types of 404s. For example, if some of the pages that 404 are pages you actually care about, you should look into why we’re seeing 404s when we crawl them! If you see a misspelling of a legitimate URL (www.example.com/awsome instead of www.example.com/awesome), it’s likely that someone intended to link to you and simply made a typo. Instead of returning a 404, you could 301 redirect the misspelled URL to the correct URL and capture the intended traffic from that link. You can also make sure that, when users do land on a 404 page on your site, you help them find what they were looking for rather than just saying “404 Not found."
  
  Q: Tell me more about “soft 404s.”
  A: A soft 404 is when a web server returns a response code other than 404 (or 410) for a URL that doesn’t exist. A common example is when a site owner wants to return a pretty 404 page with helpful information for his users, and thinks that in order to serve content to users he has to return a 200 response code. Not so! You can return a 404 response code while serving whatever content you want. Another example is when a site redirects any unknown URLs to their homepage instead of returning 404s. Both of these cases can have negative effects on our understanding and indexing of your site, so we recommend making sure your server returns the proper response codes for nonexistent content. Keep in mind that just because a page says “404 Not Found,” doesn’t mean it’s actually returning a 404 HTTP response code—use the Fetch as Googlebot feature in Webmaster Tools to double-check. If you don’t know how to configure your server to return the right response codes, check out your web host’s help documentation.
  
  Q: How do I know whether a URL should 404, or 301, or 410?
  A: When you remove a page from your site, think about whether that content is moving somewhere else, or whether you no longer plan to have that type of content on your site. If you’re moving that content to a new URL, you should 301 redirect the old URL to the new URL—that way when users come to the old URL looking for that content, they’ll be automatically redirected to something relevant to what they were looking for. If you’re getting rid of that content entirely and don’t have anything on your site that would fill the same user need, then the old URL should return a 404 or 410. Currently Google treats 410s (Gone) the same as 404s (Not found), so it’s immaterial to us whether you return one or the other.
  
  Q: Most of my 404s are for bizarro URLs that never existed on my site. What’s up with that? Where did they come from?
  A: If Google finds a link somewhere on the web that points to a URL on your domain, it may try to crawl that link, whether any content actually exists there or not; and when it does, your server should return a 404 if there’s nothing there to find. These links could be caused by someone making a typo when linking to you, some type of misconfiguration (if the links are automatically generated, e.g. by a CMS), or by Google’s increased efforts to recognize and crawl links embedded in JavaScript or other embedded content; or they may be part of a quick check from our side to see how your server handles unknown URLs, to name just a few. If you see 404s reported in Webmaster Tools for URLs that don’t exist on your site, you can safely ignore them. We don’t know which URLs are important to you vs. which are supposed to 404, so we show you all the 404s we found on your site and let you decide which, if any, require your attention.
  
  Q: Someone has scraped my site and caused a bunch of 404s in the process. They’re all “real” URLs with other code tacked on, like http://www.example.com/images/kittens.jpg" width="100" height="300" alt="kittens"/></a... Will this hurt my site?
  A: Generally you don’t need to worry about “broken links” like this hurting your site. We understand that site owners have little to no control over people who scrape their site, or who link to them in strange ways. If you’re a whiz with the regex, you could consider redirecting these URLs as described here, but generally it’s not worth worrying about. Remember that you can also file a takedown request when you believe someone is stealing original content from your website.
  
  Q: Last week I fixed all the 404s that Webmaster Tools reported, but they’re still listed in my account. Does this mean I didn’t fix them correctly? How long will it take for them to disappear?
  A: Take a look at the ‘Detected’ column on the Crawl errors page—this is the most recent date on which we detected each error. If the date(s) in that column are from before the time you fixed the errors, that means we haven’t encountered these errors since that date. If the dates are more recent, it means we’re continuing to see these 404s when we crawl.
  
  After implementing a fix, you can check whether our crawler is seeing the new response code by using Fetch as Googlebot. Test a few URLs and, if they look good, these errors should soon start to disappear from your list of Crawl errors.
  
  Q: Can I use Google’s URL removal tool to make 404 errors disappear from my account faster?
  A: No; the URL removal tool removes URLs from Google’s search results, not from your Webmaster Tools account. It’s designed for urgent removal requests only, and using it isn’t necessary when a URL already returns a 404, as such a URL will drop out of our search results naturally over time. See the bottom half of this blog post for more details on what the URL removal tool can and can’t do for you.
  
  Still want to know more about 404s? Check out 404 week from our blog, or drop by our Webmaster Help Forum.
  
  Posted by Susan Moskwa, Webmaster Trends Analyst
• 

  Our SEO Guide — now available in ten more languages

  Posted: April 1st, 2011, 7:00pm CEST by Maile Ohye

  Tags:  crawling and indexing  

  Webmaster Level: Beginner
  
  We’re very glad to announce that our recently updated SEO guide is now available in ten more languages: Spanish, French, German, Russian, Turkish, Finnish, Swedish, Hungarian, Traditional Chinese and Simplified Chinese.
  
  For this new version, we’ve thoroughly reviewed and updated the content; we’ve also added a brand new section on best practices for mobile websites.
  
  
  
  You can download each PDF file in its full 32-page glory from goo.gl/seoguide and print it to have it as a useful resource.
  
  Posted by Esperanza Navas and Mariya Moeva, Search Quality Team
• 

  Page Speed for Chrome, and in 40 languages!

  Posted: March 22nd, 2011, 11:30am CET by Maile Ohye

  Tags:  products and services  

  Webmaster Level: All
  
  This article is cross-posted on the Google Code Blog.
  
  Today we’re launching the most requested feature for Page Speed, Page Speed for Chrome. Now Google Chrome users can get Page Speed performance suggestions to make their sites faster, right inside the Chrome browser. We would like to thank all our users for your great feedback and support since we launched. We’re humbled that 1.4 M unique users are using the Page Speed extension and finding it useful to help with their web performance diagnosis.
  
  Google Chrome support has always been high on our priority list but we wanted to get it right. It was critical that the same engine that powers the Page Speed Add-On for Firefox be used here as well. So we first built the Page Speed SDK, which we then integrated into the Chrome extension.
  
  Page Speed for Chrome retains the same core features as the Firefox add-on. In addition, there are two major improvements appearing in this version first. We’ve improved scoring and suggestion ordering to help web developers focus on higher-potential optimizations first. Plus, because making the web faster is a global initiative, Page Speed now supports displaying localized rule results in 40 languages! These improvements are part of the Page Speed SDK, so they will also appear in the next release of our Firefox add-on as well.
  
  If your site serves different content based on the browser’s user agent, you now have a good method for page performance analysis as seen by different browsers, with Page Speed coverage for Firefox and Chrome through the extensions, and Internet Explorer via webpagetest.org, which integrates the Page Speed SDK.
  
  We’d love to hear from you, as always. Please try Page Speed for Chrome, and give us feedback on our mailing list about additional functionality you’d like to see. Stay tuned for updates to Page Speed for Chrome that take advantage of exciting new technologies such as Native Client.
  
  By Matthew Steele and Richard Rabbat, Page Speed Team
• 

  Making Websites Mobile Friendly

  Posted: February 22nd, 2011, 3:11am CET by Pierre Far

  Tags:  crawling and indexing  

  Webmaster level: Intermediate

  We’ve noticed a rise in the number of questions from webmasters about how best to structure a website for mobile phones and how websites can best interact with Googlebot-Mobile. In this post we’ll explain the current situation and give you specific recommendations you can implement now.

  Some Background

  Let’s start with a simple question: what do we mean by “mobile phone” when talking about mobile-friendly websites?

  A good way to answer this question is to think about the capabilities of the mobile phone’s web browser, especially in relation to the capabilities of modern desktop browsers. To simplify matters, we can break mobile phones into a few classifications:

  1. Traditional mobile phones: Phones with browsers that cannot render normal desktop webpages. This includes browsers for cHTML (iMode), WML, WAP, and the like.
  2. Smartphones: Phones with browsers that render normal desktop pages, at least to some extent. This category includes a diversity of devices, such Windows Phone 7, Blackberry devices, iPhones, and Android phones, and also tablets and eBook readers.

     We can further break down this category by support for HTML5:

     • Devices with browsers that do not support HTML5
     • Devices with browsers that support HTML5

  Once upon a time, mobile phones connected to the Internet using browsers with limited rendering capabilities; but this is clearly a changing situation with the fast rise of smartphones which have browsers that rival the full desktop experience. As such, it’s important to note that the distinction we are making here is based on the current situation as we see it and might change in the future.

  Googlebot and Mobile Content

  Google has two crawlers relevant to this topic: Googlebot and Googlebot-Mobile. Googlebot crawls desktop-browser type of webpages and content embedded in them and Googlebot-Mobile crawls mobile content. The questions we’re seeing more of can be summed up as follows:

  Given the diversity of capabilities of mobile web browsers, what kind of content should I serve to Googlebot-Mobile?

  The answer lies in the User-agent that Googlebot-Mobile supplies when crawling. There are several User-agent strings in use by Googlebot-Mobile, all of which use this format:

  [Phone name(s)] (compatible; Googlebot-Mobile/2.1; +http://www.google.com/bot.html)

  To decide which content to serve, assess which content your website has that best serves the phone(s) in the User-agent string. A full list of Googlebot-Mobile User-agents can be found here.

  Notice that we currently do not crawl with Googlebot-Mobile using a smartphone User-agent string. Thus at the current time, a correctly-configured content serving system will serve Googlebot-Mobile content only for the traditional phones described above, because that’s what the User-agent strings in use today dictate. This may change in the future, and if so, it may mean there would be a new Googlebot-Mobile User-agent string.

  For now, we expect smartphones to handle desktop experience content so there is no real need for mobile-specific effort from webmasters. However, for many websites it may still make sense for the content to be formatted differently for smartphones, and the decision to do so should be based on how you can best serve your users.

  URL Structure for Mobile Content

  The next set of questions ask about the URLs mobile content should be served from. Let’s look in detail at some common use cases.

  Websites with only Desktop Experience Content

  Most websites currently have only one version of their content, namely in HTML that is designed for desktop web browsers. This means all browsers access the content from the same URL.

  These websites may not be serving traditional mobile phone users. The quality experienced by their smartphone users depends on the mobile browser they are using and it could be as good as browsing from the desktop.

  If you serve only desktop experience content for all User Agents, you should do so for Googlebot-Mobile too; that is, treat Googlebot-Mobile as you treat all other or unknown User Agents. In these cases, Google may modify your webpages for an improved mobile experience.

  Websites with Dedicated Mobile Content

  Many websites have content specifically optimized for mobile users. The content could be simply reformatted for the typically smaller mobile displays, or it could be in a different format (e.g., served using WAP, etc.).

  A very common question we see is: Does it matter if the different types of content are served from the same URL or from different URLs? For example, some websites have www.example.com as the URL desktop browsers are meant to access and have m.example.com or wap.example.com for the different mobile devices. Other websites serve all types of content from just one URL structure like www.example.com.

  For Googlebot and Googlebot-Mobile, it does not matter what the URL structure is as long as it returns exactly what a user sees too. For example, if you redirect mobile users from www.example.com to m.example.com, that will be recognized by Googlebot-Mobile and both websites will be crawled and added to the correct index. In this case, use a 301 redirect for both users and Googlebot-Mobile.

  If you serve all types of content from www.example.com, i.e. serving desktop-optimized content or mobile-optimized content from the same URL depending on the User-agent, this will also lead to correct crawling by Googlebot and Googlebot-Mobile. This is not considered cloaking by Google.

  It is worth repeating that regardless of URL structure, you must correctly detect the User-agent as given by your users and Googlebot-Mobile, and serve both the same content. Don’t forget to keep the default content, the desktop-optimized content, for when an unknown User-agent requests it.

  Mobile Sitemaps in Webmaster Tools

  Finally, we receive many questions about what URLs to put in Mobile Sitemaps. As explained in our Mobile Sitemaps Help Center articles, you should include only mobile content URLs in Mobile Sitemaps, even if these URLs also return non-mobile content when accessed by a non-mobile User-agent.

  More Questions?

  A good place to start is our Mobile Sites Help Center articles and the relevant sections in our Search Engine Optimization Starter Guide. We also created a thread in our forums for you to ask questions about this post.

  Posted by Pierre Far, Webmaster Trends Analyst

• 

  How to deal with planned site downtime

  Posted: January 25th, 2011, 2:00am CET by Michael Wyszomierski

  Tags:  crawling and indexing  

  Webmaster level: Intermediate to Advanced
  Once in a while we get asked whether a site’s visibility in Google’s search results can be impacted in a negative way if it’s unavailable when Googlebot tries to crawl it. Sometimes downtime is unavoidable: a webmaster might decide to take a site down due to ongoing site maintenance, or legal or cultural requirements. Outages that are not clearly marked as such can negatively affect a site’s reputation. While we cannot guarantee any crawling, indexing or ranking, there are methods to deal with planned website downtime in a way that will generally not negatively affect your site’s visibility in the search results.
  For example, instead of returning an HTTP result code 404 (Not Found) or showing an error page with the status code 200 (OK) when a page is requested, it’s better to return a 503 HTTP result code (Service Unavailable) which tells search engine crawlers that the downtime is temporary. Moreover, it allows webmasters to provide visitors and bots with an estimated time when the site will be up and running again. If known, the length of the downtime in seconds or the estimated date and time when the downtime will be complete can be specified in an optional Retry-After header, which Googlebot may use to determine when to recrawl the URL.
  Returning a 503 HTTP result code can be a great solution for a number of other situations. We encounter a lot of problems with sites that return 200 (OK) result codes for server errors, downtime, bandwidth-overruns or for temporary placeholder pages (“Under Construction”). The 503 HTTP result code is the webmaster’s solution of choice for all these situations. As for planned server downtime like hardware maintenance, it’s a good idea to have a separateserver available to actually return the 503 HTTP result code. It is important, however, to not treat 503 as a permanent solution: lasting 503s can eventually be seen as a sign that the server is now permanently unavailable and can result in us removing URLs from Google’s index.
  header('HTTP/1.1 503 Service Temporarily Unavailable');header('Retry-After: Sat, 8 Oct 2011 18:27:00 GMT');
  

  If you set up a 503 (Service Unavailable) response, the header information might look like this when using PHP.

  Similar to how you can make 404 pages more useful to users, it’s also a good idea to provide a customized 503 message explaining the situation to users and letting them know when the site will be available again. For further information regarding HTTP result codes, please see RFC 2616.
  Posted by Tomer Honen and Kaspar Szymanski, Search Quality Team, Dublin
• 

  "It's on Google! YAY!" - Getting webmaster help in our forum

  Posted: December 28th, 2010, 6:27am CET by John Mueller

  Tags:  feedback and communication  

  Webmaster level: all
  
  It's been a bit more than five years now that our Webmaster Help Forum has been up and running, helping webmasters around the world. Over the years, over tens of thousands of users have discussed various topics in well over 100,000 threads, helping each other to improve their websites and to solve a variety of issues that web publishers are confronted with. Among those users is a group that we call the "Bionic Posters," users who have proven to be consistently helpful and knowledgeable, selflessly helping others to tackle seemingly insurmountable problems.
  
  It's great to have such an awesome community -- but thanks is best said by those who are helped. Here is some of the feedback that we collected this year:
  

  • "Thank you for the time you have spent helping me. It is genuinely appreciated."
  • "Thank you to everyone who helped with my problem! My creaky old 1996 era website is all cleaned up and doing just fine now! Good Guys Rule!"
  • "WOW!!!!!!!! Thank you so much for your help! I was reluctant to post because I thought you guys might think my site is too small, too insignificant, etc. Thanks so much! To me, it's a BIG deal!"
  • "My traffic has doubled and I am now either top or close to the top in search terms"
  • "Thanks. Hopefully my late night paid off then, all the help and information has been great!!!!!!!"
  • "Wow Webado Thank You Again.. You Really Know Your stuff! (…) You Are a True professional and Seriously The Only Person That Could Even Figure This out. I even Spoke to Other Top Specialists and YOU were the Only one who told me what to do and what was wrong."
  • "You are AMAZING! Thank you so much!"
  • "Finally, thanks so much for your concern and prompt reply, Squibble. Can't imagine a person would deligated his efforts to someones you don't even know and met. (…) Thanks for your help!! It means everything to me!"
  • "Wow thanks so much! I have just been to the hot springs in Banos Ecuador, the volcano is rumbling and the town has evacuated but I am still here talking Apache, now that is dedication, I will test your helpfulness out 2moro! Thanks so much!!!"
  • "THANKYOU THANKYOU THANKYOU! YOU ARE SOOOO AMAZING! MY BLOG IS GONE FROM GOOGLE AND ALL THE OLD POSTS ASSOCIATED WITH THAT TOO! THANK YOU! I LOVE YOU!"
  • "It worked! Thank you very much for your help Cristina :)"
  • "I could NEVER have seen what's possible without this forum. I am so grateful."
  • "Thank you so much for your detailed response to my questions. In 10 years of me having a website, no one has explained these concepts better to me than you did."
  • "IT WORKED!!! Thank you so much for saving me the grief and embarrassment of this problem. I truly appreciate both your knowledge and guidance."
  • "Thank you so much for such a detailed answer and putting into terms I can easily understand. (…) Where shall I send the batch of brownies?"
  • "Thank you so much Squibble you are a hero. I have done what it says and i will check to see if i appear in google! Thanks again!"
  • "you guys are amazing! thank you so much redleg! (…) and if you happen to ever be in san carlos give me a shout - you deserve at least a beer and a lunch! "
  • "Thank you Squibble, Vanessa, Cristina, and Ishigaki for weighing in on this and helping me. I hope I can pay it forward one day."
  • "Great info Guys. I really appreciate it. It was awesome of y'all to help me out. I really appreciate it. Thank you."
  • "Your amazing :-) i love you lol xx im sorted now thanks and never in a million years would i have found that out!"
  • "THANK YOU! I love Google and appreciate that they have these safety precautions in place for those nasty hackets - especially when we can fix the problem! Thanks so much for your help. Whew."
  • "Thank you very much, Robbo! With a little tweaking it worked perfectly!"
  • "yay!!! I think it finally went through - THANK YOU SO MUCH!!!"

  
  If you have a question that you would like to ask, a problem that you need help with, we'd love to see you in the forums! We just ask that you please take the time to read through our frequently asked questions, and search the forum before posting. Chances are high that a question like yours has already been answered. Tell us a little bit about yourself and then join us to learn more and help others!
  
  Posted by John Mueller, Webmaster Trends Analyst
• 

  Improving our help content: stocking stuffers in our Help Center

  Posted: December 23rd, 2010, 10:21am CET by Michael Wyszomierski

  Tags:  feedback and communication  

  Webmaster Level: All
  We provide lots of information for webmasters across many different channels — you can stay up to date with the latest features here on our blog, browse articles in our Help Center, have discussions in our forums (in 17 languages!), watch videos on our YouTube channel, or even read in-depth interviews (in English, Portuguese, and other languages).
  There’s no shortage of useful information, but sometimes the relevant bits may be a bit difficult to locate, especially for novice webmasters. We see the same questions popping up over and over again, so we’ve tried to make our most frequently searched information as accessible and visible as possible: We analysed the questions asked over the past year and a half and identified the issues you are most interested in. We then picked out the relevant bits from across our different resources and collected the answers to those questions in one new convenient FAQ page in our Help Center (available in 20 languages).
  We also frequently get questions on how to get in touch with us, so we’ve put together all the different ways you can: ...tell us about a page you want to remove from our search results; ...tell us about spam you found; ...let us know when you’ve fixed issues on your website;...and many more! All of these contact channels are now listed conveniently in one article with direct links to the relevant forms: Webmaster help and contacts (available from the homepage of our Help Center, also in 20 languages).
  Now isn’t that a nice stocking stuffer (-:? Happy webmastering in 2011, and keep the feedback coming!
  
  Posted by Mariya Moeva, Search Quality Team, Dublin
• 

  What a feeling! Even better indexing of SWF content

  Posted: November 11th, 2010, 6:20pm CET by Maile Ohye

  Tags:  crawling and indexing  

  Webmaster Level: All
  
  We often get questions from webmasters about how we index content designed for Flash Player, so we wanted to take a moment to update you on some of our latest progress.
  
  About two years ago we announced that through a collaboration with Adobe we had significantly improved Google’s capability to index Flash technology based content. Last year we followed up with an announcement that we had added external resource loading to our SWF indexing capabilities. This work has allowed us to index all kinds of textual content in SWF files, from Flash buttons and menus to self-contained Flash technology based websites. Currently almost any text a user can see as they interact with a SWF file on your site can be indexed by Googlebot and used to generate a snippet or match query terms in Google searches. Additionally, Googlebot can also discover URLs in SWF files and follow those links, so if your SWF content contains links to pages inside your website, Google may be able to crawl and index those pages as well.
  
  Last month we expanded our SWF indexing capabilities thanks to our continued collaboration with Adobe and a new library that is more robust and compatible with features supported by Flash Player 10.1. Additionally, thanks to improvements in the way we handle JavaScript, we are also now significantly better at recognizing and indexing sites that use JavaScript to embed SWF content. Finally, we have made improvements in our video indexing technology, resulting in better detection of when a page has a video and better extraction of metadata such as alternate thumbnails from Flash technology based videos. All in all, our SWF indexing technology now allows us to see content from SWF files on hundreds of millions of pages across the web.
  
  While we’ve made great progress indexing SWF content over the past few years, we’re not done yet. We are continuing to work on our ability to index deep linking (content within a Flash technology based application that is linked to from the same application) as well as further improving indexing of SWF files executed through JavaScript. You can help us improve these capabilities by creating unique links for each page that is linked from within a single Flash object and by submitting a Sitemap through Google Webmaster Tools.
  
  We’re excited about the progress we’ve made so far and we look forward to keeping you updated about further progress.
  
  Written by Jifeng Situ and Sverre Sundsdal, Software Engineers
• 

  A Chrome extension for reporting webspam

  Posted: November 8th, 2010, 4:08pm CET by Maile Ohye

  Tags:  feedback and communication  

  Webmaster Level: All
  
  At Google, we continually strive to improve our algorithms to keep search results relevant and clean. You have been supporting us on this mission by sending spam reports for websites that violate our Webmaster Guidelines, using the spam report form in Google Webmaster Tools. While you might not see changes right away, we take your reports seriously and use them to fine-tune our algorithms -- the feedback is much appreciated and helps us to protect the integrity of our search results. We also take manual action on many of these spam reports. A recent blog post covers more information on how to identify webspam.
  
  For those of you who regularly report spam, or would like to do so, we’ve now published a Chrome extension for reporting spam that makes the process more convenient and simple. The extension adds “Report spam” links to search results and your Web History, taking you directly to the spam report form and autocompleting some form fields for you. With this extension, Google’s spam report form is always just one click away.
  
  The Google Webspam Report Chrome extension provides further tools to help you quickly fill out a spam report:

  • a browser button to report the currently viewed page
  • an option to retrieve recent Google searches from your Chrome history
  • an option to retrieve recently visited URLs from your Chrome history

  As before, you need to be logged into your Google Account to report spam. You can find a more detailed walkthrough of the use cases and features in this presentation and on the Chrome Extensions Gallery page, where you can also provide feedback and suggestions. We hope that you find this extension useful and that you continue to help us fight spam.
  
  The extension is available in 16 languages. If your Chrome browser is set to a language supported by the extension, it will automatically use the localized version, otherwise defaulting to English.
  
  Note: We care about your privacy. The Google Webspam Report Chrome extension allows you to access your personal Chrome history for the purpose of reporting spam, but does not send data retrieved from it to our servers. The source code of the extension has been published under an open source license.
  
  Posted by Manuel Holtz, Support Engineer, Search Quality team
• 

  How to help Google identify web spam

  Posted: November 4th, 2010, 3:00pm CET by Maile Ohye

  Tags:  feedback and communication  

  Webmaster level: All
  
  Everyone who uses the web knows how frustrating it is to land on a page that sounds promising in the search results but ends up being useless when you visit it. We work hard to make sure Google’s algorithms catch as much as possible, but sometimes spammy sites still make it into search results. We appreciate the numerous spam reports sent in by users like you who find these issues; the reports help us improve our search results and make sure that great content is treated accordingly. Good spam reports are important to us. Here’s how to maximize the impact of any spam reports you submit:
  
  Why report spam to Google?
  
  Google’s search quality team uses spam reports as a basis for further improving the quality of the results that we show you, to provide a level playing field for webmasters, and to help with our scalable spam fighting efforts. With the release of new tools like our Chrome extension to report spam, we’ve seen people filing more spam reports and we have to allocate appropriate resources to the spam reports that are mostly likely to be useful.
  
  Spam reports are prioritized by looking at how much visibility a potentially spammy site has in our search results, in order to help us focus on high-impact sites in a timely manner. For instance, we’re likely to prioritize the investigation of a site that regularly ranks on the first or second page over that of a site that only gets a few search impressions per month. A spam report for a page that is almost never seen by users is less likely to be reviewed compared to higher-impact pages or sites. We generally use spam reports to help improve our algorithms so that we can not only recognize and handle this particular site, but also cover any similar sites. In a few cases, we may additionally choose to immediately remove or otherwise take action on a site.
  
  Which sites should I report?
  
  We love seeing reports about spammy sites that our algorithms have missed. That said, it’s a poor use of your time to report sites that are not spammy. Sites submitted through the spam report form are reviewed for spam content only. Sites that you think should be tackled for other reasons should be submitted to us through the appropriate channels: for example, for those that contain content which you have removed, use our URL removal tools; for sites with malware, use the malware report form; for paid links that you find on sites, use the paid links reporting form. If you want to report spammy links for a page, make sure that you read how to report linkspam. If you have a complaint because someone is copying your content, we have a different copyright process--see our official documentation pages for more info. There’s generally no need to report sites with technical problems or parked domains because these are typically handled automatically.
  
  The same applies to redirecting legitimate sites from one top level domain to another, e.g. example.de redirecting to example.com/de. As long as the content presented is not spammy, the technique of redirecting one domain to another does not automatically violate the Google Webmaster Guidelines.
  
  
  If you happen to come across a gibberish site similar to this one, it’s most likely spam.
  The best way to submit a compelling spam report is to take a good look at the website in question and compare it against the Google Webmaster Guidelines. For instance, these would be good reasons to report a site through the spam report form:

  • the cached version contains significantly different (often keyword-rich) content from the live version
  • you’re redirected to a completely different domain with off-topic, commercial content
  • the site is filled with auto-generated or keyword-stuffed content that seems to make no sense

  These are just a few examples of techniques that might be potentially spammy, and which we would appreciate seeing in the form of a spam report. When in doubt, please feel free to discuss your concerns on the Help Forum with other users and Google guides.
  
  What should I include in a spam report?
  
  Some spam reports are easier to understand than others; having a clear and easy-to-understand report makes it much easier for us to analyze the issue and take appropriate actions. Here are some things to keep in mind when submitting the spam report:

  • Submit the URLs of the pages where you see spam (not just the domain name). This makes it easy for us to verify the problem on those specific pages.
  • Try to specify the issue as clearly as possible using the checkboxes. Don’t just check every single box--such reports are less likely to be reviewed.
  • If only a part of the page uses spammy techniques, for example if it uses cloaking or has hidden text on an otherwise good page, provide a short explanation on how to look for the spam you’re seeing. If you’re reporting a site for spammy backlinks rather than on-page content, mention that.

  By following these guidelines, your spam reports will be reproducible and clear, making them easier to analyze on our side.
  
  What happens next?
  
  After reviewing the feedback from these reports (we want to confirm that the reported sites are actually spammy, not just sites that someone didn’t like), it may take a bit of time before we update our algorithms and a change is visible in the search results. Keep in mind that sometimes our algorithms may already be treating those techniques appropriately; for instance, perhaps we’re already ignoring all the hidden text or the exchanged links that you have reported. Submitting the same spam report multiple times is not necessary. Rest assured that we actively review spam reports and take appropriate actions, even if the changes are not immediately visible to you.
  
  With your help, we hope that we can improve the quality of and fairness in our search results for everyone! Thank you for continuing to submit spam reports and feel free to post here or in our Help Forum should you have any questions.
  
  Written by Kaspar Szymanski, Search Quality Strategist & John Mueller, Webmaster Trends Analyst
• 

  Unifying content under multilingual templates

  Posted: September 9th, 2010, 2:56am CEST by Maile Ohye

  Tags:  crawling and indexing  

  Webmaster Level: Advanced
  
  If you have a global site containing pages where the:

  • template (i.e. side navigation, footer) is machine-translated into various languages,
  • main content remains unchanged, creating largely duplicate pages,

  and sometimes search results direct users to the wrong language, we’d like to help you better target your international/multilingual audience through:
  
  <link rel=”alternate” hreflang="a-different-language" href="http://url-of-the-different-language-page" />
  
  As you know, when rel=”canonical” or a 301 response code is properly implemented, we become more precise in clustering information from duplicate URLs, such as consolidating their linking properties. Now, when rel=”alternate” hreflang=”x” is included in conjunction with rel=”canonical” or 301s, not only will our indexing and linking properties be more accurate, but we can better serve users the URL of their preferred language.
  
  Sample configuration that’s prime for rel=”alternate” hreflang=”x”
  
  How does this all work? Imagine that you’re the proud owner of example.com, a site called “The Network” where you allow users to create their very own profile. Let’s say Javier Lopez, a Spanish speaker, makes his page at http://es.example.com/javier-lopez:
  
  
  Because you’re trying to target a multilingual audience, once Javier hits “Publish,” his profile becomes immediately available in other languages with the translated templates. Also, each of the new language versions is served on a separate URL.
  
  
  Two localized versions, http://en.example.com/javier-lopez in English and http://fr.example.com/javier-lopez in French
  Background on the old issue: duplicate content caused by language variations
  
  The configuration above allowed visitors speaking different languages to more easily interpret the content, but for search engines it was slightly problematic: there are three URLs (English, French, and Spanish versions) for the same main content in Javier’s profile. Webmasters wanted to avoid duplicate content issues (such as PageRank dilution) from these multiple versions and still ensure that we would serve the appropriate version to the user.
  
  A new solution for localized templates
  
  First of all, just to be clear, the strategy we’re proposing isn’t appropriate for multilingual sites that completely translate each page’s content. We’re trying to specifically improve the situation where the template is localized but the main content of a page remains duplicate/identical across language/country variants.
  
  Before we get into the specific steps, our prior advice remains applicable:

  • Have one URL associated with one piece of content. We recommend against using the same URL for multiple languages, such as serving both French and English versions on example.com/page.html based on user information (IP address, Accept-Language HTTP header).
  
  
  • When multiple languages are at play, it’s best to include the language or country indication in the URL, e.g., example.com/en/welcome.html and example.com/fr/accueil.html (which specify “en” and “fr”) rather than example.com/welcome.html and example.com/accueil.html (which don’t contain an explicit country/language specification). More suggestions can be found in our blog posts about designing localized URLs and multilingual sites.

  For the new feature:

  Step 1: Select the proper canonical.
  The canonical designates the version of your content you’d like indexed and returned to users.

  The first step towards making the right content indexable is to pick one canonical URL that best reflects the genuine locale of the page’s main content. In the example above, since Javier is a Spanish-speaking user and he created his profile on es.example.com, http://es.example.com/javier-lopez is the logical canonical. The title and snippet in all locales will be selected from the canonical URL.
  
  Once you have the canonical URL picked out, you can either:

  A. 301 (permanent redirect) from the language variants to the canonical
  
  As an example, if a French speaker visits fr.example.com/javier-lopez (not the canonical), have this page include a cookie to remember the user's language preference of French. Then permanently redirect from fr.example.com/javier-lopez to the canonical at es.example.com/javier-lopez. Because of the cookie, es.example.com/javier-lopez will still render its boilerplate in French (even on the es.example.com subdomain!). Similarly, en.example.com/javier-lopez would set the value of this cookie to English and then 301 redirect to es.example.com/javier-lopez.
  
  Including a language selection link is also helpful should a multilingual user prefer a different experience of your site.
  
  B. Use rel=”canonical”
  
  On the other language variants, include a link rel=”canonical” tag pointing to your chosen canonical. In our example, since the canonical for Javier’s profile is the Spanish version, the English and French pages (and optionally even the Spanish page itself) would include <link rel=”canonical” href="http://es.example.com/javier-lopez" />.
  
  Cookies are not involved in this setup. Therefore, a French speaker will be served es.example.com/javier-lopez with a Spanish template. Implement step 2 if you want the French speakers to be served the French version at fr.example.com/javier-lopez in Google search results.

  Step 2: In the canonical URL, specify the various language versions via the rel=”alternate” link tag, using its hreflang attribute.
  
  rel=”alternate” URLs can be displayed in search results in accordance with a user’s language preference. The title and snippet, however, remain generated from the canonical URL (as is customary with rel=”canonical”), not from the content of any rel=”alternate”.

  You can help Google display the correctly localized variant of your URL to our international users by adding the following tags to http://es.example.com/javier-lopez, the selected canonical:
  
  <link rel=”alternate” hreflang="en" href="http://en.example.com/javier-lopez" />

<link rel=”alternate” hreflang="fr" href="http://fr.example.com/javier-lopez" />

rel=”alternate” indicates that the URL contains an alternate version located at the URI of the href value. hreflang identifies the language code of the alternate URL and can be specified with ISO-639.
  
  Please note: If your site supports many languages and you’re worried about the increased file size when declaring numerous rel=”alternate” URLs, please see our Help Center article about configuring rel=”alternate” with file size constraints.

  Once the steps are completed, the configuration on “The Network” would look like this:

  • http://en.example.com/javier-lopez

    either 301s with a language cookie or contains <link rel=”canonical” href=”http://es.example.com/javier-lopez” />

  • http://fr.example.com/javier-lopez

    either 301s with a language cookie or contains <link rel=”canonical” href=”http://es.example.com/javier-lopez” />

  • http://es.example.com/javier-lopez

    is the canonical and contains
    <link rel=”alternate” hreflang="en" href="http://en.example.com/javier-lopez" />
    and
    <link rel=”alternate” hreflang="fr" href="http://fr.example.com/javier-lopez" />

  
  Results of the above implementation
  

  • When your content is returned in search results, users will likely see the URL that corresponds to their language preference, whether or not it’s the canonical. (Good news!) This is because with with rel=”canonical” or a 301 redirect, we can cluster the language variations with the canonical. With rel=”alternate” hreflang=”x” at serve-time we can deliver the URL of the most appropriate language to the user: English speakers will be served en.example.com/javier-lopez as the result for the URL in Javier’s profile, French speakers will see fr.example.com/javier-lopez, Spanish speakers will see es.example.com/javier-lopez.
  
  
  • By implementing step 1, only content from the canonical version will be available for users in search results (i.e. content from the duplicate versions won’t be searchable). Because the Spanish version es.example.com/javier-lopez is the canonical, queries that include template content from this page, e.g. [Javier Lopez familia] -- when using any language preference -- may return his profile (content from the canonical version). On the other hand, queries that include template content of the “duplicate” version, e.g. [Javier Lopez family], are less likely to return his profile page. If you would like the other language versions indexed separately and searchable, avoid using rel=”canonical” and rel=”alternate”.
  
  
  • Indexing properties, such as linking information, from the duplicate language variants will be consolidated with the canonical.

  
  To recap (one more time, with feeling!)
  
  For sites that have their template localized but the keep their pages’ main content untranslated:
  
  Step 1: Once you have the canonical picked out you can use either rel=”canonical” or a 301 (permanent redirect) from the various localized pages to the canonical URL.
  
  Step 2: On the canonical URL, specify the language-specific duplicated content with different boilerplate via the rel=”alternate” link tag, using its hreflang attribute. This way, Google can show the correctly-localized variant of your URLs to our international users.
  
  We realize this can be a little complicated, so if you have questions, please ask in our webmaster forum!
  
  Written by Surabhi Gupta, Joachim Kupke & Jayesh Vyas, Search Localization and Indexing Teams
• 

  Google now indexes SVG

  Posted: September 1st, 2010, 1:30am CEST by Maile Ohye

  Tags:  crawling and indexing  

  Webmaster Level: All
  
  You can now use Google search to find SVG documents. SVG is an open, XML-based format for vector graphics with support for interactive elements. We’re big fans of open standards, and our mission is to organize the world’s information, so indexing SVG is a natural step.
  
  We index SVG content whether it is in a standalone file or embedded directly in HTML. The web is big, so it may take some time before we crawl and index most SVG files, but as of today you may start seeing them in your search results. If you want to see it yourself, try searching for [sitemap site:fastsvg.com] or [HideShow site:svg-whiz.com]
  
  If you host SVG files and you wish to exclude them from Google’s search results, you can use the “X-Robots-Tag: noindex” directive in the HTTP header.
  
  Check out Webmaster Central for a full list of file types we support.
  
  Posted by Bogdan Stanescu and John Sarapata, Software Engineers
• 

  Our new search index: Caffeine

  Posted: June 8th, 2010, 7:00pm CEST by Michael Wyszomierski

  Tags:  crawling and indexing  

  (Cross-posted on the Official Google Blog)
  Today, we're announcing the completion of a new web indexing system called Caffeine. Caffeine provides 50 percent fresher results for web searches than our last index, and it's the largest collection of web content we've offered. Whether it's a news story, a blog or a forum post, you can now find links to relevant content much sooner after it is published than was possible ever before.
  Some background for those of you who don't build search engines for a living like us: when you search Google, you're not searching the live web. Instead you're searching Google's index of the web which, like the list in the back of a book, helps you pinpoint exactly the information you need. (Here's a good explanation of how it all works.)
  So why did we build a new search indexing system? Content on the web is blossoming. It's growing not just in size and numbers but with the advent of video, images, news and real-time updates, the average webpage is richer and more complex. In addition, people's expectations for search are higher than they used to be. Searchers want to find the latest relevant content and publishers expect to be found the instant they publish.
  To keep up with the evolution of the web and to meet rising user expectations, we've built Caffeine. The image below illustrates how our old indexing system worked compared to Caffeine:Our old index had several layers, some of which were refreshed at a faster rate than others; the main layer would update every couple of weeks. To refresh a layer of the old index, we would analyze the entire web, which meant there was a significant delay between when we found a page and made it available to you.
  With Caffeine, we analyze the web in small portions and update our search index on a continuous basis, globally. As we find new pages, or new information on existing pages, we can add these straight to the index. That means you can find fresher information than ever before — no matter when or where it was published.
  Caffeine lets us index web pages on an enormous scale. In fact, every second Caffeine processes hundreds of thousands of pages in parallel. If this were a pile of paper it would grow three miles taller every second. Caffeine takes up nearly 100 million gigabytes of storage in one database and adds new information at a rate of hundreds of thousands of gigabytes per day. You would need 625,000 of the largest iPods to store that much information; if these were stacked end-to-end they would go for more than 40 miles.
  We've built Caffeine with the future in mind. Not only is it fresher, it's a robust foundation that makes it possible for us to build an even faster and comprehensive search engine that scales with the growth of information online, and delivers even more relevant search results to you. So stay tuned, and look for more improvements in the months to come.
  Posted by Carrie Grimes, Software Engineer
• 

  Grab bag videos are back!

  Posted: June 1st, 2010, 2:36pm CEST by Michael Wyszomierski

  Tags:  feedback and communication  

  We’re kicking off June with the start of a new round of webmaster Q&A on the Webmaster Central YouTube channel. You submitted and voted on questions for Matt Cutts to answer, and Matt sat in the studio for a full day sharing advice for webmasters.
  For those of you who watch each video (and who doesn’t?), we’ve worked hard to keep things interesting. Not only did Matt wear different colored shirts, we changed the backgrounds as well! Just don’t submit any screen grabs to We Have Lasers, okay?
  To get you started, here’s the first video, which addresses a question about geographic targeting in Webmaster Tools:
  We’ll be posting links to new videos as they’re posted on our Twitter account, so follow us there or subscribe to our YouTube channel to be notified of new answers.
  Posted by Michael Wyszomierski, Search Quality Team
• 

  Call for webspam reports in Thai, Indonesian, Romanian, Czech and Farsi

  Posted: May 5th, 2010, 1:12pm CEST by Maile Ohye

  Tags:  feedback and communication  

  Webmaster Level: All
  
  Update on May 19, 2010: We have several translated versions of this post! If you're more comfortable reading Thai, Indonesian, Romanian, Czech, or Farsi, the links above will take you to your preferred version. Thanks again for your help.
  
  We pay attention to dozens of different languages in our spam fighting, but sometimes we really want to drill down and concentrate on a small number of languages. We’d like to ask for your help to identify webspam in Thai, Indonesian, Romanian, Czech and Farsi. If you know of sites that violate our webmaster guidelines in these languages, please send us a spam report. We use this information not only to look at the sites listed in reports, but also to improve our effectiveness in the rest of your language on the web.
  
  Thanks in advance for any data you send our way about spam in these languages. Of course, you’re always welcome to submit spam reports in other languages too!
  
  Written by Viktor Nebehaj and Matt Cutts, Search Quality Team
• 

  URL removal explained, Part IV: Tracking your requests & what not to remove

  Posted: May 3rd, 2010, 2:44pm CEST by Susan Moskwa

  Tags:  crawling and indexing  

  Webmaster Level: All
  
  In this final installation in our URL removal series, let's talk about following up on your removal requests, as well as when not to use Google's URL removal tool. If you haven't already, I recommend reading the previous posts in this series:
  Part I: Removing URLs & directories
  Part II: Removing & updating cached content
  Part III: Removing content you don't own
  Companion post: Managing what information is available about you online
  
  Understanding the status of your requests
  
  Once you've submitted a removal request, it will appear in your list of requests. You can check the status of your requests at any time to see whether the content has been removed, or whether the request is still or pending or was denied.
  
  
  
  If a request was denied, you should see a "Learn more" link next to it explaining why that particular request was denied. Since different types of removals have different requirements, the reason why a particular request was denied can vary. The "Learn more" link should help you figure out what you need to change in order to make your request successful. For example, you may need to change the URL in question so that it meets the requirements for the type of removal you requested; or, if you can't do that, you may need to request a different type of removal (one whose requirements your URL currently meets).
  
  If a request has been marked "Removed" but you still see that content in search results, check the following:

  • Is the URL that's appearing in search results the exact same URL that you submitted for removal? It's fairly common for the same, or similar, content to appear on multiple URLs on a site. You may have successfully removed one URL, but still see others containing that same content.
       Solution: Request removal of the other URL(s) in question. See this article for help.
  
  
  • Keep in mind that URLs are case sensitive, so requesting removal of http://www.example.com/embarrassingstuff.html is not the same as requesting removal of http://www.example.com/EmbarrassingStuff.html
       Solution: Request removal of the exact URL(s) that appear in search results, including the same capitalization. See this article for help.
  
  
  • When a request is marked "Removed," that can mean different things depending on what type of request you submitted. If you requested removal of an entire URL, then "Removed" should mean that that entire URL no longer appears in our search results. If you requested removal of the cached copy of a URL, "Removed" means that the cached copy has been removed and will no longer appear in search results; but the URL itself may still appear.
       Solution: Double-check what type of removal you requested by looking at the "Removal Type" column. If you requested a cache removal but you want the entire URL gone, make sure the URL meets the requirements for complete removal and then file a new request for complete removal of the URL.

  When not to use the URL removal tool

  • To clean up cruft, like old pages that 404.
    The tool is intended for URLs that urgently need to be removed, such as confidential data that was accidentally exposed. If you recently made changes to your site and just have some outdated URLs in the index, Google's crawlers will see this as we recrawl your URLs, and those pages will naturally drop out of our search results over time. There's no need to request an urgent removal through this tool.
  
  
  • To remove crawl errors from your Webmaster Tools account.
    The removal tool removes URLs from Google's search results, not from your Webmaster Tools account. There's currently no way for you to manually remove URLs from this report; they will drop out naturally over time as we stop crawling URLs that repeatedly 404.
  
  
  • To "start from scratch" with your site.
    If you're worried that your site may have a penalty, or you want to "start from scratch" after purchasing a domain from someone else, we don't recommend trying to use the URL removal tool to remove your entire site and then "start over." Search engines gather a lot of information from other sites (such as who links to you, or what words they use to describe your site) and use this to help understand your site. Even if we could remove everything we currently know about your site, a lot of it would come back exactly the same once we'd recrawled all the other sites that help us understand your site and put it in context. If you're worried that your domain has some bad history, we recommend filing a reconsideration request letting us know what you're worried about and what has changed (such as that you've acquired the domain from someone else, or that you've changed certain aspects of your site).
  
  
  • To take your site "offline" after hacking.
    If your site was hacked and you want to get rid of bad URLs that got indexed, you can use the URL removal tool to remove any new URLs that the hacker created, e.g., http://www.example.com/buy-cheap-cialis-skq3w598.html. But we don't recommend removing your entire site, or removing URLs that you'll eventually want indexed; instead, simply clean up the hacking and let us recrawl your site so that we can reindex the new, cleaned-up content as soon as possible. This article contains more details on how to deal with hacking.
  
  
  • To get the right "version" of your site indexed.
    When a request to remove https://www.example.com/tattoo.html is accepted, http://www.example.com/tattoo.html is also removed. The same is true of the www and non-www versions of your URL or site. This is because the same content is often available at each of these URLs and we realize that most webmasters and searchers don't want these duplicates appearing in search results. In short, the URL removal tool should not be used as a canonicalization tool. It won't keep your favorite version, it'll remove all versions (http/https and www/non-www) of a URL.

  We hope this series has answered your questions about removing content from Google's search results, and helped you troubleshoot any issues that may arise. Join us in our Help Forum if you still have questions.
  
  Posted by Susan Moskwa, Webmaster Trends Analyst
• 

  To slash or not to slash

  Posted: April 21st, 2010, 11:05am CEST by Maile Ohye

  Tags:  crawling and indexing  

  Webmaster Level: Intermediate
  
  That is the question we hear often. Onward to the answers! Historically, it’s common for URLs with a trailing slash to indicate a directory, and those without a trailing slash to denote a file:
  
  http://example.com/foo/ (with trailing slash, conventionally a directory)
  http://example.com/foo (without trailing slash, conventionally a file)
  
  But they certainly don’t have to. Google treats each URL above separately (and equally) regardless of whether it’s a file or a directory, or it contains a trailing slash or it doesn’t contain a trailing slash.
  
  Different content on / and no-/ URLs okay for Google, often less ideal for users
  
  From a technical, search engine standpoint, it’s certainly permissible for these two URL versions to contain different content. Your users, however, may find this configuration horribly confusing -- just imagine if www.google.com/webmasters and www.google.com/webmasters/ produced two separate experiences.
  
  For this reason, trailing slash and non-trailing slash URLs often serve the same content. The most common case is when a site is configured with a directory structure:
  http://example.com/parent-directory/child-directory/
  
  Your site’s configuration and your options
  
  You can do a quick check on your site to see if the URLs:
  

  1. http://<your-domain-here>/<some-directory-here>/
     (with trailing slash)
  2. http://<your-domain-here>/<some-directory-here>
     (no trailing slash)

  don’t both return a 200 response code, but that one version redirects to the other.
  

  • If only one version can be returned (i.e., the other redirects to it), that’s great! This behavior is beneficial because it reduces duplicate content. In the particular case of redirects to trailing slash URLs, our search results will likely show the version of the URL with the 200 response code (most often the trailing slash URL) -- regardless of whether the redirect was a 301 or 302.
  
  
  • If both slash and non-trailing-slash versions contain the same content and each returns 200, you can:
    • Consider changing this behavior (more info below) to reduce duplicate content and improve crawl efficiency.
    • Leave it as-is. Many sites have duplicate content. Our indexing process often handles this case for webmasters and users. While it’s not totally optimal behavior, it’s perfectly legitimate and a-okay. :)
    • Rest assured that for your root URL specifically, http://example.com is equivalent to http://example.com/ and can’t be redirected even if you’re Chuck Norris.

  Steps for serving only one URL version
  
  What if your site serves duplicate content on these two URLs:
  
  http://<your-domain-here>/<some-directory-here>/
  http://<your-domain-here>/<some-directory-here>
  
  meaning that both URLs return 200 (neither has a redirect or contains rel=”canonical”), and you want to change the situation?

  1. Choose one URL as the preferred version. If your site has a directory structure, it’s more conventional to use a trailing slash with your directory URLs (e.g., example.com/directory/ rather than example.com/directory), but you’re free to choose whichever you like.
  
  
  2. Be consistent with the preferred version. Use it in your internal links. If you have a Sitemap, include the preferred version (and don’t include the duplicate URL).
  
  
  3. Use a 301 redirect from the duplicate to the preferred version. If that’s not possible, rel=”canonical” is a strong option. rel=”canonical” works similarly to a 301 for Google’s indexing purposes, and other major search engines as well.
  
  
  4. Test your 301 configuration through Fetch as Googlebot in Webmaster Tools. Make sure your URLs:
     http://example.com/foo/
     http://example.com/foo
     are behaving as expected. The preferred version should return 200. The duplicate URL should 301 to the preferred URL.
  
  
  5. Check for Crawl errors in Webmaster Tools, and, if possible, your webserver logs as a sanity check that the 301s are implemented.
  
  
  6. Profit! (just kidding) But you can bask in the sunshine of your efficient server configuration, warmed by the knowledge that your site is better optimized.

  
  Written by Maile Ohye, Developer Programs Tech Lead
• 

  Webmasters - configure Google services at your hosting panel

  Posted: April 15th, 2010, 10:34am CEST by Maile Ohye

  Tags:  products and services  

  Webmaster Level: All
  
  Today, as announced on the Official Google Blog, we’ve taken an additional step to improve access to Google webmaster services. Parallels, one of the leading providers of control panel software to hosting companies, has integrated Google Services for Websites into Parallels Plesk Panel, used by millions of website owners globally to manage their websites.
  
  If you use Plesk for managing your hosting and website services, you can easily configure Webmaster Tools, Custom Search, Site Search, Web Elements and AdSense for your website right from within Plesk.
  
  Since Plesk knows what domains you own, it automatically registers your domains to Webmaster Tools and allows you to automatically login to the Webmaster Tools console to verify your sites, as shown below.
  
  
  
  We’re always trying to make our tools easier to use and easier to access. Since you’re probably visiting your hosting control panel on a regular basis, we hope that you find this integration convenient. If you have feedback please let us know in the Webmaster Forum.
  
  Posted by Jonathan Simon, Webmaster Trends Analyst
• 

  URL removals explained, part II: Removing sensitive text from a page

  Posted: April 6th, 2010, 4:25pm CEST by John Mueller

  Tags:  crawling and indexing  

  Webmaster level: All
  
  Change can happen—sometimes, as we saw in our previous post on URL removals, you may completely block or remove a page from your site. Other times you might only change parts of a page, or remove certain pieces of text. Depending on how frequently a page is being crawled, it can take some time before these changes get reflected in our search results. In this blog post we'll look at the steps you can take if we're still showing old, removed content in our search results, either in the form of a "snippet" or on the cached page that's linked to from the search result. Doing this makes sense when the old content contains sensitive information that needs to be removed quickly—it's not necessary to do this when you just update a website normally.
  
  As an example, let's look at the following fictitious search result:
  
  

  Walter E. Coyote	< Title
  Chief Development Officer at Acme Corp 1948-2003: worked on the top secret velocitus incalculii capturing device which has shown potential ...	< Snippet ...
  www.example.com/about/waltercoyote - Cached	< URL + link to cached page

  
  To change the content shown in the snippet (or on the linked cached page), you'll first need to change the content on the actual (live) page. Unless a page's publicly visible content is changed, Google's automatic processes will continue to show parts of the original content in our search results.
  
  Once the page's content has been changed, there are several options available to make those changes visible in our search results:
  
  

  1. Wait for Googlebot to re-crawl and re-index the page
     This is the natural method for how most content is updated at Google. Sometimes it can take a fairly long time, depending on how frequently Googlebot currently crawls the page in question. Once we've re-crawled and re-indexed the page, the old content will usually not be visible as it'll be replaced by the current content. Provided Googlebot is not blocked from crawling the page in question (either by robots.txt or by not being able to access the server properly), you don't have to do anything special for this to take place. It's generally not possible to speed up crawling and indexing, as these processes are fully automated and depend on many external factors.
     
     
  2. Use Google's public URL removal tool to request removal of content that has been removed from someone else's webpage
     Using this tool, it's necessary to enter the exact URL of the page that has been modified, select the "Content has been removed from the page" option, and then specify one or more words that have been completely removed from that page.
     
     
     
     Note that none of the words you enter can appear on the page; even if a word has been removed from one part of the page, your request will be denied if that word still appears on another part of the page. Be sure to choose a word (or words) that no longer appear anywhere on the page. If, in the above example, you removed "top secret velocitus incalculii capturing device," you should submit those words and not something like "my project." However, if the word "top" or "device" still exists anywhere on the page, the request would be denied. To maximize your chances of success, it's often easiest to just enter one word that you're sure no longer appears anywhere on the page.
     
     Once your request has been processed and it's found that the submitted word(s) no longer appear on the page, the search result will no longer show a snippet, nor will the cached page be available. The title and the URL of the page will still be visible, and the entry may still appear in search results for searches related to the content that has been removed (such as searches for [velocitus incalculii]), even if those words no longer appear in the snippet. However, once the page has been re-crawled and re-indexed, the new snippet and cached page can be visible in our search results.
     
     Keep in mind that we will need to verify removal of the word(s) by viewing the page. If the page no longer exists and the server is returning a proper 404 or 410 HTTP result code, making us unable to view the page, you may be better off requesting removal of the page altogether.
     
     
  3. Use Google Webmaster Tools URL removal tool to request removal of information on a page from your website
     If you have access to the website in question and have verified ownership of it in Google Webmaster Tools, you can use the URL removal tool there (under Site Configuration > Crawler access) to request that the snippet and the cached page be removed until the page has been re-crawled. To use this tool, you only need to submit the exact URL of the page (you won't need to specify any removed words). Once your request has been processed, we'll remove the snippet and the cached page from search results. The title and the URL of the page will still be visible, and the page may also continue to rank in search results for queries related to content that has been removed. After the page has been re-crawled and re-indexed, the search result with an updated snippet and cached page (based on the new content) can be visible.

  
  Google indexes and ranks items based not only on the content of a page, but also on other external factors, such as the inbound links to the URL. Because of this, it's possible for a URL to continue to appear in search results for content that no longer exists on the page, even after the page has been re-crawled and re-indexed. While the URL removal tool can remove the snippet and the cached page from a search result, it will not change or remove the title of the search result, change the URL that is shown, or prevent the page from being shown for searches based on any current or previous content. If this is important to you, you should make sure that the URL fulfills the requirements for a complete removal from our search results.
  
  Removing non-HTML content
  
  If the changed content is not in (X)HTML (for example if an image, a Flash file or a PDF file has been changed), you won't be able to use the cache removal tool. So if it's important that the old content no longer be visible in search results, the fastest solution would be to change the URL of the file so that the old URL returns a 404 HTTP result code and use the URL removal tool to remove the old URL. Otherwise, if you chose to allow Google to naturally refresh your information, know that previews of non-HTML content (such as Quick View links for PDF files) can take longer to update after recrawling than normal HTML pages would.
  
  Proactively preventing the appearance of snippets or cached versions
  
  As a webmaster, you have the option to use robots meta tags to proactively prevent the appearance of snippets or cached versions without using our removal tools. While we don't recommend this as a default approach (the snippet can help users recognize a relevant search result faster, and a cached page gives them the ability to view your content even in the unexpected event of your server not being available), you can use the "nosnippet" robots meta tag to prevent showing of a snippet, or the "noarchive" robots meta tag to disable caching of a page. Note that if this is changed on existing and known pages, Googlebot will need to re-crawl and re-index those pages before this change becomes visible in search results.
  
  We hope this blog post helps to make some of the processes behind the URL removal tool for updated pages a bit clearer. In our next blog post we'll look at ways to request removal of content that you don't own; stay tuned!
  
  As always, we welcome your feedback and questions in our Webmaster Help Forum.
  
  Posted by John Mueller, Webmaster Trends Analyst, Google Switzerland
• 

  URL removal explained, Part I: URLs & directories

  Posted: March 30th, 2010, 4:38pm CEST by Susan Moskwa

  Tags:  crawling and indexing  

  Webmaster level: All
  
  There's a lot of content on the Internet these days. At some point, something may turn up online that you would rather not have out there—anything from an inflammatory blog post you regret publishing, to confidential data that accidentally got exposed. In most cases, deleting or restricting access to this content will cause it to naturally drop out of search results after a while. However, if you urgently need to remove unwanted content that has gotten indexed by Google and you can't wait for it to naturally disappear, you can use our URL removal tool to expedite the removal of content from our search results as long as it meets certain criteria (which we'll discuss below).
  
  We've got a series of blog posts lined up for you explaining how to successfully remove various types of content, and common mistakes to avoid. In this first post, I'm going to cover a few basic scenarios: removing a single URL, removing an entire directory or site, and reincluding removed content. I also strongly recommend our previous post on managing what information is available about you online.
  
  Removing a single URL
  
  In general, in order for your removal requests to be successful, the owner of the URL(s) in question—whether that's you, or someone else—must have indicated that it's okay to remove that content. For an individual URL, this can be indicated in any of three ways:

  • block the page from crawling via a robots.txt file
  • block the page from indexing via a noindex meta tag
  • indicate that the page no longer exists by returning a 404 or 410 status code

  Before submitting a removal request, you can check whether the URL is correctly blocked:

  • robots.txt: You can check whether the URL is correctly disallowed using either the Fetch as Googlebot or Test robots.txt features in Webmaster Tools.
  • noindex meta tag: You can use Fetch as Googlebot to make sure the meta tag appears somewhere between the <head> and </head> tags. If you want to check a page you can't verify in Webmaster Tools, you can open the URL in a browser, go to View > Page source, and make sure you see the meta tag between the <head> and </head> tags.
  • 404 / 410 status code: You can use Fetch as Googlebot, or tools like Live HTTP Headers or web-sniffer.net to verify whether the URL is actually returning the correct code. Sometimes "deleted" pages may say "404" or "Not found" on the page, but actually return a 200 status code in the page header; so it's good to use a proper header-checking tool to double-check.

  If unwanted content has been removed from a page but the page hasn't been blocked in any of the above ways, you will not be able to completely remove that URL from our search results. This is most common when you don't own the site that's hosting that content. We cover what to do in this situation in a subsequent post. in Part II of our removals series.
  
  If a URL meets one of the above criteria, you can remove it by going to [www.google.com], entering the URL that you want to remove, and selecting the "Webmaster has already blocked the page" option. Note that you should enter the URL where the content was hosted, not the URL of the Google search where it's appearing. For example, enter
     http://www.example.com/embarrassing-stuff.html
  not
     http://www.google.com/search?q=embarrassing+stuff
  
  This article has more details about making sure you're entering the proper URL. Remember that if you don't tell us the exact URL that's troubling you, we won't be able to remove the content you had in mind.
  
  Removing an entire directory or site
  
  In order for a directory or site-wide removal to be successful, the directory or site must be disallowed in the site's robots.txt file. For example, in order to remove the http://www.example.com/secret/ directory, your robots.txt file would need to include:
     User-agent: *
     Disallow: /secret/
  It isn't enough for the root of the directory to return a 404 status code, because it's possible for a directory to return a 404 but still serve out files underneath it. Using robots.txt to block a directory (or an entire site) ensures that all the URLs under that directory (or site) are blocked as well. You can test whether a directory has been blocked correctly using either the Fetch as Googlebot or Test robots.txt features in Webmaster Tools.
  
  Only verified owners of a site can request removal of an entire site or directory in Webmaster Tools. To request removal of a directory or site, click on the site in question, then go to Site configuration > Crawler access > Remove URL. If you enter the root of your site as the URL you want to remove, you'll be asked to confirm that you want to remove the entire site. If you enter a subdirectory, select the "Remove directory" option from the drop-down menu.
  
  Reincluding content
  
  You can cancel removal requests for any site you own at any time, including those submitted by other people. In order to do so, you must be a verified owner of this site in Webmaster Tools. Once you've verified ownership, you can go to Site configuration > Crawler access > Remove URL > Removed URLs (or > Made by others) and click "Cancel" next to any requests you wish to cancel.
  
  Still have questions? Stay tuned for the rest of our series on removing content from Google's search results. If you can't wait, much has already been written about URL removals, and troubleshooting individual cases, in our Help Forum. If you still have questions after reading others' experiences, feel free to ask. Note that, in most cases, it's hard to give relevant advice about a particular removal without knowing the site or URL in question. We recommend sharing your URL by using a URL shortening service so that the URL you're concerned about doesn't get indexed as part of your post; some shortening services will even let you disable the shortcut later on, once your question has been resolved.
  
  Posted by Susan Moskwa, Webmaster Trends Analyst
• 

  Working with multilingual websites

  Posted: March 19th, 2010, 10:03am CET by John Mueller

  Tags:  crawling and indexing  

  Webmaster Level: Intermediate
  
  A multilingual website is any website that offers content in more than one language. Examples of multilingual websites might include a Canadian business with an English and a French version of its site, or a blog on Latin American soccer available in both Spanish and Portuguese.
  
  Usually, it makes sense to have a multilingual website when your target audience consists of speakers of different languages. If your blog on Latin American soccer aims to reach the Brazilian audience, you may choose to publish it only in Portuguese. But if you’d like to reach soccer fans from Argentina also, then providing content in Spanish could help you with that.
  
  Google and language recognition
  Google tries to determine the main languages of each one of your pages. You can help to make language recognition easier if you stick to only one language per page and avoid side-by-side translations. Although Google can recognize a page as being in more than one language, we recommend using the same language for all elements of a page: headers, sidebars, menus, etc.
  
  Keep in mind that Google ignores all code-level language information, from “lang” attributes to Document Type Definitions (DTD). Some web editing programs create these attributes automatically, and therefore they aren’t very reliable when trying to determine the language of a webpage.
  
  Someone who comes to Google and does a search in their language expects to find localized search results, and this is where you, as a webmaster, come in: if you’re going to localize, make it visible in the search results with some of our tips below.
  
  The anatomy of a multilingual site: URL structure
  There's no need to create special URLs when developing a multilingual website. Nonetheless, your users might like to identify what section of your website they’re on just by glancing at the URL. For example, the following URLs let users know that they’re on the English section of this site:
  
  http://example.ca/en/mountain-bikes.html
  http://en.example.ca/mountain-bikes.html
  
  While these other URLs let users know that they’re viewing the same page in French:
  
  http://example.ca/fr/mountain-bikes.html
  http://fr.example.ca/mountain-bikes.html
  
  Additionally, this URL structure will make it easier for you to analyze the indexing of your multilingual content.
  
  If you want to create URLs with non-English characters, make sure to use UTF-8 encoding. UTF-8 encoded URLs should be properly escaped when linked from within your content. Should you need to escape your URLs manually, you can easily find an online URL encoder that will do this for you. For example, if I wanted to translate the following URL from English to French,
  
  http://example.ca/fr/mountain-bikes.html
  
  It might look something like this:
  
  http://example.ca/fr/vélo-de-montagne.html
  
  Since this URL contains one non-English character (é), this is what it would look like properly escaped for use in a link on your pages:
  
  http://example.ca/fr/v%C3%A9lo-de-montagne
  
  Crawling and indexing your multilingual website
  We recommend that you do not allow automated translations to get indexed. Automated translations don’t always make sense and they could potentially be viewed as spam. More importantly, the point of making a multilingual website is to reach a larger audience by providing valuable content in several languages. If your users can’t understand an automated translation or if it feels artificial to them, you should ask yourself whether you really want to present this kind of content to them.
  
  If you’re going to localize, make it easy for Googlebot to crawl all language versions of your site. Consider cross-linking page by page. In other words, you can provide links between pages with the same content in different languages. This can also be very helpful to your users. Following our previous example, let’s suppose that a French speaker happens to land on http://example.ca/en/mountain-bikes.html; now, with one click he can get to http://example.ca/fr/vélo-de-montagne.html where he can view the same content in French.
  
  To make all of your site's content more crawlable, avoid automatic redirections based on the user's perceived language. These redirections could prevent users (and search engines) from viewing all the versions of your site.
  
  And last but not least, keep the content for each language on separate URLs - don't use cookies to show translated versions.
  
  Working with character encodings
  Google directly extracts character encodings from HTTP headers, HTML page headers, and content. There isn’t much you need to do about character encoding, other than watching out for conflicting information - for example, between content and headers. While Google can recognize different character encodings, we recommend that you use UTF-8 on your website whenever possible.
  
  If your tongue gets twisted...
  Now that you know all of this, your tongue may get twisted when you speak many languages, but your website doesn’t have to!
  
  For more information, read our post on multi-regional sites and stay tuned for our next post, where we'll delve into special situations that may arise when working with global websites. Until then, don't hesitate to drop by the Help Forum and join the discussion!
  
  Written by Xavier deMorales, Google Search Quality
• 

  Working with multi-regional websites

  Posted: March 12th, 2010, 8:50am CET by John Mueller

  Tags:  crawling and indexing  

  Webmaster Level: Intermediate
  
  Did you know that a majority of users surveyed feel that having information in their own language was more important than a low price? Living in a non-English-speaking country, I've seen friends and family members explicitly look for and use local and localized websites—properly localized sites definitely have an advantage with users. Google works hard to show users the best possible search results. Many times those are going to be pages that are localized, for the user's location and/or in the user's language.
  
  If you're planning to take the time to create and maintain a localized version of your website, making it easy to recognize and find is a logical part of that process. In this blog post series, we'll take a look at what is involved with multi-regional and multi-lingual websites from a search engine point of view. A multi-regional website is one that explicitly targets users in various regions (generally different countries); we call it multilingual when it is available in multiple languages, and sometimes, the website targets both multiple regions and is in multiple languages. Let's start with some general preparations and then look at websites that target multiple regions.
  
  Preparing for global websitesExpanding a website to cover multiple regions and/or languages can be challenging. By creating multiple versions of your website, any issues with the base version will be multiplied; make sure that you have everything working properly before you start. Given that this generally means you'll suddenly be working with a multiplied number of URLs, don't forget that you'll need appropriate infrastructure to support the website.
  
  Planning multi-regional websitesWhen planning sites for multiple regions (usually countries), don't forget to research legal or administrative requirements that might come into play first. These requirements may determine how you proceed, for instance whether or not you would be eligible to use a country-specific domain name.
  
  All websites start with domain names; when it comes to domain names, Google differentiates between two types of domain names:
  

  • ccTLDs (country-code top level domain names): These are tied to a specific country (for example .de for Germany, .cn for China). Users and search engines use this as a strong sign that your website is explicitly for a certain country.
  • gTLDs (generic top level domain names): These are not tied to a specific country. Examples of gTLds are .com, .net, .org, .museum. Google sees regional top level domain names such as .eu and .asia as gTLDs, since they cannot be tied to a specific country. We also treat some vanity ccTLDs (such as .tv, .me, etc.) as gTLDs as we've found that users and webmasters frequently see these as being more generic than country-targeted (we don't have a complete list of such vanity ccTLDs that we treat as gTLDs as it may change over time). You can set geotargeting for websites with gTLDs using the Webmaster Tools Geographic Target setting.

  
  Geotargeting factorsGoogle generally uses the following elements to determine the geotargeting of a website (or a part of a website):
  

  1. Use of a ccTLD is generally a strong signal for users since it explicitly specifies a single country in an unmistakable way.
     or
     Webmaster Tools' manual geotargeting for gTLDs (this can be on a domain, subdomain or subdirectory level); more information on this can be found in our blog post and in the Help Center. With region tags from geotargeting being shown in search results, this method is also very clear to users. Please keep in mind that it generally does not make sense to set a geographic target if the same pages on your site target more than a single country (say, all German-speaking countries) — just write in that language and do not use the geotargeting setting (more on writing in other languages will follow soon!).
  2. Server location (through the IP address of the server) is frequently near your users. However, some websites use distributed content delivery networks (CDNs) or are hosted in a country with better webserver infrastructure, so we try not to rely on the server location alone.
  3. Other signals can give us hints. This could be from local addresses & phone numbers on the pages, use of local language and currency, links from other local sites, and/or the use of Google's Local Business Center (where available).

  
  Note that we do not use locational meta tags (like "geo.position" or "distribution") or HTML attributes for geotargeting. While these may be useful in other regards, we've found that they are generally not reliable enough to use for geotargeting.
  
  URL structuresThe first three elements used for geotargeting are strongly tied to the server and to the URLs used. It's difficult to determine geotargeting on a page by page basis, so it makes sense to consider using a URL structure that makes it easy to segment parts of the website for geotargeting. Here are some of the possible URL structures with pros and cons with regards to geotargeting:
  
  

  ccTLDs eg: example.de, example.fr	Subdomains with gTLDs eg: de.site.com, fr.site.com, etc.	Subdirectories with gTLDs eg: site.com/de/, site.com/fr/, etc.	URL parameters eg: site.com?loc=de, ?country=france, etc.
  pros (+) - clear geotargeting - server location is irrelevant - easy separation of sites - legal requirements (sometimes)	pros (+) - easy to set up - can use Webmaster Tools geotargeting - allows different server locations - easy separation of sites	pros (+) - easy to set up - can use Webmaster Tools geotargeting - low maintenance (same host)	pros (+) (not recommended)
  cons (-) - expensive (+ availability) - more infrastructure - ccTLD requirements (sometimes)	cons (-) - users might not recognize geotargeting from the URL alone (is "de" the language or country?)	cons (-) - users might not recognize geotargeting from the URL alone - single server location - separation of sites harder	cons (-) - segmentation based on the URL is difficult - users might not recognize geotargeting from the URL alone - geotargeting in Webmaster Tools is not possible

  
  As you can see, geotargeting is not an exact science (even sites using country-code top level domain names can be global in nature), so it's important that you plan for the users from the "wrong" location. One way to do this could be to show links on all pages for users to select their region and language of choice. We'll look at some other possible solutions further on in this blog post series.
  
  Dealing with duplicate content on global websitesWebsites that provide content for different regions and in different languages sometimes create content that is the same or similar but available on different URLs. This is generally not a problem as long as the content is for different users in different countries. While we strongly recommend that you provide unique content for each different group of users, we understand that this may not always be possible for all pages and variations from the start. There is generally no need to "hide" the duplicates by disallowing crawling in a robots.txt file or by using a "noindex" robots meta tag. However, if you're providing the same content to the same users on different URLs (for instance, if both "example.de/" and "example.com/de/" show German language content for users in Germany), it would make sense to choose a preferred version and to redirect (or use the "rel=canonical" link element) appropriately.
  
  Do you already have a website that targets multiple regions or do you have questions about the process of planning one? Come to the Help Forum and join the discussion. In following posts, we'll take a look at multi-lingual websites and then look at some special situations that can arise with global websites. Bis bald!
  
  Written by John Mueller, Webmaster Trends Analyst, Google Switzerland
• 

  Microdata support for Rich Snippets

  Posted: March 9th, 2010, 6:51pm CET by Maile Ohye

  Tags:  crawling and indexing  

  Webmaster Level: All
  
  HTML5 is the fifth major revision of HTML, the core language of the World Wide Web. The HTML5 specification includes a description of microdata, a new markup standard for specifying structured information within web pages.
  
  Today, we’re happy to announce support for microdata for use in rich snippets in addition to our existing support for microformats and RDFa. By using microdata markup in your web pages, you can specify reviews, people profiles, or events information on your web pages that Google may use to improve the presentation of your pages in Google search results.
  
  Here is a simple HTML block showing a section of a review of “L’Amourita Pizza”:
  
  
  Here is the same HTML with microdata added to specify the restaurant being reviewed, the author and date of the review, and the rating:
  
  
  Microdata has the nice property of balancing richness with simplicity. As you can see, it’s easy to add markup to your pages using a few HTML attributes like itemscope (to define a new item), itemtype (to specify the type of item being described), and itemprop (to specify a property of that item). Once you’ve added markup to a page, you can test it using the rich snippets testing tool to make sure that Google can parse the data on your page.
  
  As with microformats and RDFa, the vocabulary that we support -- including which item types and item properties are understood by Google -- is specified in our rich snippets documentation as well as on data-vocabulary.org. Marking up your content does not guarantee that rich snippets will show for your site; Google will expand the use of microdata markup gradually to ensure a great user experience.
  
  To get started, here are some helpful links:
  

  • Rich snippets documentation
  • Overview of microdata
  • Official microdata specification
  • Rich snippets testing tool

  
  Written by Siddhartha Chattopadhyay, Kavi Goel, Ramanathan V. Guha, Pravir Gupta, Othar Hansson
• 

  Google's SEO Report Card

  Posted: March 2nd, 2010, 5:02pm CET by Maile Ohye

  Tags:  feedback and communication  

  Webmaster Level: All
  
  How many of Google's web pages use a descriptive title tag? Do we use description meta tags? Heading tags? While we always try to focus on the user, could our products use an SEO tune up? These are just some of the questions we set out to answer with Google's SEO Report Card.
  
  Google's SEO Report Card is an effort to provide Google's product teams with ideas on how they can improve their products' pages using simple and accepted optimizations. These optimizations are intended to not only help search engines understand the content of our pages better, but also to improve our users' experience when visiting our sites. Simple steps such as fixing 404s and broken links, simplifying URL choice, and providing easier-to-understand titles and snippets for our pages can benefit both users and search engines. From the start of the project we also wanted to release the report card publicly so other companies and webmasters could learn from the report, which is filled with dozens of examples taken straight from our products' pages.
  
  The project looked at the main pages of 100 different Google products, measuring them across a dozen common optimization categories. Future iterations of the project might look at deeper Google product web pages as well as international ones. We released the report card within Google last month and since then a good number of teams have taken action on it or plan to.
  
  We hope you find our SEO Report Card useful and we'd love to hear your feedback in the comments below or in the Webmaster Central Help Forum. And if you'd like to do your own SEO tune up, a good place to start is by reading our free SEO Beginner's Guide.
  
  Written by Brandon Falls, Adi Goradia, and Charlene Perez, Search Quality Team
• 

  Is your site hacked? New Message Center notifications for hacking and abuse

  Posted: March 1st, 2010, 4:00pm CET by Michael Wyszomierski

  Tags:  feedback and communication  

  Webmaster Level: All
  As we crawl the web, we see bad content inserted on to thousands of hacked sites each day. The number of sites attacked is staggering and the problem is only getting worse. Hackers and spammers target and successfully compromise any sites they can - small personal sites, schools and universities, even multinational corporations. Spam attacks against forums and user content sections of sites, though not as shocking, are even more widespread.
  You may have read in an earlier post that we've begun notifying webmasters about new software versions via Webmaster Tools to help protect their sites. Continuing with our effort to provide more useful information to webmasters, we're happy to announce that we'll soon be sending even more notifications to the Message Center.
  Starting this month, we will notify more webmasters of more potential issues we've detected on their websites, including:

  • Spammy or abused user-generated content
  • Abused forum pages or egregious amounts of comment spam
  • Suspected hacking

  These notifications are meant to alert webmasters of potential issues and provide next steps on how to get their sites fixed and back into Google's search results. If it pertains to a hacking or abuse issue, the notification will point to example URLs exhibiting this type of behavior. These notifications will run in parallel with our existing malware notifications.
  A notice of suspected hacking, for example, will look like this:
  
  We've been notifying webmasters of suspected hacking for years, but a recent upgrade to our systems will allow us to notify many more site owners that have been hacked. We hope webmasters will find these notifications useful in making sure their sites are clean and secure, ultimately providing a better user experience for their visitors. In the future, we may extend this effort even further to include other types of vulnerabilities or abuse issues.
  Just as before, webmasters who have not already signed up for Webmaster Tools may still do so and retrieve previously sent messages within one year of their send date. And if you don't want to miss out on any important messages, remember to use the email forwarding feature to receive these alerts in your inbox.
  If you have any questions, please feel free to ask in our Webmaster Help Forum or leave your comments below.
  Posted by Jessica Wong and Jason Morrison, Search Quality Team
• 

  Protect your site from spammers with reCAPTCHA

  Posted: January 26th, 2010, 11:00am CET by Michael Wyszomierski

  Tags:  products and services  

  Webmaster Level: All
  If you allow users to publish content on your website, from leaving comments to creating user profiles, you’ll likely see spammers attempt to take advantage of these mechanisms to generate traffic to their own sites. Having this spammy content on your site isn't fun for anyone. Users may be subjected to annoying advertisements directing them to low-quality or dangerous sites containing scams or malware. And you as a webmaster may be hosting content that violates a search engine's quality guidelines, which can harm your site's standing in search results.
  There are ways to handle this abuse, such as moderating comments and reviewing new user accounts, but there is often so much spam created that it can become impossible to keep up with. Spam can easily get to this unmanageable level because most spam isn’t created manually by a human spammer. Instead, spammers use computer programs called “bots” to automatically fill out web forms to create spam, and these bots can generate spam much faster than a human can review it.
  To level the playing field, you can take steps to make sure that only humans can interact with potentially spammable features of your website. One way to determine which of your visitors are human is by using a CAPTCHA , which stands for "completely automated public Turing test to tell computers and humans apart." A typical CAPTCHA contains an image of distorted letters which humans can read, but are not easily understood by computers. Here's an example:
  
  You can easily take advantage of this technology on your own site by using reCAPTCHA, a free service owned by Google. One unique aspect of reCAPTCHA is that data collected from the service is used to improve the process of scanning text, such as from books or newspapers. By using reCAPTCHA, you're not only protecting your site from spammers; you're helping to digitize the world's books.
  Luis Von Ahn, one reCAPTCHA's co-founders, gives more details about how the service works in the video below:
  
  If you’d like to implement reCAPTCHA for free on your own site, you can sign up here. Plugins are available for easy installation on popular applications and programming environments such as WordPress and PHP.
  Posted by Michael Wyszomierski, Search Quality Team
• 

  Introducing a new Rich Snippets format: Events

  Posted: January 22nd, 2010, 11:20am CET by Maile Ohye

  Tags:  crawling and indexing  

  Webmaster Level: All
  
  Last year we introduced Rich Snippets, a new feature that makes it possible to surface structured data from your pages on Google's search results. So far, user reaction to Rich Snippets has been enthusiastic -- after all, Rich Snippets help people make more informed clicks and find what they need even faster.
  
  We originally introduced Rich Snippets with two formats: reviews and people. Later in the year we added support for marking up video information which is used to improve Video Search. Today, we're excited to kick off the new year by adding support for events.
  
  Events markup is based off of the hCalendar microformat. Here's an example of what the new events Rich Snippets will look like:
  
  
  The new format shows links to specific events on the page along with dates and locations. It provides a fast and convenient way for users to determine if a page has events they may be interested in.
  
  If you have event listings on your site, we encourage you to review the events documentation we've prepared to help you get started. Please note, however, that marking up your content is not a guarantee that Rich Snippets will show for your site. Just as we did for previous formats, we will take a gradual approach to incorporating the new event snippets to ensure a great user experience along the way.
  
  Stay tuned for more developments in Rich Snippets throughout the year!
  
  Written by Mike Danylchuk and Nitin Shetti
• 

  State of the Index 2009

  Posted: January 20th, 2010, 3:50pm CET by Michael Wyszomierski

  Tags:  feedback and communication  

  Webmaster Level: All
  At PubCon in Las Vegas in November 2009, I gave a "State of the Index" talk which covers what Google has done for users, web developers, and webmasters in the last year. I recently recreated it on video for those of you who didn't make it to the conference. You can watch it below:
  
  And here are the slides if you'd like to follow along:
  
  Posted by Matt Cutts, Search Quality Team
• 

  Answering your December Grab Bag questions

  Posted: January 14th, 2010, 10:16am CET by Michael Wyszomierski

  Tags:  feedback and communication  

  Webmaster Level: All
  You asked and Matt Cutts answered. It's time to answer the latest round of Grab Bag questions! Here's the first answer, complete with Matt's new hairstyle:
  
  We have a lot of videos ready to share, so we're not currently taking new questions for the Grab Bag. If you have a question that you would like answered, your best bet as always is to head to our Webmaster Help Forum, where you'll find plenty of knowledgeable webmasters, including some Bionic Posters!
  To be kept up-to-date on our latest video releases, you can follow @googlewmc on Twitter, where we'll announce new videos and blog posts as they're published.
  Posted by Michael Wyszomierski, Search Quality Team
• 

  Helping webmasters from user to user

  Posted: December 24th, 2009, 3:30pm CET by John Mueller

  Tags:  feedback and communication  

  You have to have some kind of super-powers to keep up with all of the issues posted in our Webmaster Help Forum—that's why we call our Top Contributors the "Bionic Posters." They're able to leap through tall questions in a single bound, providing helpful and solid information all around. We're thankful to the Bionics for tackling problems both hard and easy (well, easy if you know how). Our current Bionic Posters are: Webado (Christina), Phil Payne, Red Cardinal (Richard), Shades1 (Louis), Autocrat, Tim Abracadabra, Aaron, Cristina, Robbo, John, Becky Sharpe, Sasch, BbDeath, Beussery (Brian), Chibcha (Terry), Luzie (Herbert), 奥宁 (Andy), Ashley, Kaleh and Redleg!
  
  With thousands of webmasters visiting the English Help Forum every day, some questions naturally pop up more often than others. To help catch these common issues, the Bionic Posters have also helped to create and maintain a comprehensive list of frequently asked questions and their answers. These FAQs cover everything from "Why isn't my site indexed?" to diagnosing difficult issues with the help of Google Webmaster Tools, often referring to our Webmaster Help Center for specific topics. Before you post in the forum, make sure you've read through these resources and do a quick search in the forum; chances are high that your question has been answered there already.
  
  Besides the Bionic Posters, we're lucky to have a number of very active and helpful users in the forum, such as: squibble, Lysis, yasir, Steven Lockey, seo101, RickyD, MartinJ and many more. Thank you all for making this community so captivating and—most of the time—friendly.
  
  Here are just a few (well, a little more than a few) of the many comments that we've seen posted in the forum:
  
  

  • "Thank you for this forum... Thank you to those that take the time to answer and care!"
  • "I've only posted one question here, but have received a wealth of knowledge by reading tons of posts and answers. The time you experts put into helping people with their problems is very inspiring and my hat's off to each of you. Anyway, I just wanted to let you know that your services aren't going unnoticed and I truly appreciate the lessons."
  • "Thank you very much cristina, what you told me has done the trick. I really appriciate the help as this has been bugging me for a while now and I didn't know what was wrong."
  • "thank you ssssssssssoooo much kaleh. "
  • "OK, Phil Payne big thanks to You! I have made changes and maybe people are starting to find me in G! Thanks to Ashley, I've started to make exclusive and relevant content for people."
  • "If anything, it has helped me reflect on the sites and projects of days gone by so as to see what I could have done better - so that I can deliver that much more and better results going forward. I've learned that some things I had done right, were spot on, and other issues could have been handled differently, as well as a host of technical information that I've stored away for future use. Bottom Line: this forum rocks and is incredibly helpful."
  • "I asked a handful of questions, got GREAT help while doing a whole lot of lurking, and now I've got a site that rocks!! (...) Huge thanks to all the Top Contributors, and a very special mention to WEBADO, who helped me a TON with my .htaccess file."
  • "Over the years of reading (and sometimes contributing) to this forum I think it has helped to remove many false assumptions and doubts over Google's ranking systems. Contrary to what many have said I verily believe Google can benefit small businesses. Keep up the good work. "
  • "The forum members are awesome and are a most impressive bunch. Their contribution is immeasurable as it is huge. Not only have they helped Google in their success as a profitable business entity, but also helped webmasters both aspiring and experienced. There is also an engender(ment) of "family" or "belonging" in the group that has transcended the best and worst of times (Current forum change still TBD :-) ). We can agree, disagree and agree to disagree but remain respectful and civil (Usually :-) )."
  • "Hi Redleg, Thank you very much for all of the information. Without your help, I don't think I would ever have known how to find the problem. "
  • "What an amazing board. Over the last few days I have asked 1 question and recieved a ton of advice mainly from Autocrat. "
  • "A big thank you to the forum and the contributors that helped me get my site on Google . After some hassle with my web hosters and their naff submission service, issues over adding pages Google can see, issues over Sitemaps, I can now say that when I put my site name into the search and when i put in [custom made watch box], for instance, my site now comes up."
  • "Thank you Autocrat! You are MAGNIFICENT! (...) I am your biggest fan today. : ) Imagine Joe Cocker singing With a Little Help from My Friends...that's my theme song today."
  • "I've done a lot of reading since then and I've learned more in the last year than I learned in the previous 10. When I stumbled into this forum I had no idea what I was getting into but finding this forum was a gift from God! Words cannot express the amount of gratitude I feel for the help you have given me and I wish I could repay you some how.... I don't mean to sound so mushy, but I write this with tears in my eyes and I am truly, truly grateful..."

  
  Are you new to the Webmaster Help Forum? Tell us a little bit about yourself and then join us to learn more and help others!
  
  Posted by John Mueller, Webmaster Trends Analyst, Google Zürich
• 

  Handling legitimate cross-domain content duplication

  Posted: December 15th, 2009, 3:47pm CET by John Mueller

  Tags:  crawling and indexing  

  Webmaster level: Intermediate
  
  We've recently discussed several ways of handling duplicate content on a single website; today we'll look at ways of handling similar duplication across different websites, across different domains. For some sites, there are legitimate reasons to duplicate content across different websites — for instance, to migrate to a new domain name using a web server that cannot create server-side redirects. To help with issues that arise on such sites, we're announcing our support of the cross-domain rel="canonical" link element.
  
  
  
  Ways of handling cross-domain content duplication:
  

  • Choose your preferred domain
    When confronted with duplicate content, search engines will generally take one version and filter the others out. This can also happen when multiple domain names are involved, so while search engines are generally pretty good at choosing something reasonable, many webmasters prefer to make that decision themselves.

  • Reduce in-site duplication
    Before starting on cross-site duplicate content questions, make sure to handle duplication within your site first.

  • Enable crawling and use 301 (permanent) redirects where possible
    Where possible, the most important step is often to use appropriate 301 redirects. These redirects send visitors and search engine crawlers to your preferred domain and make it very clear which URL should be indexed. This is generally the preferred method as it gives clear guidance to everyone who accesses the content. Keep in mind that in order for search engine crawlers to discover these redirects, none of the URLs in the redirect chain can be disallowed via a robots.txt file. Don't forget to handle your www / non-www preference with appropriate redirects and in Webmaster Tools.

  • Use the cross-domain rel="canonical" link element
    There are situations where it's not easily possible to set up redirects. This could be the case when you need to move your website from a server that does not feature server-side redirects. In a situation like this, you can use the rel="canonical" link element across domains to specify the exact URL of whichever domain is preferred for indexing. While the rel="canonical" link element is seen as a hint and not an absolute directive, we do try to follow it where possible.

  
  
  Still have questions?
  
  Q: Do the pages have to be identical?
  A: No, but they should be similar. Slight differences are fine.
  
  Q: For technical reasons I can't include a 1:1 mapping for the URLs on my sites. Can I just point the rel="canonical" at the homepage of my preferred site?
  A: No; this could result in problems. A mapping from old URL to new URL for each URL on the old site is the best way to use rel="canonical".
  
  Q: I'm offering my content / product descriptions for syndication. Do my publishers need to use rel="canonical"?
  A: We leave this up to you and your publishers. If the content is similar enough, it might make sense to use rel="canonical", if both parties agree.
  
  Q: My server can't do a 301 (permanent) redirect. Can I use rel="canonical" to move my site?
  A: If it's at all possible, you should work with your webhost or web server to do a 301 redirect. Keep in mind that we treat rel="canonical" as a hint, and other search engines may handle it differently. But if a 301 redirect is impossible for some reason, then a rel="canonical" may work for you. For more information, see our guidelines on moving your site.
  
  Q: Should I use a noindex robots meta tag on pages with a rel="canonical" link element?
  A: No, since those pages would not be equivalent with regards to indexing - one would be allowed while the other would be blocked. Additionally, it's important that these pages are not disallowed from crawling through a robots.txt file, otherwise search engine crawlers will not be able to discover the rel="canonical" link element.
  
  We hope this makes it easier for you to handle duplicate content in a user-friendly way. Are there still places where you feel that duplicate content is causing your sites problems? Let us know in the Webmaster Help Forum!
  
  
  Posted by John Mueller, Webmaster Trends Analyst, Google Zürich
• 

  New User Agent for News

  Posted: December 2nd, 2009, 9:02am CET by Jonathan Simon

  Tags:  crawling and indexing  

  Webmaster Level: Intermediate
  
  Today we are announcing a new user agent for robots.txt called Googlebot-News that gives publishers even more control over their content. In case you haven't heard of robots.txt, it's a web-wide standard that has been in use since 1994 and which has support from all major search engines and well-behaved "robots" that process the web. When a search engine checks whether it has permission to crawl and index a web page, the "check if we're allowed to crawl this page" mechanism is robots.txt.
  
  Publishers could easily contact us via a form if they didn't want to be included in Google News but did want to be in Google's web search index. Now, publishers can manage their content in Google News in an even more automated way. Site owners can just add Googlebot-News specific directives to their robots.txt file. Similar to the Googlebot and Googlebot-Image user agents, the new Googlebot-News user agent can be used to specify which pages of a website should be crawled and ultimately appear in Google News.
  
  Here are a few examples for publishers:
  
  Include pages in both Google web search and News:
  User-agent: Googlebot
  Disallow:
  
  This is the easiest case. In fact, a robots.txt file is not even required for this case.
  
  Include pages in Google web search, but not in News:
  User-agent: Googlebot
  Disallow:
  
  User-agent: Googlebot-News
  Disallow: /
  
  This robots.txt file says that no files are disallowed from Google's general web crawler, called Googlebot, but the user agent "Googlebot-News" is blocked from all files on the website.
  
  Include pages in Google News, but not Google web search:
  User-agent: Googlebot
  Disallow: /
  
  User-agent: Googlebot-News
  Disallow:
  
  When parsing a robots.txt file, Google obeys the most specific directive. The first two lines tell us that Googlebot (the user agent for Google's web index) is blocked from crawling any pages from the site. The next directive, which applies to the more specific user agent for Google News, overrides the blocking of Googlebot and gives permission for Google News to crawl pages from the website.
  
  Block different sets of pages from Google web search and Google News:
  User-agent: Googlebot
  Disallow: /latest_news
  
  User-agent: Googlebot-News
  Disallow: /archives
  
  The pages blocked from Google web search and Google News can be controlled independently. This robots.txt file blocks recent news articles (URLs in the /latest_news folder) from Google web search, but allows them to appear on Google News. Conversely, it blocks premium content (URLs in the /archives folder) from Google News, but allows them to appear in Google web search.
  
  Stop Google web search and Google News from crawling pages:
  User-agent: Googlebot
  Disallow: /
  
  This robots.txt file tells Google that Googlebot, the user agent for our web search crawler, should not crawl any pages from the site. Because no specific directive for Googlebot-News is given, our News search will abide by the general guidance for Googlebot and will not crawl pages for Google News.
  
  For some queries, we display results from Google News in a discrete box or section on the web search results page, along with our regular web search results. We sometimes do this for Images, Videos, Maps, and Products, too. This is known as Universal search results. Since Google News powers Universal "News" search results, if you block the Googlebot-News user agent then your site's news stories won't be included in Universal search results.
  
  We are currently testing our support for the new user agent. If you see any problems please let us know. Note that it is possible for Google to return a link to a page in some situations even when we didn't crawl that page. If you'd like to read more about robots.txt, we provide additional documentation on our website. We hope webmasters will enjoy the flexibility and easier management that the Googlebot-News user agent provides.
  
  Written by Jonathan Simon, Webmaster Trends Analyst
• 

  Region Tags in Google Search Results

  Posted: December 1st, 2009, 5:25pm CET by Maile Ohye

  Tags:  crawling and indexing  

  Webmaster Level: All
  
  Country-code top-level domains (or ccTLDs) can provide people with a quick and valuable clue about the location of a website—for example, ".fr" for France or ".co.jp" for Japan. However, for certain top level domains like .com, .info and .org, it's not as easy to figure out the location. That's why today we're adding region information supplied by webmasters to the green address line on some Google search results.
  
  This feature is easiest to explain through an example. Let's say you've heard about a boxing club in Canada called "Capital City Boxing." You try a search for [capital city boxing] to find out more, but it's hard to tell which result is the one you're looking for. Here's a screen shot:
  
  
  None of the results provide any location information in the title or snippet, nor do they have a regional TLD (such as .ca for Canada). The only way to find the result you're looking for is to refine your search ([capital city boxing canada] works) or click through the various links to figure it out. Clicking through the first result reveals that there's apparently another "Capital City Boxing" club in Alabama.
  
  Region tags improve search results by providing valuable information about website location right in the green URL line. Continuing our prior example, here's a screen shot of the new region tag (circled in red):
  
  
  
  As you can see, the fourth result now includes the region name "Canada" after the green URL, so you can immediately tell that this result relates to the boxing club in Canada. With the new display, you no longer need to refine your search or click through the results to figure out which page is the one you're looking for. In general, our hope is that these region tags will help searchers more quickly identify which results are most relevant to their queries.
  
  As a webmaster, you can control how this feature works by adjusting your Geographic Targeting settings. Log in to Webmaster Tools and choose Site configuration > Settings > Geographic Target. From here you can associate a particular country/region with your site. These settings will determine the name that appears as a region tag. You can learn more about using the Geographic Target tool in a prior blog post and in our Help Center.
  
  We currently show region tags only for certain domains such as .com and .net where the location information would otherwise be unclear. We don't show region tags for results on domains like .br for Brazil, because the location is already implied by the green URL line in our default display. In addition, we only display region tags when the region supplied by the site owner is different from the domain where the search was entered. For example, if you do a search from the Singapore Google domain (google.com.sg), we won't show you region tags for all the websites webmasters have targeted to Singapore because we'd end up tagging too many results, and the tag is really most relevant for foreign regions. For the initial release, we anticipate roughly 1% of search results pages will include webpages with a region tag.
  
  We hope you'll find this new feature useful, and we welcome your feedback.
  
  Written by Piyush Prahladka, Software Engineer
• 

  Changes in First Click Free

  Posted: December 1st, 2009, 10:02am CET by John Mueller

  Tags:  products and services  

  Webmaster level: intermediate
  
  We love helping publishers make their content available to large groups of readers, and working on ways to make the world's information useful and accessible through our search results. At the same time, we're also aware of the fact that creating high-quality content is not easy and, in many cases, expensive. This is one of the reasons why we initially launched First Click Free for Google News and Google Web Search -- to allow publishers to sell access to their content in general while still allowing users to find it through our search results.
  
  While we're happy to see that a number of publishers are already using First Click Free, we've found that some who might try it are worried about people abusing the spirit of First Click Free to access almost all of their content. As most users are generally happy to be able to access just a few pages from these premium content providers, we've decided to allow publishers to limit the number of accesses under the First Click Free policy to five free accesses per user each day. This change applies to both Google News publishers as well as websites indexed in Google's Web Search. We hope that this encourages even more publishers to open up more content to users around the world!
  
  Questions and answers about First Click Free
  
  Q: Do the rest of the old guidelines still apply?
  A: Yes, please check the guidelines for Google News as well as the guidelines for Web Search and the associated blog post for more information.
  
  Q: Can I apply First Click Free to only a section of my site / only for Google News (or only for Web Search)?
  A: Sure! Just make sure that both Googlebot and users from the appropriate search results can view the content as required. Keep in mind that showing Googlebot the full content of a page while showing users a registration page would be considered cloaking.
  
  Q: Do I have to sign up to use First Click Free?
  A: Please let us know about your decision to use First Click Free if you are using it for Google News. There's no need to inform us of the First Click Free status for Google Web Search.
  
  Q: What is the preferred way to count a user's accesses?
  A: Since there are many different site architectures, we believe it's best to leave this up to the publisher to decide.
  
  (Please see our related blog post for more information on First Click Free for Google News.)
  
  Posted by John Mueller, Webmaster Trends Analyst, Google Zürich
• 

  GENERIC CIALIS on my website? I think my site has been hacked!

  Posted: November 26th, 2009, 4:23am CET by John Mueller

  Tags:  crawling and indexing  

  How to use "Fetch as Googlebot", part 1
  Webmaster level: Intermediate
  
  Has your site ever dropped suddenly from the index or disappeared mysteriously from search results? Have you ever received a notice that your site is using cloaking techniques? Unfortunately, sometimes a malicious party "hacks" a website: they penetrate the security of a site and insert undesirable content. Sophisticated attackers can camouflage this spammy or dangerous content so that it doesn't appear for normal users, and appears only to Googlebot, which could negatively impact your site in Google's results.
  
  In such cases it used to be very difficult to detect the problem, because the site would appear normal in the eyes of the user. It may be possible that only requests with a User-agent: of Googlebot and coming from Googlebot's IP could see the hidden content. But that's over: with Fetch as Googlebot, the new Labs feature in Webmaster Tools, you can see exactly what Googlebot is seeing, and avoid any kind of cloaking problems. We'll show you how:
  
  Let's imagine that Bob, the administrator of www.example.com, is searching for his site but he finds this instead:
  
  
  
  That's strange, because when he looks at the source code of www.example.com, it looks fine:
  
  
  
  With much surprise Bob may receive a notice from Google warning him that his site is not complying with Google's quality guidelines. Fortunately he has his site registered with Webmaster Tools, let's see how he can check what Googlebot sees:
  
  First Bob logs into Webmaster Tools and selects www.example.com. The Fetch as Googlebot feature will be at the bottom of the navigation menu, in the Labs section:
  
  
  
  The page will contain a field where you can insert the URL to fetch. It can also be left blank to fetch the homepage.
  
  
  
  Bob can simply click Fetch and wait a few seconds. After refreshing the page, he can see the status of the fetch request. If it succeeds, he can click on the "Success" link...
  
  
  
  ...and that will show the details, with the content of the fetched page:
  
  
  
  Aha! There's the spammy content! Now Bob can be certain that www.example.com has been hacked.
  
  Confirming that the website has been hacked (and perhaps is still hacked) is an important step. It is, however, only the beginning. For more information, we strongly suggest getting help from your server administrator or hoster and reading our previous blog posts on the subject of hacked sites:
  
  

  • My site's been hacked - now what?
  
  
  • A quick security checklist for webmasters
  
  
  • Message center warnings for hackable sites
  
  
  • Best practices against hacking

  
  If you have any questions about how to use the Fetch as Googlebot feature, feel free to drop by the Webmaster Help Forum. If you feel that your website might be hacked but are having problems resolving it, you might want to ask the experts in our "Malware and Hacked sites" category.
  
  PS Keep in mind that once you have removed hacked content from your site, it will generally still take time for us to update our search results accordingly. There are a number of factors that affect crawling and indexing of your content so it's impossible to give a time-frame for that.
  
  Written by Javier Tordable, Software Engineer and John Mueller, Webmaster Trends Analyst
• 

  Running desktop and mobile versions of your site

  Posted: November 18th, 2009, 11:15pm CET by Maile Ohye

  Tags:  crawling and indexing  

  (This post was largely translated from our Japanese version of the Webmaster Central Blog )
  
  Recently I introduced several methods to ensure your mobile site is properly indexed by Google. Today I'd like to share information useful for webmasters who manage both desktop and mobile phone versions of a site.
  
  One of the most common problems for webmasters who run both mobile and desktop versions of a site is that the mobile version of the site appears for users on a desktop computer, or that the desktop version of the site appears when someone finds them from a mobile device. In dealing with this scenario, here are two viable options:
  
  Redirect mobile users to the correct version

  When a mobile user or crawler (like Googlebot-Mobile) accesses the desktop version of a URL, you can redirect them to the corresponding mobile version of the same page. Google notices the relationship between the two versions of the URL and displays the standard version for searches from desktops and the mobile version for mobile searches.
  
  If you redirect users, please make sure that the content on the corresponding mobile/desktop URL matches as closely as possible. For example, if you run a shopping site and there's an access from a mobile phone to a desktop-version URL, make sure that the user is redirected to the mobile version of the page for the same product, and not to the homepage of the mobile version of the site. We occasionally find sites using this kind of redirect in an attempt to boost their search rankings, but this practice only results in a negative user experience, and so should be avoided at all costs.
  
  On the other hand, when there's an access to a mobile-version URL from a desktop browser or by our web crawler, Googlebot, it's not necessary to redirect them to the desktop-version. For instance, Google doesn't automatically redirect desktop users from their mobile site to their desktop site, instead they include a link on the mobile-version page to the desktop version. These links are especially helpful when a mobile site doesn't provide the full functionality of the desktop version -- users can easily navigate to the desktop-version if they prefer.

  
  Switch content based on User-agent

  Some sites have the same URL for both desktop and mobile content, but change their format according to User-agent. In other words, both mobile users and desktop users access the same URL (i.e. no redirects), but the content/format changes slightly according to the User-agent. In this case, the same URL will appear for both mobile search and desktop search, and desktop users can see a desktop version of the content while mobile users can see a mobile version of the content.
  
  However, note that if you fail to configure your site correctly, your site could be considered to be cloaking, which can lead to your site disappearing from our search results. Cloaking refers to an attempt to boost search result rankings by serving different content to Googlebot than to regular users. This causes problems such as less relevant results (pages appear in search results even though their content is actually unrelated to what users see/want), so we take cloaking very seriously.
  
  So what does "the page that the user sees" mean if you provide both versions with a URL? As I mentioned in the previous post, Google uses "Googlebot" for web search and "Googlebot-Mobile" for mobile search. To remain within our guidelines, you should serve the same content to Googlebot as a typical desktop user would see, and the same content to Googlebot-Mobile as you would to the browser on a typical mobile device. It's fine if the contents for Googlebot are different from the one for Googlebot-Mobile.
  
  One example of how you could be unintentionally detected for cloaking is if your site returns a message like "Please access from mobile phones" to desktop browsers, but then returns a full mobile version to both crawlers (so Googlebot receives the mobile version). In this case, the page which web search users see (e.g. "Please access from mobile phones") is different from the page which Googlebot crawls (e.g. "Welcome to my site"). Again, we detect cloaking because we want to serve users the same relevant content that Googlebot or Googlebot-Mobile crawled.

  
  Diagram of serving content from your mobile-enabled site
  
  We're working on a daily basis to improve search results and solve problems, but because the relationship between PC and mobile versions of a web site can be nuanced, we appreciate the cooperation of webmasters. Your help will result in more mobile content being indexed by Google, improving the search results provided to users. Thank you for your cooperation in improving the mobile search user experience.
  
  Posted by Jun Mukai, Software Engineer, Mobile Search Team
• 

  Pros and cons of watermarked images

  Posted: November 17th, 2009, 12:40pm CET by Maile Ohye

  Tags:  crawling and indexing  

  Webmaster Level: All
  What's our take on watermarked images for Image Search? It's a complicated topic. I talked with Peter Linsley—my friend at the 'plex, video star, and Product Manager for Image Search—to hear his thoughts.
  Maile: So, Peter... "watermarked images". Can you break it down for us?

  Peter: It's understandable that webmasters find watermarking images beneficial.

  Pros of watermarked images

  • Photographers can claim credit/be recognized for their art.
  • Unknown usage of the image is deterred.

  If search traffic is important to a webmaster, then he/she may also want to consider some of our findings:

  Findings relevant to watermarked images

  • Users prefer large, high-quality images (high-resolution, in-focus).
  • Users are more likely to click on quality thumbnails in search results. Quality pictures (again, high-res and in-focus) often look better at thumbnail size.
  • Distracting features such as loud watermarks, text over the image, and borders are likely to make the image look cluttered when reduced to thumbnail size.

  In summary, if a feature such as watermarking reduces the user-perceived quality of your image or your image's thumbnail, then searchers may select it less often. Preview your images at thumbnail size to get an idea of how the user might perceive it.

  Maile: Ahh, I see: Webmasters concerned with search traffic likely want to balance the positives of watermarking with the preferences of their users -- keeping in mind that sites that use clean images without distracting artifacts tend to be more popular, and that this can also impact rankings. Will Google rank an image differently just because it's watermarked?

  Peter: Nope. The presence of a watermark doesn't itself cause an image to be ranked higher or lower.

  
  Do you have questions or opinions on the topic? Let's chat in the webmaster forum.
  Written by Maile Ohye, Developer Programs Tech Lead
• 

  Help Google index your mobile site

  Posted: November 13th, 2009, 2:48pm CET by Maile Ohye

  Tags:  crawling and indexing  

  (This post was largely translated from our Japanese Webmaster Central Blog.)
  It seems the world is going mobile, with many people using mobile phones on a daily basis, and a large user base searching on Google’s mobile search page. However, as a webmaster, running a mobile site and tapping into the mobile search audience isn't easy. Mobile sites not only use a different format from normal desktop site, but the management methods and expertise required are also quite different. This results in a variety of new challenges. As a mobile search engineer, it's clear to me that while many mobile sites were designed with mobile viewing in mind, they weren’t designed to be search friendly. I'd like to help ensure that your mobile site is also available for users of mobile search.
  Here are troubleshooting tips to help ensure that your site is properly crawled and indexed:
  Verify that your mobile site is indexed by Google
  If your web site doesn't show up in the results of a Google mobile search even using the 'site:' operator, it may be that your site has one or both of the following issues:

  Googlebot may not be able to find your site
  Googlebot, our crawler, must crawl your site before it can be included in our search index. If you just created the site, we may not yet be aware of it. If that's the case, create a Mobile Sitemap and submit it to Google to inform us to the site’s existence. A Mobile Sitemap can be submitted using Google Webmaster Tools, in the same way as with a standard Sitemap.

  Googlebot may not be able to access your site
  Some mobile sites refuse access to anything but mobile phones, making it impossible for Googlebot to access the site, and therefore making the site unsearchable. Our crawler for mobile sites is "Googlebot-Mobile". If you'd like your site crawled, please allow any User-agent including "Googlebot-Mobile" to access your site. You should also be aware that Google may change its User-agent information at any time without notice, so it is not recommended that you check if the User-agent exactly matches "Googlebot-Mobile" (which is the string used at present). Instead, check whether the User-agent header contains the string "Googlebot-Mobile". You can also use DNS Lookups to verify Googlebot.

  
  Verify that Google can recognize your mobile URLs
  Once Googlebot-Mobile crawls your URLs, we then check for whether the URL is viewable on a mobile device. Pages we determine aren't viewable on a mobile phone won't be included in our mobile site index (although they may be included in the regular web index). This determination is based on a variety of factors, one of which is the "DTD (Doc Type Definition)" declaration. Check that your mobile-friendly URLs' DTD declaration is in an appropriate mobile format such as XHTML Mobile or Compact HTML. If it's in a compatible format, the page is eligible for the mobile search index. For more information, see the Mobile Webmaster Guidelines.
  If you have any question regarding mobile site, post your question to our Webmaster Help Forum and webmasters around the world as well as we are happy to help you with your problem.
  Posted by Jun Mukai, Software Engineer, Mobile Search Team
• 

  New personalization features in Google Friend Connect

  Posted: November 4th, 2009, 8:40am CET by Maile Ohye

  Tags:  products and services  

  Webmaster Level: All
  Just a few weeks ago, we made Google Friend Connect a lot easier to use by dramatically simplifying the setup process. Today, we're excited to announce several new features that make it possible for website owners to get to know their users, encourage users to get to know each other, and match their site content (including Google ads) to visitors' interests.
  
  To learn more about these new features, check out the Google Social Web Blog.
  Posted by Mussie Shore, Product Manager, Google Friend Connect
• 

  Using RSS/Atom feeds to discover new URLs

  Posted: October 29th, 2009, 6:50pm CET by Maile Ohye

  Tags:  crawling and indexing  

  Webmaster Level: Intermediate
  
  Google uses numerous sources to find new webpages, from links we find on the web to submitted URLs. We aim to discover new pages quickly so that users can find new content in Google search results soon after they go live. We recently launched a feature that uses RSS and Atom feeds for the discovery of new webpages.
  
  RSS/Atom feeds have been very popular in recent years as a mechanism for content publication. They allow readers to check for new content from publishers. Using feeds for discovery allows us to get these new pages into our index more quickly than traditional crawling methods. We may use many potential sources to access updates from feeds including Reader, notification services, or direct crawls of feeds. Going forward, we might also explore mechanisms such as PubSubHubbub to identify updated items.
  
  In order for us to use your RSS/Atom feeds for discovery, it's important that crawling these files is not disallowed by your robots.txt. To find out if Googlebot can crawl your feeds and find your pages as fast as possible, test your feed URLs with the robots.txt tester in Google Webmaster Tools.
  
  Written by Raymond Lo, Guhan Viswanathan, and Dave Weissman, Crawl and Indexing Team
• 

  Help us make the web better: An update on Rich Snippets

  Posted: October 26th, 2009, 3:00pm CET by Maile Ohye

  Tags:  crawling and indexing  

  Webmaster Level: All
  
  In May this year we announced Rich Snippets which makes it possible to show structured data from your pages on Google's search results.
  
  
  We're convinced that structured data makes the web better, and we've worked hard to expand Rich Snippets to more search results and collect your feedback along the way. If you have review or people/social networking content on your site, it's easier than ever to mark up your content using microformats or RDFa so that Google can better understand it to generate useful Rich Snippets. Here are a few helpful improvements on our end to enable you to mark up your content:
  
  Testing tool. See what Google is able to extract, and preview how microformats or RDFa marked-up pages would look on Google search results. Test your URLs on the Rich Snippets Testing Tool.
  
  
  Google Custom Search users can also use the Rich Snippets Testing Tool to test markup usable in their Custom Search engine.
  
  Better documentation. We've extended our documentation to include a new section containing Tips & Tricks and Frequently Asked Questions. Here we have responded to common points of confusion and provided instructions on how to maximize the chances of getting Rich Snippets for your site.
  
  Extended RDFa support. In addition to the Person RDFa format, we have added support for the corresponding fields from the FOAF and vCard vocabularies for all those of you who asked for it.
  
  Videos. If you have videos on your page, you can now mark up your content to help Google find those videos.
  
  As before, marking up your content does not guarantee that Rich Snippets will be shown for your site. We will continue to expand this feature gradually to ensure a great user experience whenever Rich Snippets are shown in search results.
  
  Written by Kavi Goel, Pravir Gupta, and Othar Hansson
• 

  One million YouTube views!

  Posted: October 21st, 2009, 2:34pm CEST by Michael Wyszomierski

  Tags:  feedback and communication  

  Earlier this year, we launched our very own Webmaster Central channel on YouTube. Just today, we saw our total video views exceed one million! On the road to this milestone, we uploaded 154 videos, for a total of nearly 11 hours of webmaster-focused media. These videos have brought you conference presentations, updates on tools for webmasters, general tips, and of course answers to your "Grab bag" questions for Matt Cutts.
  
  To celebrate our one million views, we're sharing a fun video with you in which Matt Cutts shows us what happened when he lost a bet with his team:
  
  
  
  We're also pleased to announce that we've added captions to all of our videos and plan to do so for our future videos as well. Thank you to everyone who has watched, shared, and commented on our videos. We look forward to the next million views!
  
  Posted by Michael Wyszomierski, Search Quality Team
• 

  Managing your reputation through search results

  Posted: October 15th, 2009, 5:00pm CEST by Susan Moskwa

  Tags:  crawling and indexing  

  (Cross-posted on the Official Google Blog)
  
  A few years ago I couldn't wait to get married. Because I was in love, yeah; but more importantly, so that I could take my husband's name and people would stop getting that ridiculous picture from college as a top result when they searched for me on Google.
  
  After a few years of working here, though, I've learned that you don't have to change your name just because it brings up some embarrassing search results. Below are some tips for "reputation management": influencing how you're perceived online, and what information is available relating to you.
  
  Think twice
  
  The first step in reputation management is preemptive: Think twice before putting your personal information online. Remember that although something might be appropriate for the context in which you're publishing it, search engines can make it very easy to find that information later, out of context, including by people who don't normally visit the site where you originally posted it. Translation: don't assume that just because your mom doesn't read your blog, she'll never see that post about the new tattoo you're hiding from her.
  
  Tackle it at the source
  
  If something you dislike has already been published, the next step is to try to remove it from the site where it's appearing. Rather than immediately contacting Google, it's important to first remove it from the site where it's being published. Google doesn't own the Internet; our search results simply reflect what's already out there on the web. Whether or not the content appears in Google's search results, people are still going to be able to access it — on the original site, through other search engines, through social networking sites, etc. — if you don't remove it from the original site. You need to tackle this at the source.

  • If the content in question is on a site you own, easy — just remove it. It will naturally drop out of search results after we recrawl the page and discover the change.
  • It's also often easy to remove content from sites you don't own if you put it there, such as photos you've uploaded, or content on your profile page.
  • If you can't remove something yourself, you can contact the site's webmaster and ask them to remove the content or the page in question.

  After you or the site's webmaster has removed or edited the page, you can expedite the removal of that content from Google using our URL removal tool.
  
  Proactively publish information
  
  Sometimes, however, you may not be able to get in touch with a site's webmaster, or they may refuse to take down the content in question. For example, if someone posts a negative review of your business on a restaurant review or consumer complaint site, that site might not be willing to remove the review. If you can't get the content removed from the original site, you probably won't be able to completely remove it from Google's search results, either. Instead, you can try to reduce its visibility in the search results by proactively publishing useful, positive information about yourself or your business. If you can get stuff that you want people to see to outperform the stuff you don't want them to see, you'll be able to reduce the amount of harm that that negative or embarrassing content can do to your reputation.
  
  You can publish or encourage positive content in a variety of ways:

  • Create a Google profile. When people search for your name, Google can display a link to your Google profile in our search results and people can click through to see whatever information you choose to publish in your profile.
  • If a customer writes a negative review of your business, you could ask some of your other customers who are happy with your company to give a fuller picture of your business.
  • If a blogger is publishing unflattering photos of you, take some pictures you prefer and publish them in a blog post or two.
  • If a newspaper wrote an article about a court case that put you in a negative light, but which was subsequently ruled in your favor, you can ask them to update the article or publish a follow-up article about your exoneration. (This last one may seem far-fetched, but believe it or not, we've gotten multiple requests from people in this situation.)

  Hope these tips have been helpful! Feel free to stop by our Web Search Forum and share your own advice or stories about how you manage your reputation online.
  
  Posted by Susan Moskwa, Webmaster Trends Analyst
• 

  A proposal for making AJAX crawlable

  Posted: October 7th, 2009, 12:51pm CEST by John Mueller

  Tags:  crawling and indexing  

  Webmaster level: Advanced
  
  Today we're excited to propose a new standard for making AJAX-based websites crawlable. This will benefit webmasters and users by making content from rich and interactive AJAX-based websites universally accessible through search results on any search engine that chooses to take part. We believe that making this content available for crawling and indexing could significantly improve the web.
  
  While AJAX-based websites are popular with users, search engines traditionally are not able to access any of the content on them. The last time we checked, almost 70% of the websites we know about use JavaScript in some form or another. Of course, most of that JavaScript is not AJAX, but the better that search engines could crawl and index AJAX, the more that developers could add richer features to their websites and still show up in search engines.
  
  Some of the goals that we wanted to achieve with this proposal were:
  

  • Minimal changes are required as the website grows
  
  
  • Users and search engines see the same content (no cloaking)
  
  
  • Search engines can send users directly to the AJAX URL (not to a static copy)
  
  
  • Site owners have a way of verifying that their AJAX website is rendered correctly and thus that the crawler has access to all the content

  
  
  Here's how search engines would crawl and index AJAX in our initial proposal:
  

  • Slightly modify the URL fragments for stateful AJAX pages
    Stateful AJAX pages display the same content whenever accessed directly. These are pages that could be referred to in search results. Instead of a URL like http://example.com/page?query#state we would like to propose adding a token to make it possible to recognize these URLs: http://example.com/page?query#[FRAGMENTTOKEN]state . Based on a review of current URLs on the web, we propose using "!" (an exclamation point) as the token for this. The proposed URL that could be shown in search results would then be: http://example.com/page?query#!state.
  
  
  • Use a headless browser that outputs an HTML snapshot on your web server
    The headless browser is used to access the AJAX page and generates HTML code based on the final state in the browser. Only specially tagged URLs are passed to the headless browser for processing. By doing this on the server side, the website owner is in control of the HTML code that is generated and can easily verify that all JavaScript is executed correctly. An example of such a browser is HtmlUnit, an open-sourced "GUI-less browser for Java programs.
  
  
  • Allow search engine crawlers to access these URLs by escaping the state
    As URL fragments are never sent with requests to servers, it's necessary to slightly modify the URL used to access the page. At the same time, this tells the server to use the headless browser to generate HTML code instead of returning a page with JavaScript. Other, existing URLs - such as those used by the user - would be processed normally, bypassing the headless browser. We propose escaping the state information and adding it to the query parameters with a token. Using the previous example, one such URL would be http://example.com/page?query&[QUERYTOKEN]=state . Based on our analysis of current URLs on the web, we propose using "_escaped_fragment_" as the token. The proposed URL would then become http://example.com/page?query&_escaped_fragment_=state .
  
  
  • Show the original URL to users in the search results
    To improve the user experience, it makes sense to refer users directly to the AJAX-based pages. This can be achieved by showing the original URL (such as http://example.com/page?query#!state from our example above) in the search results. Search engines can check that the indexable text returned to Googlebot is the same or a subset of the text that is returned to users.

  
  
  
  (Graphic by Katharina Probst)
  
  In summary, starting with a stateful URL such as
  http://example.com/dictionary.html#AJAX , it could be available to both crawlers and users as
  http://example.com/dictionary.html#!AJAX which could be crawled as
  http://example.com/dictionary.html?_escaped_fragment_=AJAX which in turn would be shown to users and accessed as
  http://example.com/dictionary.html#!AJAX
  
  -->View the presentation
  
  We're currently working on a proposal and a prototype implementation. Feedback is very welcome — please add your comments below or in our Webmaster Help Forum. Thank you for your interest in making the AJAX-based web accessible and useful through search engines!
  
  Proposal by Katharina Probst, Bruce Johnson, Arup Mukherjee, Erik van der Poel and Li Xiao, Google
  Blog post by John Mueller, Webmaster Trends Analyst, Google Zürich
• 

  Reunifying duplicate content on your website

  Posted: October 6th, 2009, 5:14pm CEST by John Mueller

  Tags:  crawling and indexing  

  Handling duplicate content within your own website can be a big challenge. Websites grow; features get added, changed and removed; content comes—content goes. Over time, many websites collect systematic cruft in the form of multiple URLs that return the same contents. Having duplicate content on your website is generally not problematic, though it can make it harder for search engines to crawl and index the content. Also, PageRank and similar information found via incoming links can get diffused across pages we aren't currently recognizing as duplicates, potentially making your preferred version of the page rank lower in Google.
  
  Steps for dealing with duplicate content within your website
  

  1. Recognize duplicate content on your website.
     The first and most important step is to recognize duplicate content on your website. A simple way to do this is to take a unique text snippet from a page and to search for it, limiting the results to pages from your own website by using a site:query in Google. Multiple results for the same content show duplication you can investigate.
     
  
  
  2. Determine your preferred URLs.
     Before fixing duplicate content issues, you'll have to determine your preferred URL structure. Which URL would you prefer to use for that piece of content?
     
  
  
  3. Be consistent within your website.
     Once you've chosen your preferred URLs, make sure to use them in all possible locations within your website (including in your Sitemap file).
     
  
  
  4. Apply 301 permanent redirects where necessary and possible.
     If you can, redirect duplicate URLs to your preferred URLs using a 301 response code. This helps users and search engines find your preferred URLs should they visit the duplicate URLs. If your site is available on several domain names, pick one and use the 301 redirect appropriately from the others, making sure to forward to the right specific page, not just the root of the domain. If you support both www and non-www host names, pick one, use the preferred domain setting in Webmaster Tools, and redirect appropriately.
     
  
  
  5. Implement the rel="canonical" link element on your pages where you can.
     Where 301 redirects are not possible, the rel="canonical" link element can give us a better understanding of your site and of your preferred URLs. The use of this link element is also supported by major search engines such as Ask.com, Bing and Yahoo!.
     
  
  
  6. Use the URL parameter handling tool in Google Webmaster Tools where possible.
     If some or all of your website's duplicate content comes from URLs with query parameters, this tool can help you to notify us of important and irrelevant parameters within your URLs. More information about this tool can be found in our announcement blog post.
     

  
  What about the robots.txt file?
  
  One item which is missing from this list is disallowing crawling of duplicate content with your robots.txt file. We now recommend not blocking access to duplicate content on your website, whether with a robots.txt file or other methods. Instead, use the rel="canonical" link element, the URL parameter handling tool, or 301 redirects. If access to duplicate content is entirely blocked, search engines effectively have to treat those URLs as separate, unique pages since they cannot know that they're actually just different URLs for the same content. A better solution is to allow them to be crawled, but clearly mark them as duplicate using one of our recommended methods. If you allow us to crawl these URLs, Googlebot will learn rules to identify duplicates just by looking at the URL and should largely avoid unnecessary recrawls in any case. In cases where duplicate content still leads to us crawling too much of your website, you can also adjust the crawl rate setting in Webmaster Tools.
  
  We hope these methods will help you to master the duplicate content on your website! Information about duplicate content in general can also be found in our Help Center. Should you have any questions, feel free to join the discussion in our Webmaster Help Forum.
  
  Posted by John Mueller, Webmaster Trends Analyst, Google Zürich
• 

  Translate your website with Google: Expand your audience globally

  Posted: September 30th, 2009, 4:54pm CEST by Susan Moskwa

  Tags:  products and services  

  (This has been cross-posted from the Official Google Blog)
  
  How long would it take to translate all the world's web content into 50 languages? Even if all of the translators in the world worked around the clock, with the current growth rate of content being created online and the sheer amount of data on the web, it would take hundreds of years to make even a small dent.
  
  Today, we're happy to announce a new website translator gadget powered by Google Translate that enables you to make your site's content available in 51 languages. Now, when people visit your page, if their language (as determined by their browser settings) is different than the language of your page, they'll be prompted to automatically translate the page into their own language. If the visitor's language is the same as the language of your page, no translation banner will appear.
  
  
  After clicking the Translate button, the automatic translations are shown directly on your page.
  
  
  It's easy to install — all you have to do is cut and paste a short snippet into your webpage to increase the global reach of your blog or website.
  
  
  Automatic translation is convenient and helps people get a quick gist of the page. However, it's not a perfect substitute for the art of professional translation. Today happens to be International Translation Day, and we'd like to take the opportunity to celebrate the contributions of translators all over the world. These translators play an essential role in enabling global communication, and with the rapid growth and ease of access to digital content, the need for them is greater than ever. We hope that professional translators, along with translation tools such as Google Translator Toolkit and this Translate gadget, will continue to help make the world's content more accessible to everyone.
  
  Posted by Jeff Chin, Product Manager, Google Translate
• 

  Spanish Site Clinic now live

  Posted: September 17th, 2009, 8:09am CEST by Cristina, Equipo de Calidad de Búsqueda

  Tags:  feedback and communication  

  The Google Webmaster Central blog in Spanish has launched a Site Clinic especially for the Spanish-speaking market. We're offering to analyze a series of websites in order to share some best practices with our community using real web sites. The plan is to offer constructive advice on accessibility and improvements that can lead to better visibility in Google's search results.
  During this month, we will be receiving submissions from any legitimate website, but it must be primarily in Spanish. So before you submit your site, please visit the original post and if you want to participate fill out the form as soon as possible, because we will only be selecting 3-5 websites from the first 200 submitted for this Site Clinic, so don't miss out!
  
  Posted by Esperanza, Search Quality Team
• 

  Supporting Facebook Share and RDFa for videos

  Posted: September 14th, 2009, 1:30pm CEST by Michael Wyszomierski

  Tags:  crawling and indexing  

  Have you ever wondered how to increase the chances of your videos appearing in Google's results? Over the last year, the Video Search team has been working hard to improve our index of video on the Web. Today, we're beginning the first in a series of posts to explain some best practices for sites hosting video content.
  We previously talked about the importance of submitting a Video Sitemap or mRSS feed to Google and following Google's webmaster guidelines. However, we wanted to offer webmasters an additional tool, so today we're taking a page from the rich snippets playbook and announcing support for Facebook Share and Yahoo! SearchMonkey RDFa. Both of these markup formats allow you to specify information essential to video indexing, such as a video's title and description, within the HTML of a video page. While we've become smarter at discovering this information on our own, we'd certainly appreciate some hints directly from webmasters. Also, to maximize the chances that we find the markup on your video pages, you should make sure it appears in the HTML without the execution of JavaScript or Flash.
  So, check out Facebook Share and RDFa and help Google find your videos!
  
  Facebook Share:
  

  <meta name="title" content="Baroo? - cute puppies" />
  <meta name="description" content="The cutest canine head tilts on the Internet!" />
  <link rel="image_src" href="http://example.com/thumbnail_preview.jpg" />
  <link rel="video_src" href="http://example.com/video_object.swf?id=12345"/>
  <meta name="video_height" content="296" />
  <meta name="video_width" content="512" />
  <meta name="video_type" content="application/x-shockwave-flash" />

  RDFa (Yahoo! SearchMonkey):
  

  <object width="512" height="296" rel="media:video"
  resource="http://example.com/video_object.swf?id=12345"
  xmlns:media="http://search.yahoo.com/searchmonkey/media/"
  xmlns:dc="http://purl.org/dc/terms/">
  <param name="movie" value="http://example.com/video_object.swf?id=12345" />
  <embed src="http://example.com/video_object.swf?id=12345"
  type="application/x-shockwave-flash" width="512" height="296"></embed>
  <a rel="media:thumbnail" href="http://example.com/thumbnail_preview.jpg" />
  <a rel="dc:license" href="http://example.com/terms_of_service.html" />
  <span property="dc:description" content="Cute Overload defines Baroo? as: Dogspeak for 'Whut the...?'
  Frequently accompanied by the Canine Tilt and/or wrinkled brow for enhanced effect." />
  <span property="media:title" content="Baroo? - cute puppies" />
  <span property="media:width" content="512" />
  <span property="media:height" content="296" />
  <span property="media:type" content="application/x-shockwave-flash" />
  <span property="media:region" content="us" />
  <span property="media:region" content="uk" />
  <span property="media:duration" content="63" />
  </object>
  

  Posted by Michael Cohen, Product Manager, Video Search Team
• 

  New tools for Google Services for Websites

  Posted: August 11th, 2009, 1:05pm CEST by Maile Ohye

  Tags:  products and services  

  Webmaster Level: All
  (A nearly duplicate version :) cross-posted on the Official Google Blog)
  
  Earlier this year, we launched Google Services for Websites, a program that helps partners, e.g., web hoster and access providers, offer useful and powerful tools to their customers. By making services, such as Webmaster Tools, Custom Search, Site Search and AdSense, easily accessible via the hoster control panel, hosters can easily enable these services for their webmasters. The tools help website owners understand search performance, improve user retention and monetize their content — in other words, run more effective websites.
  
  Since we launched the program, several hosting platforms have enhanced their offerings by integrating with the appropriate APIs. Webmasters can configure accounts, submit Sitemaps with Webmaster Tools, create Custom Search Boxes for their sites and monetize their content with AdSense, all with a few clicks at their hoster control panel. More partners are in the process of implementing these enhancements.
  
  We've just added new tools to the suite:

  • Web Elements allows your customers to enhance their websites with the ease of cut-and-paste. Webmasters can provide maps, real-time news, calendars, presentations, spreadsheets and YouTube videos on their sites. With the Conversation Element, websites can create more engagement with their communities. The Custom Search Element provides inline search over your own site (or others you specify) without having to write any code and various options to customize further.
  • Page Speed allows webmasters to measure the performance of their websites. Snappier websites help users find things faster; the recommendations from these latency tools allow hosters and webmasters to optimize website speed. These techniques can help hosters reduce resource use and optimize network bandwidth.
  • The Tips for Hosters page offers a set of tips for hosters for creating a richer website hosting platform. Hosters can improve the convenience and accessibility of tools, while at the same time saving platform costs and earning referral fees. Tips include the use of analytics tools such as Google Analytics to help webmasters understand their traffic and linguistic tools such as Google Translate to help websites reach a broader audience.

  If you're a hoster and would like to participate in the Google Services for Websites program, please apply here. You'll have to integrate with the service APIs before these services can be made available to your customers, so the earlier you start that process, the better.
  
  And if your hosting service doesn't have Google Services for Websites yet, send them to this page. Once they become a partner, you can quickly configure the services you want at your hoster's control panel (without having to come to Google).
  
  As always, we'd love to get feedback on how the program is working for you, and what improvements you'd like to see.
  
  Posted by Rajat Mukherjee, Group Product Manager
• 

  Optimize your crawling & indexing

  Posted: August 10th, 2009, 12:40am CEST by Susan Moskwa

  Tags:  crawling and indexing  

  Webmaster Level: Intermediate to Advanced
  
  Many questions about website architecture, crawling and indexing, and even ranking issues can be boiled down to one central issue: How easy is it for search engines to crawl your site? We've spoken on this topic at a number of recent events, and below you'll find our presentation and some key takeaways on this topic.
  
  
  
  The Internet is a big place; new content is being created all the time. Google has a finite number of resources, so when faced with the nearly-infinite quantity of content that's available online, Googlebot is only able to find and crawl a percentage of that content. Then, of the content we've crawled, we're only able to index a portion.
  
  URLs are like the bridges between your website and a search engine's crawler: crawlers need to be able to find and cross those bridges (i.e., find and crawl your URLs) in order to get to your site's content. If your URLs are complicated or redundant, crawlers are going to spend time tracing and retracing their steps; if your URLs are organized and lead directly to distinct content, crawlers can spend their time accessing your content rather than crawling through empty pages, or crawling the same content over and over via different URLs.
  
  In the slides above you can see some examples of what not to do—real-life examples (though names have been changed to protect the innocent) of homegrown URL hacks and encodings, parameters masquerading as part of the URL path, infinite crawl spaces, and more. You'll also find some recommendations for straightening out that labyrinth of URLs and helping crawlers find more of your content faster, including:
  

  • Remove user-specific details from URLs.
    URL parameters that don't change the content of the page—like session IDs or sort order—can be removed from the URL and put into a cookie. By putting this information in a cookie and 301 redirecting to a "clean" URL, you retain the information and reduce the number of URLs pointing to that same content.
  • Rein in infinite spaces.
    Do you have a calendar that links to an infinite number of past or future dates (each with their own unique URL)? Do you have paginated data that returns a status code of 200 when you add &page=3563 to the URL, even if there aren't that many pages of data? If so, you have an infinite crawl space on your website, and crawlers could be wasting their (and your!) bandwidth trying to crawl it all. Consider these tips for reining in infinite spaces.
  • Disallow actions Googlebot can't perform.
    Using your robots.txt file, you can disallow crawling of login pages, contact forms, shopping carts, and other pages whose sole functionality is something that a crawler can't perform. (Crawlers are notoriously cheap and shy, so they don't usually "Add to cart" or "Contact us.") This lets crawlers spend more of their time crawling content that they can actually do something with.
  • One man, one vote. One URL, one set of content.
    In an ideal world, there's a one-to-one pairing between URL and content: each URL leads to a unique piece of content, and each piece of content can only be accessed via one URL. The closer you can get to this ideal, the more streamlined your site will be for crawling and indexing. If your CMS or current site setup makes this difficult, you can use the rel=canonical element to indicate the preferred URL for a particular piece of content.

  
  If you have further questions about optimizing your site for crawling and indexing, check out some of our previous writing on the subject, or stop by our Help Forum.
  
  Posted by Susan Moskwa, Webmaster Trends Analyst
• 

  Google Friend Connect - Now in 47 new languages

  Posted: July 14th, 2009, 1:05am CEST by Maile Ohye

  Tags:  products and services  

  
  Have you been holding off on using Google Friend Connect because your site isn't in English? Starting today, Friend Connect is available in 47 new languages, including French, Italian, German, Spanish, Chinese, Japanese, Hindi and Portuguese. Now you can easily add social features that match the language of other content on your site.
  
  Most Google-created gadgets (such as the members, comments, and recommendation gadgets) are now available in these new languages. Some developers have also created gadgets that support additional languages and we hope that there will be more to come in the future. To see a list of gadgets available in your language, visit the gadget gallery.
  
  When you add Friend Connect to a new site, it will default to your primary language. But if your site is in another language, simply select it on the site settings tab and Friend Connect will automatically render the gadgets in that language. And if have multiple sites in different languages, you can select a different language for each of your sites.
  
  To learn more or see the full list of languages, check out the Social Web Blog.
  
  Posted by Mendel Chuang, Product Marketing Manager, Google Friend Connect
• 

  Webmaster Central YouTube update for July 6th - 10th

  Posted: July 13th, 2009, 7:55pm CEST by Michael Wyszomierski

  Tags:  feedback and communication  

  Want to see what's new on the Webmaster Central YouTube channel? Check out the answers to the latest Grab Bag questions:

  • How do meta geo tags influence the search results?
  • Will Google find text in images someday?
  • Is Google doing away with use of the meta description?
  • "Query deserves freshness." Fact or fiction?
  • Which is more reliable for searching: Google or Twitter?

  Below is Matt's clarification about Google's use of the meta description tag:
  
  Feel free to leave comments letting us know how you liked the videos, and if you have any specific questions, ask the experts in the Webmaster Help Forum.
  Posted by Michael Wyszomierski, Search Quality Team
• 

  Google Maps gadget: Help customers find their way

  Posted: July 7th, 2009, 6:30pm CEST by Maile Ohye

  Tags:  products and services  

  Webmaster Level: All
  
  With the new directions gadget from Google Maps, any business can offer customized Google Maps directions to their business locations. With the directions gadget, you no longer need to type and update multiple sets of text directions. Let's face it: customers are only looking for directions from their specific location.
  
  Last week, I looked up directions to the hotel in Sacramento that I had booked for the 4th of July weekend. As I had never been to that part of the state before, I was puzzled by the limited directions offered by their website - I had no idea whether I was approaching from the North, South, East or West or where the major highways were. What I needed were step-by-step directions from my exact starting point to the hotel that I could easily print and go.
  
  Google has made this process easy for you. By copying and pasting a single line of code, any website can offer customized door-to-door directions powered by Google Maps to their users.
  
  
  The gadget allows you to pre-fill the "To" field with one or multiple addresses. Customers are able to print their directions with a single click. And for those who prefer not to drive, the gadget also provides walking and public transit directions.
  
  
  If someone enters a vague starting address, they have the option to specify a more exact address, which will then fly into the "From" field.
  
  
  Providing directions from Google Maps is very flexible; in addition to run-of-the-mill addresses, you can give your customers directions to everything from a generic ZIP code to a specific set of latitude-longitude coordinates for any of your locations.
  
  Example destinations:
  123 River St, Woods, MA
  90210
  42.06782° N, 71.756963° W
  
  You can also give your address an alias, or a name that everyone will understand. You can do this by putting the alias in parentheses just after the address:
  
  42.06782° N, 71.756963° W (Grandma's House)
  
  
  Take a look at how Legoland California, Emeril Lagasse, and Harvard University are using the gadget. And then test and create your own directions gadget here. For the many locations outside of the US, the gadget is available in 23 different languages.
  
  Written by Julie Zhou, Product Marketing Manager, Google Maps
• 

  Webmaster Central YouTube update for June 29th - July 3rd

  Posted: July 7th, 2009, 4:03pm CEST by Michael Wyszomierski

  Tags:  feedback and communication  

  Want to see what's new on the Webmaster Central YouTube channel? Check out the answers to the latest Grab Bag questions:

  • How can I optimize my site on a small budget?
  • When will the data in Webmaster Tools be improved?
  • Can my blogroll affect my blog's reputation in Google?
  • Will a link to a page disallowed in robots.txt transfer PageRank?

  Do you run a small business? Check out Matt's tips on how to attract visitors on a small budget:
  
  Feel free to leave comments letting us know how you liked the videos, and if you have any specific questions, ask the experts in the Webmaster Help Forum.
  Posted by Michael Wyszomierski, Search Quality Team
• 

  Webmaster Central YouTube update for June 22nd - 26th

  Posted: June 30th, 2009, 12:22pm CEST by Michael Wyszomierski

  Tags:  feedback and communication  

  Want to know what's new on the Webmaster Central YouTube channel? Here's what we've uploaded in the past week:
  As part of Google's goal to make the web faster, we uploaded several video tips about optimizing the speed of your website. Check out the tutorials page to view the tutorials and associated videos.
  Matt Cutts answered a new question each day from the Grab Bag:

  • What are some best practices for moving to a new CMS?
  • Is over-optimization bad for a website?
  • Will having my software listed in low-quality directories harm my website's ranking?
  • Can the geographic location of a web server affect SEO?
  • Will I be penalized if all of my Sitemap URLs have the same priority?

  And during Adam Lasnik's visit to India, he was interviewed by Webmaster Help Forum guide Jayan Tharayil about issues related to webmasters in India. We have the full three-part interview right here.
  We'll get you started on this batch of videos with Matt's tips for targeting your site to a specific region:
  
  Feel free to leave comments letting us know how you liked the videos, and if you have any specific questions, ask the experts in the Webmaster Help Forum.
  Posted by Michael Wyszomierski, Search Quality Team
• 

  Traffic drops and site architecture issues

  Posted: June 28th, 2009, 5:39am CEST by Luisella Mazza

  Tags:  crawling and indexing  

  Webmaster Level: Intermediate.
  
  We hear lots of questions about site architecture issues and traffic drops, so it was a pleasure to talk about it in greater detail at SMX London and I'd like to highlight some key concepts from my presentation here. First off, let's gain a better understanding of drops in traffic, and then we'll take a look at site design and architecture issues.
  
  Understanding drops in traffic
  
  As you know, fluctuations in search results happen all the time; the web is constantly evolving and so is our index. Improvements in our ability to understand our users' interests and queries also often lead to differences in how our algorithms select and rank pages. We realize, however, that such changes might be confusing and sometimes foster misconceptions, so we'd like to address a couple of these myths head-on.
  
  Myth number 1: Duplicate content causes drops in traffic!
  Webmasters often wonder if the duplicates on their site can have a negative effect on their site's traffic. As mentioned in our guidelines, unless this duplication is intended to manipulate Google and/or users, the duplication is not a violation of our Webmaster Guidelines. The second part of my presentation illustrates in greater detail how to deal with duplicate content using canonicalization.
  
  Myth number 2: Affiliate programs cause drops in traffic!
  Original and compelling content is crucial for a good user experience. If your website participates in affiliate programs, it's essential to consider whether the same content is available in many other places on the web. Affiliate sites with little or no original and compelling content are not likely to rank well in Google search results, but including affiliate links within the context of original and compelling content isn't in itself the sort of thing that leads to traffic drops.
  
  Having reviewed a few of the most common concerns, I'd like to highlight two important sections of the presentation. The first illustrates how malicious attacks -- such as an injection of hidden text and links -- might cause your site to be removed from Google's search results. On a happier note, it also covers how you can use the Google cache and Webmaster Tools to identify this issue. On a related note, if we've found a violation of the Webmaster Guidelines such as the use of hidden text or the presence of malware on your site, you will typically find a note regarding this in your Webmaster Tools Message center.
  You may also find your site's traffic decreased if your users are being redirected to another site...for example, due to a hacker-applied server- or page-level redirection triggered by referrals from search engines. A similar scenario -- but with different results -- is the case in which a hacker has instituted a redirection for crawlers only. While this will cause no immediate drop in traffic since users and their visits are not affected, it might lead to a decrease in pages indexed over time.
  
  
  
  
  
  Site design and architecture issues
  Now that we've seen how malicious changes might affect your site and its traffic, let's examine some design and architecture issues. Specifically, you want to ensure that your site is able to be both effectively crawled and indexed, which is the prerequisite to being shown in our search results. What should you consider?
  
  

  • First off, check that your robots.txt file has the correct status code and is not returning an error.
  • Keep in mind some best practices when moving to a new site and the new "Change of address" feature recently added to Webmaster Tools.
  • Review the settings of the robots.txt file to make sure no pages -- particularly those rewritten and/or dynamic -- are blocked inappropriately.
  • Finally, make good use of the rel="canonical" attribute to reduce the indexing of duplicate content on your domain. The example in the presentation shows how using this attribute helps Google understand that a duplicate can be clustered with the canonical and that the original, or canonical, page should be indexed.

  
  
  
  In conclusion, remember that fluctuations in search results are normal but there are steps that you can take to avoid malicious attacks or design and architecture issues that might cause your site to disappear or fluctuate unpredictably in search results. Start by learning more about attacks by hackers and spammers, make sure everything is running properly at crawling and indexing level by double-checking the HTML suggestions in Webmaster Tools, and finally, test your robots.txt file in case you are accidentally blocking Googlebot. And don't forget about those "robots.txt unreachable" errors!
  
  Written by Luisella Mazza, Search Quality Senior Analyst
• 

  Webmaster Central YouTube update for June 15th - 19th

  Posted: June 22nd, 2009, 5:36pm CEST by Michael Wyszomierski

  Tags:  feedback and communication  

  Want to know what's new on the Webmaster Central YouTube channel? Here's what we've uploaded in the past week:
  
  Maile Ohye gave a webmaster-focused presentation about Product Search.
  
  Matt Cutts answered a new question each day from the Grab Bag:
  

  • Do dates in URLs determine freshness?
  • What impact does "page bloat" have on Google rankings?
  • What types of directories are seen as sources of paid links?
  • Should I include my logo text using the 'alt' attribute or CSS?
  • What's the preferred way to check for links to my site?

  To get you started on watching this latest batch of videos, here's Matt's answer about directories and paid links:
  
  
  
  Feel free to leave comments letting us know how you liked the videos, and if you have any specific questions, ask the experts in the Webmaster Help Forum.
  
  Posted by Michael Wyszomierski, Search Quality Team
• 

  Flash indexing with external resource loading

  Posted: June 19th, 2009, 1:27am CEST by Maile Ohye

  Tags:  crawling and indexing  

  Webmaster Level: All
  We just added external resource loading to our Flash indexing capabilities. This means that when a SWF file loads content from some other file—whether it's text, HTML, XML, another SWF, etc.—we can index this external content too, and associate it with the parent SWF file and any documents that embed it.
  This new capability improves search quality by allowing relevant content contained in external resources to appear in response to users' queries. For example, this result currently comes up in response to the query [2002 VW Transporter 888]:
  
  
  Prior to this launch, this result did not appear, because all of the relevant content is contained in an XML file loaded by a SWF file.
  
  To date, when Google encounters SWF files on the web, we can:

  • Index textual content displayed as a user interacts with the file. We click buttons and enter input, just like a user would.
  • Discover links within Flash files.
  • Load external resources and associate the content with the parent file.
  • Support common JavaScript techniques for embedding Flash, such as SWFObject and SWFObject2.
  • Index sites scripted with AS1 and AS2, even if the ActionScript is obfuscated. Update on June 19, 2009: We index sites with AS3 as well. The ActionScript version isn't particularly relevant in our Indexing process, so we support older versions of AS in addition to the latest.

  If you don't want your SWF file or any of its external resources crawled by search engines, please use an appropriate robots.txt directive.
  
  Written by Janis Stipins, Software Engineer
• 

  Best practices for Product Search

  Posted: June 17th, 2009, 10:27am CEST by Maile Ohye

  Tags:  products and services  

  Webmaster Level: Beginner to Intermediate
  
  If you run an e-commerce site and you'd like your products to be eligible to be shown in Google search results, then check out our "Product Search for Webmasters" video. In addition to the basics on Product Search, I cover:

  • Attributes to include in your feed
  • FAQs
  • Will my products' rankings improve if I include custom attributes in my feed?
  • Do product listings expire after 30 days?
  • How often should I submit my feed?

  
  
  
  More information can be found in the Product Search Help Center.
  
  Written by Maile Ohye, Developer Programs Tech Lead
• 

  Best practices for Product Search

  Posted: June 17th, 2009, 10:27am CEST by Maile Ohye

  Tags:  products and services  

  Webmaster Level: Beginner to Intermediate
  
  If you run an e-commerce site and you'd like your products to be eligible to be shown in Google search results, then check out our "Product Search for Webmasters" video. In addition to the basics on Product Search, I cover:

  • Attributes to include in your feed
  • FAQs
  • Will my products' rankings improve if I include custom attributes in my feed?
  • Do product listings expire after 30 days?
  • How often should I submit my feed?

  
  
  
  More information can be found in the Product Search Help Center.
  
  Written by Maile Ohye, Developer Programs Tech Lead
• 

  Webmaster Central YouTube update for June 8th - 12th

  Posted: June 15th, 2009, 4:22pm CEST by Michael Wyszomierski

  Tags:  feedback and communication  

  Want to know what's new on the Webmaster Central YouTube channel? Here's what we've uploaded in the past week:
  
  Matt Cutts answered a few new questions from the Grab Bag:
  

  • What if a search for my business triggers "Did you mean?"
  • Why do big brands rank so well?
  • Are CSS-based layouts better than tables for SEO?

  Matt also went over a great example of whitehat linkbait:
  
  
  
  And if you've ever thought about hiding text, here's one technique that didn't fool Google:
  
  
  
  Feel free to leave comments letting us know how you liked the videos, and if you have any specific questions, ask the experts in the Webmaster Help Forum.
  
  Posted by Michael Wyszomierski, Search Quality Team
• 

  Webmaster Central YouTube update for June 1st - 5th

  Posted: June 8th, 2009, 4:33pm CEST by Michael Wyszomierski

  Tags:  feedback and communication  

  Want to know what's new on the Webmaster Central YouTube channel? Here's what we've uploaded in the past week:
  
  Matt Cutts answered a new question each day from the Grab Bag:
  

  • Are shortened URLs treated differently than other redirects?
  • How can I make sure Google reaches my deeper pages?
  • What impact does server location have on rankings?
  • Will a "coming soon" page negatively impact my site?
  • How many pages can Google index from a single site?

  In response to questions we've been getting about the videos themselves, we created a behind the scenes look at our setup. And just to throw in some more variety, Matt took a break from SEO questions for a few minutes to show off a fun use of a barcode scanner with Google Book Search. You can read more about it on the Inside Google Book Search blog.
  
  For your instant viewing pleasure, here's the answer to the question about URL shortners:
  
  
  And now a peek behind the scenes:
  
  
  Feel free to leave comments letting us know how you liked the videos, and if you have any specific questions, ask the experts in the Webmaster Help Forum.
  
  Posted by Michael Wyszomierski, Search Quality Team
• 

  SEO Starter Guide now available in 40 languages

  Posted: June 5th, 2009, 9:10am CEST by Maile Ohye

  Tags:  crawling and indexing  

  Webmaster Level: Beginner
  
  Since the release of our Search Engine Optimization Starter Guide last November, we've steadily been adding localized versions for our international users. Today, we're happy to announce that the guide is now available in 40 languages, which covers 98% of the global Internet audience. We hope that webmasters around the world can use the guide to improve their sites' crawlability and indexing in search engines.
  
  We'd also like to thank everyone for their continued feedback on the guide. We're taking notes on what you liked and what you thought was missing for use in future updates and possibly a version for advanced users.
  
  Here's a list of the languages available:

  • Arabic
  • Bulgarian
  • Catalan
  • Chinese (Simplified)
  • Chinese (Traditional)
  • Croatian
  • Czech
  • Danish
  • Dutch
  • English
  • English (GB)
  • Filipino
  • Finnish
  • French
  • German
  • Greek
  • Hebrew
  • Hindi
  • Hungarian
  • Indonesian
  • Italian
  • Japanese
  • Korean
  • Latvian
  • Lithuanian
  • Norwegian
  • Polish
  • Portuguese (BR)
  • Portuguese (PT)
  • Romanian
  • Russian
  • Serbian
  • Slovak
  • Slovenian
  • Spanish
  • Swedish
  • Thai
  • Turkish
  • Ukrainian
  • Vietnamese

  Written by Brandon Falls, Search Quality Team
• 

  Webmaster Central YouTube update for May 26-29

  Posted: June 1st, 2009, 5:18pm CEST by Michael Wyszomierski

  Tags:  feedback and communication  

  In February, we launched the Webmaster Central YouTube channel, and since then we've been busy keeping it fresh with new content. We're not yet uploading 20 hours every minute, but did you know that we've been releasing a new video almost every weekday? You can keep up with the latest updates by subscribing to our channel in YouTube, or by adding our feed to your RSS reader.
  
  If you're more of a weekly digest kind of person, we're here for you too. For every week that we have new videos available, we'll let you know here on the blog. In the videos uploaded in the past week, you'll find answers to these questions from Matt's Grab Bag:
  
  Can product descriptions be considered duplicate content?
  How can new pages get indexed quickly?
  What are your views on PageRank sculpting?
  Will you ever switch to a Mac?
  
  Feel free to leave comments letting us know how you liked the videos, and if you have any specific questions, ask the experts in the Webmaster Help Forum.
  
  Posted by Michael Wyszomierski, Search Quality Team
• 

  Google Webmaster Help Forums in more languages

  Posted: April 30th, 2009, 12:48pm CEST by Reid Yokoyama

  Tags:  feedback and communication  

  Traditionally when we launch a new communication channel, we also give the shareholders a chance to introduce themselves. We did so when we opened webmaster help communities for European webmasters almost two years ago, and also more than a year ago, when we were able to expand and add groups in three more languages. Last December we were very happy to announce the re-launch of two of our Help Forums in a new and cool look and feel.
  
  Today, we're happy to announce that we keep on increasing the global dialogue with webmasters, opening an Arabic and a Czech/Slovak Webmaster Help Forum. Furthermore, we would like to highlight the support we offer in Chinese, Japanese, and Korean. While we've offered support to Chinese webmasters for a little more than a year, the Japanese and Korean forums are only a few weeks old. Keeping with tradition, the guides monitoring our new forums would like to introduce themselves to the global webmaster family:
  
  Arabic Webmaster Help Forum
  
  مرحبا! My name is Adel and I'll be monitoring the Arabic Webmaster Help Forum. I'm originally from Beirut, Lebanon. After finishing computer science studies, I joined Google, some 18 months ago.
  
  Besides working on search quality in Arabic and building a community on our forum, I enjoy traveling and listening to really loud heavy metal music; sometimes I get to do both at the same time! ;-)
  
  I am looking forward to a lot of questions regarding Arabic Google Search and of course ranking and indexing issues on your sites to come. I hope I'll see you there soon!
  - Adel
  
  Czech/Slovak Webmaster Help Forum
  
  Zdravím! I am Marcel, the Google Guide on the Czech/Slovak Webmaster Help Forum. I am originally from Slovakia. After graduating in New Media and Industrial Design, it took me some time and traveling around the globe before moving to Dublin and eventually joining Google some 3 years ago.
  
  Ever since, I've been working in different teams. I was lucky to be part of the AdSense team where I participated in launching AdSense for Content for Czech and Slovak. Since my transition to Search Quality, I enjoy working on improving the quality of our natural search results in Czech, Slovak, and Polish.
  
  Besides my work I have a few more passions, such as listening to live music in Irish pubs, challenging my colleagues in occasional Soulcalibur skirmishes on Playstation and testing burger places all over the world :-) If you want to discuss any of these topics or maybe something about your sites, please join the community. I am looking forward to meeting you there :-)
  - Marcel
  Chinese Webmaster Help Forum
  
  你好! Hi from the Chinese Webmaster Help Forum team! The Chinese Webmaster Help Forum has received great support from webmasters since its launch in March 2008. In March 2009, the Chinese Webmaster Help Forum moved to a new system with many more user-friendly features for better information sharing. It has become a good platform for webmasters to share their knowledge of Google search and Webmaster Tools and to communicate with Google.
  
  The Chinese Forum now has 6 Google Guides: Xiang (降龙十巴掌), Eric (趙錢孫李), Marina (小馬過河), Chris (城镇), Hyson (草帽路飞), and Fa (法人戴表). We are from many different provinces of mainland China. When not spending time in the forum, we enjoy playing ping-pong and foosball in the office. A few of us are huge video game fans. You may learn more about us when you participate in the forum :)
  
  A big thank you to everyone who has taken part in forum discussions! We hope to see both familiar faces and newcomers join in the Chinese Webmaster Help Forum!
  - Xiang (降龙十巴掌), Eric (趙錢孫李), Marina (小馬過河), Chris (城镇), Hyson (草帽路飞), and Fa (法人戴表)
  
  Japanese Webmaster Help Forum
  
  こんにちは! Hello from the Japanese Webmaster Help Forum team! Our names are Nao ( なお ), Kaede ( 楓 ), Haru ( ハル ), and Kyotaro. We are the four guides working in Google Search Quality for Japanese. We've just launched our forum on March 6th.
  
  All of us were born in Japan and grew up here. Nao has also lived in Greece, the Netherlands, and New York. Haru is from the west side of Japan, which is known for its talkative culture and traditional Japanese comedy. Maybe you will read Haru's unique communication on our forum :)
  
  As for our interests, we love eating and drinking! Between posting on the forum, we enjoy Google's excellent lunches and sweets a lot. After working, of course, we sometimes go out for a drink with our team members :) Kaede knows all the nice bars in Tokyo.
  
  Nao and Kyotaro love Sumo wrestling. We've watched two tournaments this year with Googlers from other locations. Haru, of course, loves watching comedies!
  
  We are really excited and happy to see many users joining our forum and sharing tips with each other. Looking forward to seeing you there!
  - Nao ( なお ), Kaede ( 楓 ), Haru ( ハル ), and Kyotaro
  
  Korean Webmaster Help Forum
  
  안녕하세요! Hello everyone, my name is Joowon and I work in Google Search Quality for Korean. I was born in Germany and lived in Korea for a few years before moving to Hawaii, California and New York to attend high school and college. After all that traveling, I'm only fluent in Korean and English, with a bit of proficiency in Japanese. Some of the interests I've developed over the years are design, wine, cooking, yoga, and sustainability issues.
  
  Currently I'm back in Seoul and enjoying the dynamic atmosphere here, with lots of interesting people and great food. The Korean Webmaster Help Forum was launched only a few weeks ago, and I'm very much looking forward to talking to all of you. See you in the forum!
  - Joowon
  
  Hello world! ;) I am Andrew and I am part of the Search Quality team in Seoul. I grew up in a port city in the southern part of Korea. Ironically, I don't eat seafood because it looks scary to me :( Many of my friends and colleagues love to make jokes about that, but I still don't eat any seafood yet. Playing drums, traveling and photography are my main interests. Currently I'm a drummer of "Spring Fingers", the first band of Google Korea, and we'll have our first concert at the end of April!
  
  I love playing around with web technologies/APIs and find it very exciting to exchange information and ideas on the web. The Korean Webmaster Help Forum was recently launched and I hope to see you there!
  - Andrew
  
  If you're curious about our Webmaster Help Forums in other languages, please feel free to peak in. Here's a list of our currently monitored Webmaster Help Forums: Arabic, Chinese, Czech/Slovak, Dutch, Danish, English, Finnish, French, German, Hebrew, Hungarian, Italian, Japanese, Korean, Polish, Portuguese, Russian, Spanish, Swedish, and Turkish.
  
  Written by Joowon Ahn, Andrew Baek, Adel Saud, Kaspar Szymanski, Fa Wang, Reid Yokoyama, and Marcel Zavacky, Search Quality Team
• 

  Get up-to-date on Image Search

  Posted: March 10th, 2009, 6:13pm CET by Maile Ohye

  Tags:  crawling and indexing  

  Webmaster Level: All
  
  Recently at SMX West, I gave an Image Search presentation that I'd like to share with our broader webmaster community. The goal of the presentation was to provide insights into how image search is used, how it works, and how webmasters can optimize their pages for image searchers.
  
  You'll see more information about:

  • Some background on the reach of Image Search
  • Interesting findings on the behavior of image searchers
  • Our efforts at handling multiple image referrers
  • How to best feature images (image quality and placement, relevant surrounding text, etc.)

  Take a look and let us know your thoughts in the comments. We'd love to hear from you.
  
  
  
  Posted by Peter Linsley, Product Manager
• 

  Using stats from site: and Sitemap details

  Posted: March 4th, 2009, 5:40pm CET by Chark

  Tags:  crawling and indexing  

  Webmaster Level: Beginner to Intermediate
  
  Every now and then in the webmaster blogosphere and forums, this issue comes up: when a webmaster performs a [site:example.com] query on their website, the number of indexed results differs from what is displayed in their Sitemaps report in Webmaster Tools. Such a discrepancy may smell like a bug, but it's actually by design. Your Sitemap report only reflects the URLs you've submitted in your Sitemap file. The site operator, on the other hand, takes into account whatever Google has crawled, which may include URLs not included in your Sitemap, such as newly added URLs or other URLs discovered via links.
  
  Think of the site operator as a quick diagnosis of the general health of your site in Google's index. Site operator results can show you:
  

  • a rough estimate of how many pages have been indexed
  • one indication of if your site has been hacked
  • if you have duplicate titles or snippets

  Here is an example query using the site operator:
  
  
  
  Your Sitemap report provides more granular statistics about the URLs you submitted, such as the number of indexed URLs vs. the number submitted for crawling, and Sitemap-specific warnings or errors that may have occurred when Google tried to access your URLs.
  
  
  
  Feel free to check out our Help Center for more on the site: operator and Sitemaps. If you have further questions or issues, please post to our Webmaster Help Forum, where experienced webmasters and Googlers are happy to help.
  
  Posted by Charlene Perez, Search Quality Team
• 

  Canonical Link Element: presentation from SMX West

  Posted: February 25th, 2009, 7:50am CET by Michael Wyszomierski

  Tags:  crawling and indexing  

  A little while ago, Google and other search engines announced support for a canonical link element that can help site owners with duplicate content issues. I recreated my presentation from SMX West and you can watch it below:
  
  
  
  You can access the slides directly or follow along here:
  
  
  
  By the way, Ask just announced that they will support the canonical link element. Read all about it in the Ask.com blog entry.
  
  Thanks again to Wysz for turning this into a great video.
  In fact, you might not have seen it, but we recently created a webmaster videos channel on YouTube. If you're interested, you can watch the new webmaster channel. If you subscribe to that channel, you'll always find out about new webmaster-related videos from Google.
  Posted by Matt Cutts, Search Quality Team
• 

  Introducing the Google Webmaster Central YouTube Channel

  Posted: February 23rd, 2009, 1:30pm CET by Michael Wyszomierski

  Tags:  feedback and communication  

  In his State of the Index presentation, Matt Cutts said that one of the things to look for from Google in 2009 is continued communication with webmasters. On the Webmaster Central team, we've found that using video is a great way to reach people. We've shown step-by-step instructions on how to use features of Webmaster Tools, shared our presentations with folks who were unable to make it to conferences, and even taken you through a day in the life of our very own Maile Ohye as she meets with many Googlers involved in webmaster support.
  
  We plan on releasing more videos like these in the future, so we've opened up our own channel on YouTube to host webmaster-related videos. Our first video is already up, and we'll have more to share with you soon. If you want to be the first to know when we release something new, you can subscribe to us using your YouTube account, or grab this RSS feed if you'd like to keep track in your feed reader. Please let us know how you like the channel, and use the comments in this post to share your ideas for future videos.
  
  And while we'll all do our best to make sure Matt Cutts understands that Rick Rolling is so last year, be careful where you click on April 1st.
  
  Posted by Michael Wyszomierski, Search Quality Team
• 

  State of the Index: my presentation from PubCon Vegas

  Posted: February 18th, 2009, 5:15pm CET by Michael Wyszomierski

  Tags:  feedback and communication  

  It seems like people enjoyed when I recreated my Virtual Blight talk from the Web 2.0 Summit late last year, so we decided to post another video. This video recreates the "State of the Index" talk that I did at PubCon in Las Vegas late last year as well.
  
  Here's the video of the presentation:
  
  
  and if you'd like to follow along, here are the slides:
  
  
  
  You can also access the presentation directly. Thanks again to Wysz for recording this video and splicing the slides into the video.
  Posted by Matt Cutts, Search Quality Team
• 

  Specify your canonical

  Posted: February 12th, 2009, 1:30pm CET by Maile Ohye

  Tags:  crawling and indexing  

  Carpe diem on any duplicate content worries: we now support a format that allows you to publicly specify your preferred version of a URL. If your site has identical or vastly similar content that's accessible through multiple URLs, this format provides you with more control over the URL returned in search results. It also helps to make sure that properties such as link popularity are consolidated to your preferred version.
  
  Let's take our old example of a site selling Swedish fish. Imagine that your preferred version of the URL and its content looks like this:
  
  http://www.example.com/product.php?item=swedish-fish
  
  
  However, users (and Googlebot) can access Swedish fish through multiple (not as simple) URLs. Even if the key information on these URLs is the same as your preferred version, they may show slight content variations due to things like sort parameters or category navigation:
  
  http://www.example.com/product.php?item=swedish-fish&category=gummy-candy
  Or they have completely identical content, but with different URLs due to things such as a tracking parameters or a session ID:
  
  http://www.example.com/product.php?item=swedish-fish&trackingid=1234&sessionid=5678
  Now, you can simply add this <link> tag to specify your preferred version:
  
  <link rel="canonical" href="http://www.example.com/product.php?item=swedish-fish" />
  
  inside the <head> section of the duplicate content URLs:
  
  http://www.example.com/product.php?item=swedish-fish&category=gummy-candy
  http://www.example.com/product.php?item=swedish-fish&trackingid=1234&sessionid=5678
  
  and Google will understand that the duplicates all refer to the canonical URL: http://www.example.com/product.php?item=swedish-fish. Additional URL properties, like PageRank and related signals, are transferred as well.
  
  This standard can be adopted by any search engine when crawling and indexing your site.
  
  Of course you may have more questions. Joachim Kupke, an engineer from our Indexing Team, is here to provide us with the answers:
  
  Is rel="canonical" a hint or a directive?
  It's a hint that we honor strongly. We'll take your preference into account, in conjunction with other signals, when calculating the most relevant page to display in search results.
  
  Can I use a relative path to specify the canonical, such as <link rel="canonical" href="product.php?item=swedish-fish" />?
  Yes, relative paths are recognized as expected with the <link> tag. Also, if you include a <base> link in your document, relative paths will resolve according to the base URL.
  
  Is it okay if the canonical is not an exact duplicate of the content?
  We allow slight differences, e.g., in the sort order of a table of products. We also recognize that we may crawl the canonical and the duplicate pages at different points in time, so we may occasionally see different versions of your content. All of that is okay with us.
  
  What if the rel="canonical" returns a 404?
  We'll continue to index your content and use a heuristic to find a canonical, but we recommend that you specify existent URLs as canonicals.
  
  What if the rel="canonical" hasn't yet been indexed?
  Like all public content on the web, we strive to discover and crawl a designated canonical URL quickly. As soon as we index it, we'll immediately reconsider the rel="canonical" hint.
  
  Can rel="canonical" be a redirect?
  Yes, you can specify a URL that redirects as a canonical URL. Google will then process the redirect as usual and try to index it.
  
  What if I have contradictory rel="canonical" designations?
  Our algorithm is lenient: We can follow canonical chains, but we strongly recommend that you update links to point to a single canonical page to ensure optimal canonicalization results.
  
  Can this link tag be used to suggest a canonical URL on a completely different domain?
  No. To migrate to a completely different domain, permanent (301) redirects are more appropriate. Google currently will take canonicalization suggestions into account across subdomains (or within a domain), but not across domains. So site owners can suggest www.example.com vs. example.com vs. help.example.com, but not example.com vs. example-widgets.com.
  
  Sounds great—can I see a live example?
  Yes, wikia.com helped us as a trusted tester. For example, you'll notice that the source code on the URL http://starwars.wikia.com/wiki/Nelvana_Limited specifies its rel="canonical" as: http://starwars.wikia.com/wiki/Nelvana.
  
  The two URLs are nearly identical to each other, except that Nelvana_Limited, the first URL, contains a brief message near its heading. It's a good example of using this feature. With rel="canonical", properties of the two URLs are consolidated in our index and search results display wikia.com's intended version.
  
  Feel free to ask additional questions in our comments below. And if you're unable to implement a canonical designation link, no worries; we'll still do our best to select a preferred version of your duplicate content URLs, and transfer linking properties, just as we did before.
  
  Updated: This link-tag is currently also supported by Ask.com, Microsoft Live Search and Yahoo!.
  
  Written by Joachim Kupke, Senior Software Engineer, and Maile Ohye, Developer Programs Tech Lead
• 

  Help us help you

  Posted: February 11th, 2009, 3:32pm CET by Matt Dougherty

  Tags:  feedback and communication  

  You're a webmaster, right? Well, we love webmasters! To ensure we give you the best support possible, we've set up a survey to get your thoughts on Webmaster Central and our related support efforts. If you have a few extra minutes this week, please click here to give us your honest feedback.
  
  Thanks from all of us on the Webmaster Central Team.
  
  Posted by Matt Dougherty, Search Quality
  
• 

  A new Google Sitemap Generator for your website

  Posted: January 13th, 2009, 6:12am CET by John Mueller

  Tags:  crawling and indexing  

  It's been well over three years since we initially announced the Python Sitemap generator in June 2005. In this time, we've seen lots of people create great third-party Sitemap generators to help webmasters create better Sitemap files. While most Sitemap generators either crawl websites or list the files on a server, we have created a different kind of Sitemap generator that uses several ways to find URLs on your website and then allows you to automatically create and maintain different kinds of Sitemap files.
  
  
  
  About Google Sitemap Generator
  Our new open-source Google Sitemap Generator finds new and modified URLs based on your webserver's traffic, its log files, or the files found on the server. By combining these methods, Google Sitemap Generator can be very fast in finding these URLs and calculating relevant metadata, thereby making your Sitemap files as effective as possible. Once Google Sitemap Generator has collected the URLs, it can create the following Sitemap files for you:
  

  • XML Sitemaps for Web Search according to the sitemaps.org standard
  
  
  • Mobile Sitemaps for mobile-friendly websites
  
  
  • Code Search Sitemaps for source code that you make available to users

  
  In addition, Google Sitemap Generator can send a ping to Google Blog Search for all of your new or modified URLs. You can optionally include the URLs of the Sitemap files in your robots.txt file as well as "ping" the other search engines that support the sitemaps.org standard.
  
  Sending the URLs to the right Sitemap files is simple thanks to the web-based administration console. This console gives you access to various features that make administration a piece of cake while maintaining a high level of security by default.
  
  Getting started
  Google Sitemap Generator is a server plug-in that can be installed on both Linux/Apache and Microsoft IIS Windows-based servers. As with other server-side plug-ins, you will need to have administrative access to the server to install it. You can find detailed information for the installation in the Google Sitemap Generator documentation.
  
  We're excited to release Google Sitemap Generator with the source code and hope that this will encourage more web hosters to include this or similar tools in their hosting packages!
  
  Do you have any questions? Feel free to drop by our Help Group for Google Sitemap Generator or ask general Sitemaps question in our Webmaster Help Forum.
  
  Posted by John Mueller, Webmaster Trends Analyst, Google Zürich
• 

  Reintroducing your English Webmaster Help Google Guides

  Posted: December 7th, 2008, 9:30pm CET by Reid Yokoyama

  Tags:  feedback and communication  

  When we announced our latest round of Bionic Posters in the old Help Group, an astute webmaster noted that we had never fully introduced ourselves to the webmaster community. With our announcement of a new English Webmaster Help forum, I'm happy to (re)introduce you all to the Google Guides from around the world who will be hanging out in the English forum:
  
  Guides in Mountain View, California (from left to right): Adam, Maile, Matt Cutts, Chris, Wysz, Matt, Chark, and Adi.
  
  Also in Mountain View: Evan, Jessica, and Nate.
  
  Guides in Kirkland, Washington: Susan, Riona, and Jonathan.
  
  Guides in Zürich, Switzerland: John Mueller and Balázs.
  
  Guides in Hyderabad, India: Koti, Reid (visiting from Mountain View), and Jayan.
  
  Guide in Ottowa Ottawa :), Ontario: Oliver. All of us look forward to seeing you around the forum! Our Google Guides in non-English Webmaster Help Groups introduce themselves here.
  
  Written by Reid Yokoyama, Search Quality
• 

  Friend Connect now available in beta to everyone

  Posted: December 5th, 2008, 4:38am CET by Maile Ohye

  Tags:  products and services  

  If you've been looking for a way to grow traffic and make your site more interactive, check out Google Friend Connect -- now in beta and available to all webmasters. Remember that with Friend Connect, you can easily add social features to your site by just copying and pasting a few snippets of code, no programming necessary! Your visitors will be able to join your site, create or link to a profile, interact with other visitors, and invite friends to visit your site. Best of all, your visitors won't be required to create yet another username and password -- Friend Connect lets them sign in using an existing Google, Yahoo, AOL, or OpenID account. To learn more, watch the video below:
  
  
  So for any of you who were interested in Friend Connect after our first announcement, and also to all of the newer readers out there, go ahead and give Friend Connect a try.
  
  Written by Mendel Chuang, Product Marketing Manager
• 

  A new look for our Webmaster Help Group

  Posted: December 4th, 2008, 9:15am CET by Susan Moskwa

  Tags:  products and services  

  Googlers strongly believe in dogfooding our own products. We manage our work schedules with Google Calendar, publish our blogs on Blogger, and store scads of documentation on Google Sites. So, ever since we launched our first Webmaster Help Group, we've been using Google Groups to facilitate conversations about Webmaster Tools and web search issues.

  Today, however, I'm thrilled to announce that our English and Polish Help Groups are getting a makeover. And the changes are more than just skin-deep. Our new Help Forums should make it easier for you to find answers, share resources with others, and have your participation acknowledged.

  You can read more about the changes on the Official Google Blog, and then check it out for yourself: English, Polish.

  Q: What will happen to the old English and Polish Help Groups?
  A: While our old groups are now closed to new posts, they will still be available in read-only mode in case you want to reference any of your favorite posts from the good old days. Many of the most frequently-asked questions (and answers!) have already been transferred to our new Help Forums.

  Q: If I was a member of the old group, will I automatically be a member of the new forum?
  A: We won't be "transferring" membership from the old groups to the new, so even if you were a member of our Google Groups forum, you'll still need to join the new forum in order to participate. Nicknames and user profiles are also managed separately, so you're welcome to recreate your Google Groups profile in our new forum, or reinvent yourself.

  Q: What about the Webmaster Help Groups in other languages?
  A: They'll be moving to the new Help Forum format in 2009. Specific dates will be announced in each of the groups as they get closer to their moving date.

  Feel free to post any other questions about the new Help Forums in the comments below.

  
  Written by Susan Moskwa, Webmaster Trends Analyst
• 

  Date with Googlebot, Part II: HTTP status codes and If-Modified-Since

  Posted: November 27th, 2008, 1:12am CET by Gerald Hibbert

  Tags:  crawling and indexing  

  Our date with Googlebot was so wonderful, but it's hard to tell if we, the websites, said the right thing. We returned 301 permanent redirect, but should we have responded with 302 temporary redirect (so he knows we're playing hard to get)? If we sent a few new 404s, will he ever call our site again? Should we support the header "If-Modified-Since?" These questions can be confusing, just like young love. So without further ado, let's ask the expert, Googlebot, and find out how he judged our response (code).
  
  
  Supporting the "If-Modified-Since" header and returning 304 can save bandwidth.
  
  -----------
  Dearest Googlebot,
    Recently, I did some spring cleaning on my site and deleted a couple of old, orphaned pages. They now return the 404 "Page not found" code. Is this ok, or have I confused you?
  Frankie O'Fore
  
  Dear Frankie,
    404s are the standard way of telling me that a page no longer exists. I won't be upset—it's normal that old pages are pruned from websites, or updated to fresher content. Most websites will show a handful of 404s in the Crawl Diagnostics over at Webmaster Tools. It's really not a big deal. As long as you have good site architecture with links to all your indexable content, I'll be happy, because it means I can find everything I need.
  
    But don't forget, it's not just me who comes to your website—there may be humans seeing these pages too. If you've only got a very simple '404 page not found' message, visitors who aren't as savvy could be baffled. There are lots of ways to make your 404 page more friendly; a quick one is our 404 widget over at Webmaster Tools, which will help direct people to content which does exist. For more information, you can read the blog post. Most web hosting companies, big and small, will let you customise your 404 page (and other return codes too).
  
  Love and kisses,
  Googlebot
  
  
  Hey Googlebot,
    I was just reading your reply to Frankie above, and it raised a couple of questions.
  What if I have someone linking to a page that no longer exists? How can I make sure my visitors still find what they're after? Also, what if I just move some pages around? I'd like to better organise my site, but I'm worried you'll get confused. How can I help you?
  Yours hopefully,
  Little Jimmy
  
  
  Hello Jimmy,
     Let's pretend there are no anachronisms in your letter, and get to the meat of the matter. Firstly, let's look at links coming from other sites. Obviously, these can be a great source of traffic, and you don't want visitors presented with an unfriendly 'Page not found' message. So, you can harness the power of the mighty redirect.
  
     There are two types of redirect—301 and 302. Actually, there are lots more, but these are the two we'll concern ourselves with now. Just like 404, 301 and 302 are different types of responses codes you can send to users and search engine crawlers. They're both redirects, but a 301 is permanent and a 302 is temporary. A 301 redirect tells me that whatever this page used to be, now it lives somewhere else. This is perfect for when you're re-organising your site, and also helps with links from offsite. Whenever I see a 301, I'll update all references to that old page with the new one you've told me about. Isn't that easy?
  
     If you don't know where to begin with redirects, let me get you started. It depends on your webserver, but here are some searches that may be helpful:
  Apache: [www.google.com]
  IIS: [www.google.com]
  You can also check your manual, or the README files that came with your server.
  
     As an alternative to a redirect, you can email the webmaster of the site linking to you and ask them to update their link. Not sure what sites are linking to you? Don't despair - my human co-workers have made that easy to figure out. In the "Links" portion of Webmaster Tools, you can enter a specific URL on your site to determine who's linking to it.
  
    My human co-workers also just released a tool which shows URLs linking to non-existent pages on your site. You can read more about that here.
  
  Yours informationally,
  Googlebot
  
  
  Darling Googlebot,
     I have a problem—I live in a very dynamic part of the web, and I keep changing my mind about things. When you ask me questions, I never respond the same way twice—my top threads change every hour, and I get new content all the time! You seem like a straightforward guy who wants straightforward answers. How can I tell you when things change without confusing you?
  Temp O'Rary
  
  
  Dear Temp,
     I just told little Jimmy that 301's are the best way to tell a Googlebot about your new address, but what you're looking for is a 302.
     Once you're indexed, it's the polite way to tell your visitors that your address is still the right one, but that the content can temporarily be found elsewhere. In these situations, a 302 (or the rarer '307 Temporary Redirect') would be better. For example, orkut redirects from http://orkut.com to http://google.com/accounts/login?service=orkut, which isn't a page that humans would find particularly useful when searching for Orkut***.
  It's on a different domain, for starters. So, a 302 has been used to tell me that all the content and linking properties of the URL shouldn't be updated to the target - it's just a temporary page.
  
    That's why when you search for orkut, you see orkut.com and not that longer URL.
  
    Remember: simple communication is the key to any relationship.
  
  Your friend,
  Googlebot
  
  *** Please note, I simplified the URL to make it easier to read. It's actually much more complex than that.
  
  Captain Googlebot,
     I am the kind of site who likes to reinvent herself. I noticed that the links to me on my friends' sites are all to URLs I got rid of several redesigns ago! I had set up 301s to my new URLs for those pages, but after that I 301'ed the newer URLs to my next version. Now I'm afraid that if you follow their directions when you come to crawl, you'll end up following a string of 301s so long that by the end you won't come calling any more.
  Ethel Binky
  
  
  Dear Ethel,
     It sounds like you have set up some URLs that redirect to more redirects to... well, goodness! In small amounts, these "repeat redirects" are understandable, but it may be worth considering why you're using them in the first place. If you remove the 301s in the middle and send me straight to the final destination on all of them, you'll save the both of us a bunch of time and HTTP requests. But don't just think of us. Other people get tired of seeing that same old 'contacting.... loading ... contacting...' game in their status bar.
  
     Put yourself in their shoes—if your string of redirects starts to look rather long, users might fear that you have set them off into an infinite loop! Bots and humans alike can get scared by that kind of "eternal commitment." Instead, try to get rid of those chained redirects, or at least keep 'em short. Think of the humans!
  
  Yours thoughtfully,
  Googlebot
  
  Dear Googlebot,
     I know you must like me—you even ask me for unmodified files, like my college thesis that hasn't changed in 10 years. It's starting to be a real hassle! Is there anything I can do to prevent your taking up my lovely bandwidth?
  
  Janet Crinklenose
  
  
  Janet, Janet, Janet,
     It sounds like you might want to learn a new phrase—'304 Not Modified'. If I've seen a URL before, I insert an 'If-Modified-Since' in my request's header. This line also includes an HTTP-formatted date string. If you don't want to send me yet another copy of that file, stand up for yourself and send back a normal HTTP header with the status '304 Not Modified'! I like information, and this qualifies too. When you do that, there's no need to send me a copy of the file—which means you don't waste your bandwidth, and I don't feel like you're palming me off with the same old stuff.
  
     You'll probably notice that a lot of browsers and proxies will say 'If-Modified-Since' in their headers, too. You can be well on your way to curbing that pesky bandwidth bill.
  
  Now go out there and save some bandwidth!
  Good ol' Googlebot
  -----------
  
  Googlebot has been so helpful! Now we know how to best respond to users and search engines. The next time we get together, though, it's time to sit down for a good long heart-to-heart with the guy (Date with Googlebot: Part III, is coming soon!).
  
  As told to Chewy [Trewhella], Developer Advocate
  
  UPDATE: Added a missing link. Thanks to Boris for pointing that out.
• 

  Google's SEO Starter Guide

  Posted: November 12th, 2008, 12:55pm CET by Maile Ohye

  Tags:  crawling and indexing  

  Webmasters often ask us at conferences or in the Webmaster Help Group, "What are some simple ways that I can improve my website's performance in Google?" There are lots of possible answers to this question, and a wealth of search engine optimization information on the web, so much that it can be intimidating for newer webmasters or those unfamiliar with the topic. We thought it'd be useful to create a compact guide that lists some best practices that teams within Google and external webmasters alike can follow that could improve their sites' crawlability and indexing.
  
  Our Search Engine Optimization Starter Guide covers around a dozen common areas that webmasters might consider optimizing. We felt that these areas (like improving title and description meta tags, URL structure, site navigation, content creation, anchor text, and more) would apply to webmasters of all experience levels and sites of all sizes and types. Throughout the guide, we also worked in many illustrations, pitfalls to avoid, and links to other resources that help expand our explanation of the topics. We plan on updating the guide at regular intervals with new optimization suggestions and to keep the technical advice current.
  
  So, the next time we get the question, "I'm new to SEO, how do I improve my site?", we can say, "Well, here's a list of best practices that we use inside Google that you might want to check out."
  
  Update: The guide is now available in Chinese, German, Portuguese (Brazil, Portugal), Spanish, and Turkish.
  
  Written by Brandon Falls, Search Quality Team
• 

  Helping you break the language barrier

  Posted: October 14th, 2008, 1:03pm CEST by Maile Ohye

  Tags:  products and services  

  When webmasters put content out on the web it's there for the world to see. Unfortunately, most content on the web is only published in a single language, understandable by only a fraction of the world's population.
  
  In a continued effort to make the world's information universally accessible, Google Translate has a number of tools for you to automatically translate your content into the languages of the world.
  
  
  Users may already be translating your webpage using Google Translate, but you can make it even easier by including our "Translate My Page" gadget, available at [translate.google.com].
  
  The gadget will be rendered in the user's language, so if they come to your page and can't understand anything else, they'll be able to read the gadget, and translate your page into their language.
  
  Sometimes there may be some content on your page that you don't want us to translate. You can now add class=notranslate to any HTML element to prevent that element from being translated. For example, you may want to do something like:
  

  Email us at <span class="notranslate">sales at mydomain dot com</span>

  And if you have an entire page that should not be translated, you can add:
  

  <meta name="google" value="notranslate">

  to the <head> of your page and we won't translate any of the content on that page.
  
  Update on 12/15/2008: We also support:

  <meta name="google" content="notranslate">

  Thanks to chaoskaizer for pointing this out in the comments. :)
  
  Lastly, if you want to do some fancier automatic translation integrated directly into your page, check out the AJAX Language API we launched last March.
  
  With these tools we hope you can more easily make your content available in all the languages we support, including Arabic, Bulgarian, Catalan, Chinese, Croatian, Czech, Danish, Dutch, English, Filipino, Finnish, French, German, Greek, Hebrew, Hindi, Indonesian, Italian, Japanese, Korean, Latvian, Lithuanian, Norwegian, Polish, Portuguese, Romanian, Russian, Serbian, Slovak, Slovenian, Spanish, Swedish, Ukrainian, and Vietnamese.
  
  Written by Josh Estelle, Google Translate
• 

  Webmaster Tools shows Crawl error sources

  Posted: October 13th, 2008, 4:03pm CEST by Jonathan Simon

  Tags:  crawling and indexing  

  Ever since we released the crawl errors feature in Webmaster Tools, webmasters have asked for the sources of the URLs causing the errors. Well, we're listening! We know it was difficult for those of you who wanted to identify the cause of a particular "Not found" error, in order to prevent it in the future or even to request a correction, without knowing the source URL. Now, Crawl error sources makes the process of tracking down the causes of "Not found" errors a piece of cake. This helps you improve the user experience on your site and gives you a jump start for links week (check out our updated post on "Good times with inbound links" to get the scoop).
  
  In our "Not Found" and "Errors for URLs in Sitemaps" reports, we've added the "Linked From" column. For every error in these reports, the "Linked From" column now lists the number of pages that link to a specific "Not found" URL.
  
  
  
  Clicking on an item in the "Linked From" column opens a separate dialog box which lists each page that linked to this URL along with the date it was discovered. The source URL for the 404 can be within or external to your site.
  
  
  
  
  
  For those of you who just want the data, we've also added the ability to download all your crawl error sources at once. Just click the "Download all sources of errors on this site" link to download all your site's crawl error sources.
  
  
  
  Again, if we report crawl errors for your website, you can use crawl error sources to quickly determine if the cause is from your site or someone else's. You'll have the information you need to contact them to get it fixed, and if needed, you can still put in place redirects on your own site to the appropriate URL. Just sign in to Webmaster Tools and check it out for your verified site. You can help people visiting your site—from anywhere on the web—find what they're looking for.
  
  Written by Jonathan Simon, Webmaster Trends Analyst and Michael Williamson, Webmaster Tools Intern
• 

  Good times with inbound links

  Posted: October 10th, 2008, 1:17am CEST by Maile Ohye

  Tags:  crawling and indexing  

  Inbound links are links from pages on external sites linking back to your site. Inbound links can bring new users to your site, and when the links are merit-based and freely-volunteered as an editorial choice, they're also one of the positive signals to Google about your site's importance. Other signals include things like our analysis of your site's content, its relevance to a geographic location, etc. As many of you know, relevant, quality inbound links can affect your PageRank (one of many factors in our ranking algorithm). And quality links often come naturally to sites with compelling content or offering a unique service.
  
  How do these signals factor into ranking?
  
  Let's say I have a site, example.com, that offers users a variety of unique website templates and design tips. One of the strongest ranking factors is my site's content. Additionally, perhaps my site is also linked from three sources -- however, one inbound link is from a spammy site. As far as Google is concerned, we want only the two quality inbound links to contribute to the PageRank signal in our ranking.
  
  Given the user's query, over 200 signals (including the analysis of the site's content and inbound links as mentioned above) are applied to return the most relevant results to the user.
  
  
  So how can you engage more users and potentially increase merit-based inbound links?
  
  Many webmasters have written about their success in growing their audience. We've compiled several ideas and resources that can improve the web for all users.
  

  Create unique and compelling content on your site and the web in general

  • Start a blog: make videos, do original research, and post interesting stuff on a regular basis. If you're passionate about your site's topic, there are lots of great avenues to engage more users.
    
    If you're interested in blogging, see our Help Center for specific tips for bloggers.
  
  
  • Teach readers new things, uncover new news, be entertaining or insightful, show your expertise, interview different personalities in your industry and highlight their interesting side. Make your site worthwhile.
  
  
  • Participate thoughtfully in blogs and user reviews related to your topic of interest. Offer your knowledgeable perspective to the community.
  
  
  • Provide a useful product or service. If visitors to your site get value from what you provide, they're more likely to link to you.
  
  For more actionable ideas, see one of my favorite interviews with Matt Cutts for no-cost tips to help increase your traffic. It's a great primer for webmasters. (Even before this post, I forwarded the URL to many of my friends. :)

  Pursue business development opportunities

  Use Webmaster Tools for "Links > Pages with external links" to learn about others interested in your site. Expand the web community by figuring out who links to you and how they're linking. You may have new audiences or demographics you didn't realize were interested in your niche. For instance, if the webmasters for example.com noticed external links coming from art schools, they may start to engage with the art community -- receiving new feedback and promoting their site and ideas.
  
  Of course, be responsible when pursuing possible opportunities in this space. Don't engage in mass link-begging; no one likes form letters, and few webmasters of quality sites are likely to respond positively to such solicitations. In general, many of the business development techniques that are successful in human relationships can also be reflected online for your site.

  Now that you've read more information about internal links, outbound links, and inbound links (today's post :), we'll see you in the blog comments! Thanks for joining us for links week.
  
  Update -- Here's one more business development opportunity:
  Investigate your "Diagnostics > Web/mobile crawl > Crawl error sources" to not only correct broken links, but also to cultivate relationships with external webmasters who share an interest in your site. (And while you're chatting, see if they'll correct the broken link. :) This is a fantastic way to turn broken links into free links to important parts of your site.
  
  In addition to contacting these webmasters, you may also wish to use 301 redirects to redirect incoming traffic from old pages to their new locations. This is good for users who may still have bookmarks with links to your old pages... and you'll be happy to know that Google appropriately flows PageRank and related signals through these redirects.
  
  Written by Maile Ohye, Developer Programs Tech Lead
• 

  Importance of link architecture

  Posted: October 7th, 2008, 12:51am CEST by Maile Ohye

  Tags:  crawling and indexing  

  In Day 2 of links week, we'd like to discuss the importance of link architecture and answer more advanced questions on the topic. Link architecture—the method of internal linking on your site—is a crucial step in site design if you want your site indexed by search engines. It plays a critical role in Googlebot's ability to find your site's pages and ensures that your visitors can navigate and enjoy your site.
  
  Keep important pages within several clicks from the homepage
  
  Although you may believe that users prefer a search box on your site rather than category navigation, it's uncommon for search engine crawlers to type into search boxes or navigate via pulldown menus. So make sure your important pages are clickable from the homepage and for easy for Googlebot to find throughout your site. It's best to create a link architecture that's intuitive for users and crawlable for search engines. Here are more ideas to get started:

  Intuitive navigation for users
  
  Create common user scenarios, get "in character," then try working through your site. For example, if your site is about basketball, imagine being a visitor (in this case a "baller" :) trying to learn the best dribbling technique.
  

  • Starting at the homepage, if the user doesn't use the search box on your site or a pulldown menu, can they easily find the desired information (ball handling like a superstar) from the navigation links?
  
  
  • Let's say a user found your site through an external link, but they didn't land on the homepage. Starting from any (sub-/child) page on your site, make sure they can easily find their way to the homepage and/or other relevant sections. In other words, make sure users aren't trapped or stuck. Was the "best dribbling technique" easy for your imaginary user to find? Often breadcrumbs such as "Home > Techniques > Dribbling" help users to understand where they are.

  Crawlable links for search engines
  

  • Text links are easily discovered by search engines and are often the safest bet if your priority is having your content crawled. While you're welcome to try the latest technologies, keep-in-mind that when text-based links are available and easily navigable for users, chances are that search engines can crawl your site as well.
    
    This <a href="new-page.html">text link</a> is easy for search engines to find.
  
  
  • Sitemap submission is also helpful for major search engines, though it shouldn't be a substitute for crawlable link architecture. If your site utilizes newer techniques, such as AJAX, see "Verify that Googlebot finds your internal links" below.

  Use descriptive anchor text
  
  Writing descriptive anchor text, the clickable words in a link, is a useful signal to help search engines and users alike to better understand your content. The more Google knows about your site—through your content, page titles, anchor text, etc.—the more relevant results we can return for users (and your potential search visitors). For example, if you run a basketball site and you have videos to accompany the textual content, a not-very-optimal way of linking would be:
  
  To see all our basketball videos, <a href="videos.html">click here</a> for the entire listing.
  
  However, instead of the generic "click here," you could rewrite the anchor text more descriptively as:
  
  Feel free to browse all of our <a href="videos.html">basketball videos</a>.
  
  Verify that Googlebot finds your internal links
  
  For verified site owners, Webmaster Tools has the feature "Links > Pages with internal links" that's great for verifying that Googlebot finds most of the links you'd expect. This is especially useful if your site uses navigation involving JavaScript (which Googlebot doesn't always execute)—you'll want to make sure that Googlebot is finding other internal links as expected.
  
  Here's an abridged snapshot of our internal links to the introductory post for "404 week at Webmaster Central." Our internal links are discovered as we had hoped.
  
  
  Feel free to ask more internal linking questions
  Here are some to get you started...
  
  Q: What about using rel="nofollow" for maximizing PageRank flow in my internal link architecture (such as PageRank sculpting, or PageRank siloing)?
  

  A: It's not something we, as webmasters who also work at Google, would really spend time or energy on. In other words, if your site already has strong link architecture, it's far more productive to work on keeping users happy with fresh and compelling content rather than to worry about PageRank sculpting.
  
  Matt Cutts answered more questions about "appropriate uses of nofollow" in our webmaster discussion group.

  Q: Let's say my website is about my favorite hobbies: biking and camping. Should I keep my internal linking architecture "themed" and not cross-link between the two?

  A: We haven't found a case where a webmaster would benefit by intentionally "theming" their link architecture for search engines. And, keep-in-mind, if a visitor to one part of your site can't easily reach other parts of your site, that may be a problem for search engines as well.

  Perhaps it's cliche, but at the end of the day, and at the end of this post, :) it's best to create solid link architecture (making navigation intuitive for users and crawlable for search engines)—implementing what makes sense for your users and their experience on your site.
  
  Thanks for your time today! Information about outbound links will soon be available in Day 3 of links week. And, if you have helpful tips about internal links or questions for our team, please share them in the comments below.
  
  Written by Maile Ohye, Developer Programs Tech Lead
• 

  Links information straight from the source

  Posted: October 6th, 2008, 4:41am CEST by Maile Ohye

  Tags:  crawling and indexing  

  We hope that you're able to focus on helping users (and improving the web) by creating great content or providing a great service on your site. In between creating content and working on your site, you may have read some of the (often conflicting) link discussions circling the web. If you're asking, "What's going on -- what do I need to know about links?" then welcome to the first day of links week!
  
  Day 2: Internal links (links within your site)

  Internal linking is your homepage linking to your "Contact us" page, or your "Contact us" page linking to your "About me" page. Internal linking (also known as link architecture) is important because it's a major factor in how easily visitors can navigate your site. Additionally, internal linking contributes to your site's "crawlability" -- how easily a spider can reach your pages. More in Day 2 of links week.

  Day 3: Outbound links (sites you link to)

  Outbound links are external sites that you're linking to. For example, www.google.com/webmasters links to the domain googlewebmastercentral.blogspot.com (our lovely blog!). Outbound links allow us to surf the web -- they're a big reason why the web is so exciting and collaborative. Without outbound links, your site can seem isolated from the community because each page becomes "brochure-ware." Most sites include outbound links naturally and it shouldn't be a big concern. If you still have questions, we'll be covering outbound linking in more detail on Day 3.

  Day 4: Inbound links (sites linking to you)

  Inbound links are external sites linking to you. There are many webmasters who (rightfully) aren't preoccupied by the subject of inbound links. So why do some webmasters care? It's likely because merit-based or volunteered inbound links may seem like a quick way to increase rankings and traffic. Answers to your questions like, "Are there no-cost methods to maximize my merit-based links?" are provided on Day 4.

  Update: Included references to blog posts as they were published throughout links week.
  
  Written by Maile Ohye, Developer Programs Tech Lead
• 

  More webmaster questions - Answered!

  Posted: September 23rd, 2008, 3:24pm CEST by Maile Ohye

  Tags:  crawling and indexing  

  When it comes to answering your webmaster related questions, we just can't get enough. I wanted to follow-up and answer some additional questions that webmasters asked in our latest installment of Popular Picks. In case you missed it, you can find our answers to image search ranking, sitelinks, reconsideration requests, redirects, and our communication with webmasters in this blog post.
  
  
  
  Check out these resources for additional details on questions answered in the video:

  • Duplicate Content: check out Susan's blog post, Demystifying the "duplicate content penalty."
  • Website Design: we've written a thorough set of Help Center articles on creating a Google-friendly site.
  • Meta tags: you can read our Help Center article on how to block or remove pages using meta tags.
  • Local Search: Some tips to rank well for local searches include signing up with Google's Local Business Center, registering the appropriate top level domain or using the geographic target tool in Webmaster Tools.
  • Problems with Webmaster Tools: check the Webmaster Tools subgroup in our Help Group. If you're experiencing errors, other webmasters may be experiencing the same. In the past, we've pinned a message at the top of the Webmaster Tools subgroup to let users know we're aware of a problem and working to fix it.

  Video Transcript:
  
  Hi everyone, I'm Reid from the Search Quality team. Today I'd like to answer some of the unanswered questions from our latest round of popular picks.
  
  Searchmaster had a question about duplicate content. Understandably, this is a popular concern from webmasters. You should check out the Google Webmaster Central Blogwhere my colleague Susan Moskwa recently posted "Demystifying the 'duplicate content penalty," which answers many questions and concerns about duplicate content.
  
  Jay is the Boss wanted to know if e-commerce websites suffer if they have two or more different themes. For example, you could have a site that sells auto parts, but also sightseeing guides. In general, I'd encourage webmasters to create a website that they feel is relevant for users. If it makes sense to sell auto parts and sightseeing guides, then go for it. Those are the sites that perform well, because users want to visit those sites, and they'll link to them as well.
  
  emma2 wanted to know if Google will follow links on a page using the "noindex" attribute in the "robots" meta tag. To answer this question, Googlebot will follow links on a page which uses the meta "noindex" tag, but that page will not appear in our search results. As a reminder, if you would like to prevent Googlebot from crawling any links on a page, use the "nofollow" attribute in the "robots" meta tag.
  
  Aaron Pratt wanted to know about some ways a webmaster can rank well for local searches. A quick recommendation is to add your business to the Local Business Center. There, you can add contact information as well as store operating hours and coupons as well. Another example, or a tip, is to take advantage and purchase a country-specific top-level domain, or use the geotargeting feature in Webmaster Tools.
  
  jdeb901 said it would be helpful if we could let webmasters know if we are having problems with Webmaster Tools. This is an excellent point, and we're always thinking about better ways to communicate with webmasters. If you're having problems with Webmaster Tools, chances are someone else is as well, and they've posted to the Google Webmaster Help Group about this. In the past, if we've experienced problems with Webmaster Tools, we've also created a "sticky" post to let users know that we know about these issues with Webmaster Tools, and we're working to find a solution.
  
  Well, that about wraps it up with our Popular Picks. Thanks again for all of your questions, and I look forward to seeing you around the group.
  
  Written by Reid Yokoyama, Search Quality
• 

  Dynamic URLs vs. static URLs

  Posted: September 22nd, 2008, 5:20pm CEST by Maile Ohye

  Tags:  crawling and indexing  

  Chatting with webmasters often reveals widespread beliefs that might have been accurate in the past, but are not necessarily up-to-date any more. This was the case when we recently talked to a couple of friends about the structure of a URL. One friend was concerned about using dynamic URLs, since (as she told us) "search engines can't cope with these." Another friend thought that dynamic URLs weren't a problem at all for search engines and that these issues were a thing of the past. One even admitted that he never understood the fuss about dynamic URLs in comparison to static URLs. For us, that was the moment we decided to read up on the topic of dynamic and static URLs. First, let's clarify what we're talking about:
  
  What is a static URL?
  A static URL is one that does not change, so it typically does not contain any url parameters. It can look like this: http://www.example.com/archive/january.htm. You can search for static URLs on Google by typing filetype:htm in the search field. Updating these kinds of pages can be time consuming, especially if the amount of information grows quickly, since every single page has to be hard-coded. This is why webmasters who deal with large, frequently updated sites like online shops, forum communities, blogs or content management systems may use dynamic URLs.
  
  What is a dynamic URL?
  If the content of a site is stored in a database and pulled for display on pages on demand, dynamic URLs maybe used. In that case the site serves basically as a template for the content. Usually, a dynamic URL would look something like this: http://code.google.com/p/google-checkout-php-sample-code/issues/detail?id=31. You can spot dynamic URLs by looking for characters like: ? = &. Dynamic URLs have the disadvantage that different URLs can have the same content. So different users might link to URLs with different parameters which have the same content. That's one reason why webmasters sometimes want to rewrite their URLs to static ones.
  
  Should I try to make my dynamic URLs look static?
  Following are some key points you should keep in mind while dealing with dynamic URLs:
  

  1. It's quite hard to correctly create and maintain rewrites that change dynamic URLs to static-looking URLs.
  2. It's much safer to serve us the original dynamic URL and let us handle the problem of detecting and avoiding problematic parameters.
  3. If you want to rewrite your URL, please remove unnecessary parameters while maintaining a dynamic-looking URL.
  4. If you want to serve a static URL instead of a dynamic URL you should create a static equivalent of your content.

  Which can Googlebot read better, static or dynamic URLs?
  We've come across many webmasters who, like our friend, believed that static or static-looking URLs were an advantage for indexing and ranking their sites. This is based on the presumption that search engines have issues with crawling and analyzing URLs that include session IDs or source trackers. However, as a matter of fact, we at Google have made some progress in both areas. While static URLs might have a slight advantage in terms of clickthrough rates because users can easily read the urls, the decision to use database-driven websites does not imply a significant disadvantage in terms of indexing and ranking. Providing search engines with dynamic URLs should be favored over hiding parameters to make them look static.
  
  Let's now look at some of the widespread beliefs concerning dynamic URLs and correct some of the assumptions which spook webmasters. :)
  
  Myth: "Dynamic URLs cannot be crawled."
  Fact: We can crawl dynamic URLs and interpret the different parameters. We might have problems crawling and ranking your dynamic URLs if you try to make your urls look static and in the process hide parameters which offer the Googlebot valuable information. One recommendation is to avoid reformatting a dynamic URL to make it look static. It's always advisable to use static content with static URLs as much as possible, but in cases where you decide to use dynamic content, you should give us the possibility to analyze your URL structure and not remove information by hiding parameters and making them look static.
  
  Myth: "Dynamic URLs are okay if you use fewer than three parameters."
  Fact: There is no limit on the number of parameters, but a good rule of thumb would be to keep your URLs short (this applies to all URLs, whether static or dynamic). You may be able to remove some parameters which aren't essential for Googlebot and offer your users a nice looking dynamic URL. If you are not able to figure out which parameters to remove, we'd advise you to serve us all the parameters in your dynamic URL and our system will figure out which ones do not matter. Hiding your parameters keeps us from analyzing your URLs properly and we won't be able to recognize the parameters as such, which could cause a loss of valuable information.
  
  Following are some questions we thought you might have at this point.
  
  Does that mean I should avoid rewriting dynamic URLs at all?
  That's our recommendation, unless your rewrites are limited to removing unnecessary parameters, or you are very diligent in removing all parameters that could cause problems. If you transform your dynamic URL to make it look static you should be aware that we might not be able to interpret the information correctly in all cases. If you want to serve a static equivalent of your site, you might want to consider transforming the underlying content by serving a replacement which is truly static. One example would be to generate files for all the paths and make them accessible somewhere on your site. However, if you're using URL rewriting (rather than making a copy of the content) to produce static-looking URLs from a dynamic site, you could be doing harm rather than good. Feel free to serve us your standard dynamic URL and we will automatically find the parameters which are unnecessary.
  
  Can you give me an example?
  If you have a dynamic URL which is in the standard format like foo?key1=value&key2=value2 we recommend that you leave the url unchanged, and Google will determine which parameters can be removed; or you could remove uncessary parameters for your users. Be careful that you only remove parameters which do not matter. Here's an example of a URL with a couple of parameters:
  
  www.example.com/article/bin/answer.foo?language=en&answer=3&sid=98971298178906&query=URL
  

  • language=en - indicates the language of the article
  • answer=3 - the article has the number 3
  • sid=8971298178906 - the session ID number is 8971298178906
  • query=URL - the query with which the article was found is [URL]

  Not all of these parameters offer additional information. So rewriting the URL to www.example.com/article/bin/answer.foo?language=en&answer=3 probably would not cause any problems as all irrelevant parameters are removed.
  
  The following are some examples of static-looking URLs which may cause more crawling problems than serving the dynamic URL without rewriting:

  • www.example.com/article/bin/answer.foo/en/3/98971298178906/URL
  • www.example.com/article/bin/answer.foo/language=en/answer=3/
    sid=98971298178906/query=URL
  • www.example.com/article/bin/answer.foo/language/en/answer/3/
    sid/98971298178906/query/URL
  • www.example.com/article/bin/answer.foo/en,3,98971298178906,URL

  Rewriting your dynamic URL to one of these examples could cause us to crawl the same piece of content needlessly via many different URLs with varying values for session IDs (sid) and query. These forms make it difficult for us to understand that URL and 98971298178906 have nothing to do with the actual content which is returned via this URL. However, here's an example of a rewrite where all irrelevant parameters have been removed:

  • www.example.com/article/bin/answer.foo/en/3

  Although we are able to process this URL correctly, we would still discourage you from using this rewrite as it is hard to maintain and needs to be updated as soon as a new parameter is added to the original dynamic URL. Failure to do this would again result in a static looking URL which is hiding parameters. So the best solution is often to keep your dynamic URLs as they are. Or, if you remove irrelevant parameters, bear in mind to leave the URL dynamic as the above example of a rewritten URL shows:

  • www.example.com/article/bin/answer.foo?language=en&answer=3

  We hope this article is helpful to you and our friends to shed some light on the various assumptions around dynamic URLs. Please feel free to join our discussion group if you have any further questions.
  
  Written by Juliane Stiller and Kaspar Szymanski, Search Quality Team
• 

  Demystifying the "duplicate content penalty"

  Posted: September 12th, 2008, 10:30am CEST by Susan Moskwa

  Tags:  crawling and indexing  

  Duplicate content. There's just something about it. We keep writing about it, and people keep asking about it. In particular, I still hear a lot of webmasters worrying about whether they may have a "duplicate content penalty."

  Let's put this to bed once and for all, folks: There's no such thing as a "duplicate content penalty." At least, not in the way most people mean when they say that.

  There are some penalties that are related to the idea of having the same content as another site—for example, if you're scraping content from other sites and republishing it, or if you republish content without adding any additional value. These tactics are clearly outlined (and discouraged) in our Webmaster Guidelines:

  • Don't create multiple pages, subdomains, or domains with substantially duplicate content.
  • Avoid... "cookie cutter" approaches such as affiliate programs with little or no original content.
  • If your site participates in an affiliate program, make sure that your site adds value. Provide unique and relevant content that gives users a reason to visit your site first.

  (Note that while scraping content from others is discouraged, having others scrape you is a different story; check out this post if you're worried about being scraped.)

  But most site owners whom I hear worrying about duplicate content aren't talking about scraping or domain farms; they're talking about things like having multiple URLs on the same domain that point to the same content. Like www.example.com/skates.asp?color=black&brand=riedell and www.example.com/skates.asp?brand=riedell&color=black. Having this type of duplicate content on your site can potentially affect your site's performance, but it doesn't cause penalties. From our article on duplicate content:

  Duplicate content on a site is not grounds for action on that site unless it appears that the intent of the duplicate content is to be deceptive and manipulate search engine results. If your site suffers from duplicate content issues, and you don't follow the advice listed above, we do a good job of choosing a version of the content to show in our search results.

  This type of non-malicious duplication is fairly common, especially since many CMSs don't handle this well by default. So when people say that having this type of duplicate content can affect your site, it's not because you're likely to be penalized; it's simply due to the way that web sites and search engines work.

  Most search engines strive for a certain level of variety; they want to show you ten different results on a search results page, not ten different URLs that all have the same content. To this end, Google tries to filter out duplicate documents so that users experience less redundancy. You can find details in this blog post, which states:

  1. When we detect duplicate content, such as through variations caused by URL parameters, we group the duplicate URLs into one cluster.
  2. We select what we think is the "best" URL to represent the cluster in search results.
  3. We then consolidate properties of the URLs in the cluster, such as link popularity, to the representative URL.

  Here's how this could affect you as a webmaster:

  • In step 2, Google's idea of what the "best" URL is might not be the same as your idea. If you want to have control over whether www.example.com/skates.asp?color=black&brand=riedell or www.example.com/skates.asp?brand=riedell&color=black gets shown in our search results, you may want to take action to mitigate your duplication. One way of letting us know which URL you prefer is by including the preferred URL in your Sitemap.
  • In step 3, if we aren't able to detect all the duplicates of a particular page, we won't be able to consolidate all of their properties. This may dilute the strength of that content's ranking signals by splitting them across multiple URLs.

  In most cases Google does a good job of handling this type of duplication. However, you may also want to consider content that's being duplicated across domains. In particular, deciding to build a site whose purpose inherently involves content duplication is something you should think twice about if your business model is going to rely on search traffic, unless you can add a lot of additional value for users. For example, we sometimes hear from Amazon.com affiliates who are having a hard time ranking for content that originates solely from Amazon. Is this because Google wants to stop them from trying to sell Everyone Poops? No; it's because how the heck are they going to outrank Amazon if they're providing the exact same listing? Amazon has a lot of online business authority (most likely more than a typical Amazon affiliate site does), and the average Google search user probably wants the original information on Amazon, unless the affiliate site has added a significant amount of additional value.

  Lastly, consider the effect that duplication can have on your site's bandwidth. Duplicated content can lead to inefficient crawling: when Googlebot discovers ten URLs on your site, it has to crawl each of those URLs before it knows whether they contain the same content (and thus before we can group them as described above). The more time and resources that Googlebot spends crawling duplicate content across multiple URLs, the less time it has to get to the rest of your content.

  In summary: Having duplicate content can affect your site in a variety of ways; but unless you've been duplicating deliberately, it's unlikely that one of those ways will be a penalty. This means that:

  • You typically don't need to submit a reconsideration request when you're cleaning up innocently duplicated content.
  • If you're a webmaster of beginner-to-intermediate savviness, you probably don't need to put too much energy into worrying about duplicate content, since most search engines have ways of handling it.
  • You can help your fellow webmasters by not perpetuating the myth of duplicate content penalties! The remedies for duplicate content are entirely within your control. Here are some good places to start.

  Posted by Susan Moskwa, Webmaster Trends Analyst

• 

  More on 404

  Posted: August 15th, 2008, 4:52pm CEST by Maile Ohye

  Tags:  crawling and indexing  

  Now that we've bid farewell to soft 404s, in this post for 404 week we'll answer your burning 404 questions.
  
  How do you treat the response code 410 "Gone"?
  Just like a 404.
  
  Do you index content or follow links from a page with a 404 response code?
  We aim to understand as much as possible about your site and its content. So while we wouldn't want to show a hard 404 to users in search results, we may utilize a 404's content or links if it's detected as a signal to help us better understand your site.
  
  Keep in mind that if you want links crawled or content indexed, it's far more beneficial to include them in a non-404 page.
  
  What about 404s with a 10-second meta refresh?
  Yahoo! currently utilizes this method on their 404s. They respond with a 404, but the 404 content also shows:
  
  <meta http-equiv="refresh" content="10;url=http://www.yahoo.com/?xxx">
  
  We feel this technique is fine because it reduces confusion by giving users 10 seconds to make a new selection, only offering the homepage after 10 seconds without the user's input.
  
  Should I 301-redirect misspelled 404s to the correct URL?
  Redirecting/301-ing 404s is a good idea when it's helpful to users (i.e. not confusing like soft 404s). For instance, if you notice that the Crawl Errors of Webmaster Tools shows a 404 for a misspelled version of your URL, feel free to 301 the misspelled version of the URL to the correct version.
  
  For example, if we saw this 404 in Crawl Errors:
  http://www.google.com/webmsters  <-- typo for "webmasters"
  
  we may first correct the typo if it exists on our own site, then 301 the URL to the correct version (as the broken link may occur elsewhere on the web):
  http://www.google.com/webmasters
  
  Have you guys seen any good 404s?
  Yes, we have! (Confession: no one asked us this question, but few things are as fun to discuss as response codes. :) We've put together a list of some of our favorite 404 pages. If you have more 404-related questions, let us know, and thanks for joining us for 404 week!

  [www.metrokitchen.com]
  "If you're looking for an item that's no longer stocked (as I was), this makes it really easy to find an alternative."
  -Riona, domestigeek
  [www.comedycentral.com]
  "Blame the robot monkeys"
  -Reid, tells really bad jokes
  [www.splicemusic.com]
  "Boost your 'Time on site' metrics with a 404 page like this."
  -Susan, dabbler in music and Analytics
  [www.treachery.net]
  "It's not reassuring, but it's definitive."
  -Jonathan, has trained actual spiders to build websites, ants handle the 404s
  [www.apple.com]
  "Good with respect to usability."
  [thcnet.net]
  "At least there's a mailbox."
  -JohnMu, adventurous
  [lookitsme.co.uk]
  "It's pretty cute. :)"
  -Jessica, likes cute things
  [www.orangecoat.com]
  "Flow charts rule."
  -Sahala, internet traveller
  [icanhascheezburger.com]
  "I can has useful links and even e-mail address for questions! But they could have added 'OH NOES! IZ MISSING PAGE! MAYBE TIPO OR BROKN LINKZ?' so folks'd know what's up."
  -Adam, lindy hop geek

  
  Written by Maile Ohye, Developer Programs Tech Lead
• 

  Farewell to soft 404s

  Posted: August 12th, 2008, 4:54pm CEST by Maile Ohye

  Tags:  crawling and indexing  

  We see two kinds of 404 ("File not found") responses on the web: "hard 404s" and "soft 404s." We discourage the use of so-called "soft 404s" because they can be a confusing experience for users and search engines. Instead of returning a 404 response code for a non-existent URL, websites that serve "soft 404s" return a 200 response code. The content of the 200 response is often the homepage of the site, or an error page.
  
  How does a soft 404 look to the user? Here's a mockup of a soft 404: This site returns a 200 response code and the site's homepage for URLs that don't exist.
  
  
  
  As exemplified above, soft 404s are confusing for users, and furthermore search engines may spend much of their time crawling and indexing non-existent, often duplicative URLs on your site. This can negatively impact your site's crawl coverage—because of the time Googlebot spends on non-existent pages, your unique URLs may not be discovered as quickly or visited as frequently.
  
  What should you do instead of returning a soft 404?
  It's much better to return a 404 response code and clearly explain to users that the file wasn't found. This makes search engines and many users happy.
  
  Return 404 response code
  
  
  Return clear message to users
  
  
  Can your webserver return 404, but send a helpful "Not found" message to the user?
  Of course! More info as "404 week" continues!
  
  Written by Maile Ohye, Developer Programs Tech Lead
• 

  It's 404 week at Webmaster Central

  Posted: August 11th, 2008, 3:40pm CEST by Maile Ohye

  Tags:  crawling and indexing  

  This week we're publishing several blog posts dedicated to helping you with one response code: 404.
  
  Response codes are a numeric status (like 200 for "OK", 301 for "Moved Permanently") that a webserver returns in response to a request for a URL. The 404 response code should be returned for a file "Not Found".
  
  When a user sends a request for your webpage, your webserver looks for the corresponding file for the URL. If a file exists, your webserver likely responds with a 200 response code along with a message (often the content of the page, such as the HTML).
  
  
  
  So what's a 404? Let's say that in the link to "Visit Google Apps" above, the link is broken because of a typing error when coding the page. Now when a user clicks "Visit Google Apps", the particular webpage/file isn't located by the webserver. The webserver should return a 404 response code, meaning "Not Found".
  
  
  
  Now that we're all on board with the basics of 404s, stay tuned 4 even more information on making 404s good 4 users and 4 search engines.
  
  Written by Maile Ohye, Developer Programs Tech Lead
• 

  To infinity and beyond? No!

  Posted: August 5th, 2008, 3:27pm CEST by Maile Ohye

  Tags:  crawling and indexing  

  When Googlebot crawls the web, it often finds what we call an "infinite space". These are very large numbers of links that usually provide little or no new content for Googlebot to index. If this happens on your site, crawling those URLs may use unnecessary bandwidth, and could result in Googlebot failing to completely index the real content on your site.
  
  Recently, we started notifying site owners when we discover this problem on their web sites. Like most messages we send, you'll find them in Webmaster Tools in the Message Center. You'll probably want to know right away if Googlebot has this problem - or other problems - crawling your sites. So verify your site with Webmaster Tools, and check the Message Center every now and then.
  
  
  
  Examples of an infinite space
  
  The classic example of an "infinite space" is a calendar with a "Next Month" link. It may be possible to keep following those "Next Month" links forever! Of course, that's not what you want Googlebot to do. Googlebot is smart enough to figure out some of those on its own, but there are a lot of ways to create an infinite space and we may not detect all of them.
  
  
  Another common scenario is websites which provide for filtering a set of search results in many ways. A shopping site might allow for finding clothing items by filtering on category, price, color, brand, style, etc. The number of possible combinations of filters can grow exponentially. This can produce thousands of URLs, all finding some subset of the items sold. This may be convenient for your users, but is not so helpful for the Googlebot, which just wants to find everything - once!
  
  Correcting infinite space issues
  
  Our Webmaster Tools Help article describes more ways infinite spaces can arise, and provides recommendations on how to avoid the problem. One fix is to eliminate whole categories of dynamically generated links using your robots.txt file. The Help Center has lots of information on how to use robots.txt. If you do that, don't forget to verify that Googlebot can find all your content some other way. Another option is to block those problematic links with a "nofollow" link attribute. If you'd like more information on "nofollow" links, check out the Webmaster Help Center.
  
  Written by Torrey Hoffman, Webmaster Tools team
• 

  A day in the life of webmaster support

  Posted: July 30th, 2008, 7:08pm CEST by Maile Ohye

  Tags:  feedback and communication  

  It was an ordinary Wednesday morning when I decided to film my day at the Googleplex...
  
  
  
  A recap of the day's events
  
  10AM Meeting with the webmaster support team

  Our team shares a doc containing our current agenda and the previous meetings' agenda, minutes, and action items. In this meeting, we discussed:

  • Feedback from blog post on Duplicate content due to scrapers. Some webmasters suggested that we could improve our detection. In order to improve quality, it would help to get feedback with specific examples. Susan Moskwa, one of our Webmaster Trends Analysts based in Kirkland, Washington, volunteered to post a blog comment to solicit more information.
  
  
  • Recent and upcoming releases
  • Webmaster Tools API on schedule
  • "Skip intro" in search results
  
  
  • JuneTune online chat agenda
  
  
  • Two recent spam techniques mentioned in the blogosphere. Brian White, who leads one of the Webspam-fighting groups at Google, explained that one technique is new twist on old idea, both are already handled.

  11AM Meeting with Matt Cutts

  Matt provided feedback on:

  • Proposal to write follow-up blog comment on duplicate content caused by scrapers to solicit specific examples. Approved.
  
  
  • "URLs and case sensitivity basics" presentation for online chat

  1PM Lunch with Shyam, a Crawl engineer, and Jason, and AdSense engineer
  2PM Meeting with Wysz to review slides for Google Trifecta
  5PM Little "drive-by" to catch Reid, Evan, Charlene, Jessica, and Wysz while they're monitoring the discussion group
  7PM Dinner with Matthias
  p.s. A huge thanks to Wysz for his film editing skillz.
  
  Written by Maile Ohye, Developer Programs Tech Lead
• 

  Google Trifecta recording now available

  Posted: July 17th, 2008, 5:18pm CEST by Michael Wyszomierski

  Tags:  products and services  

  Last Tuesday, Google hosted an online presentation about three free tools to help analyze and improve your sites. Dubbed "The Google Trifecta," this presentation explains the key features of Google Webmaster Tools, Google Analytics, and Google Website Optimizer, and how you can use them together.
  
  To make the information from this presentation accessible to those who were unable to listen in last week or would like to review it again, we've put a recording of the session on YouTube for your viewing pleasure:
  
  
  
  If you have any followup questions about the topics covered in the video, we welcome you to ask! Each of these products has a discussion group available for you to discuss the product with other users and Googlers. Please post in the appropriate group, so your question is seen by the right people:
  
  Google Webmaster Help (Webmaster Tools)
  Analytics Help
  Website Optimizer Forum
  
  Posted by Michael Wyszomierski, Search Quality Team
• 

  Improved Flash indexing

  Posted: June 30th, 2008, 11:31pm CEST by A Googler

  Tags:  crawling and indexing  

  Written by Ron Adler, Janis Stipins, and Maile Ohye
  
  We've received numerous requests to improve our indexing of Adobe Flash files. Today, Ron Adler and Janis Stipins—software engineers on our indexing team—will provide us with more in-depth information about our recent announcement that we've greatly improved our ability to index Flash.
  
  Q: Which Flash files can Google better index now?
  We've improved our ability to index textual content in SWF files of all kinds. This includes Flash "gadgets" such as buttons or menus, self-contained Flash websites, and everything in between.
  
  Q: What content can Google better index from these Flash files?
  All of the text that users can see as they interact with your Flash file. If your website contains Flash, the textual content in your Flash files can be used when Google generates a snippet for your website. Also, the words that appear in your Flash files can be used to match query terms in Google searches.
  
  In addition to finding and indexing the textual content in Flash files, we're also discovering URLs that appear in Flash files, and feeding them into our crawling pipeline—just like we do with URLs that appear in non-Flash webpages. For example, if your Flash application contains links to pages inside your website, Google may now be better able to discover and crawl more of your website.
  
  Q: What about non-textual content, such as images?
  At present, we are only discovering and indexing textual content in Flash files. If your Flash files only include images, we will not recognize or index any text that may appear in those images. Similarly, we do not generate any anchor text for Flash buttons which target some URL, but which have no associated text.
  
  Also note that we do not index FLV files, such as the videos that play on YouTube, because these files contain no text elements.
  
  Q: How does Google "see" the contents of a Flash file?
  We've developed an algorithm that explores Flash files in the same way that a person would, by clicking buttons, entering input, and so on. Our algorithm remembers all of the text that it encounters along the way, and that content is then available to be indexed. We can't tell you all of the proprietary details, but we can tell you that the algorithm's effectiveness was improved by utilizing Adobe's new Searchable SWF library.
  
  Q: What do I need to do to get Google to index the text in my Flash files?
  Basically, you don't need to do anything. The improvements that we have made do not require any special action on the part of web designers or webmasters. If you have Flash content on your website, we will automatically begin to index it, up to the limits of our current technical ability (see next question).
  
  That said, you should be aware that Google is now able to see the text that appears to visitors of your website. If you prefer Google to ignore your less informative content, such as a "copyright" or "loading" message, consider replacing the text within an image, which will make it effectively invisible to us.
  
  Q: What are the current technical limitations of Google's ability to index Flash?
  There are three main limitations at present, and we are already working on resolving them:
  
  1. Googlebot does not execute some types of JavaScript. So if your web page loads a Flash file via JavaScript, Google may not be aware of that Flash file, in which case it will not be indexed.
  2. We currently do not attach content from external resources that are loaded by your Flash files. If your Flash file loads an HTML file, an XML file, another SWF file, etc., Google will separately index that resource, but it will not yet be considered to be part of the content in your Flash file.
  3. While we are able to index Flash in almost all of the languages found on the web, currently there are difficulties with Flash content written in bidirectional languages. Until this is fixed, we will be unable to index Hebrew language or Arabic language content from Flash files.
  
  We're already making progress on these issues, so stay tuned!
  
  Update: Everyone, thanks for your great questions and feedback. Our focus is to improve search quality for all users, and with better Flash indexing we create more meaningful search results. Listed below, we’ve also answered some of the most prevalent questions. Thanks again!
  
  Flash site in search results before improvements
  
  
  Flash site after improved indexing, querying [nasa deep impact animation]
  
  
  Helping us access and index your Flash files
  

  @fintan: We verified with Adobe that the textual content from legacy sites, such as those scripted with AS1 and AS2, can be indexed by our new algorithm.
  
  @andrew, jonny m, erichazann, mike, ledge, stu, rex, blog, dis: For our July 1st launch, we didn't enable Flash indexing for Flash files embedded via SWFObject. We're now rolling out an update that enables support for common JavaScript techniques for embedding Flash, including SWFObject and SWFObject2.
  
  @mike: At this time, content loaded dynamically from resource files is not indexed. We’ve noted this feature request from several webmasters -- look for this in a near future update.

  
  Interaction of HTML pages and Flash
  

  @captain cuisine: The text found in Flash files is treated similarly to text found in other files, such as HTML, PDFs, etc. If the Flash file is embedded in HTML (as many of the Flash files we find are), its content is associated with the parent URL and indexed as single entity.
  
  @jeroen: Serving the same content in Flash and an alternate HTML version could cause us to find duplicate content. This won't cause a penalty -- we don’t lower a site in ranking because of duplicate content. Be aware, though, that search results will most likely only show one version, not both.
  
  @All: We’re trying to serve users the most relevant results possible regardless of the file type. This means that standalone Flash, HTML with embedded Flash, HTML only, PDFs, etc., can all have the potential to be returned in search results.

  
  Indexing large Flash files
  

  @dsfdgsg: We’ve heard requests for deep linking (linking to specific content inside file) not just for Flash results, but also for other large documents and presentations. In the case of Flash, the ability to deep link will require additional functionality in Flash with which we integrate.
  
  @All: The majority of the existing Flash files on the web are fine in regard to filesize. It shouldn’t be too much of a concern.

  
  More details about our Flash indexing algorithm
  

  @brian, marcos, bharath: Regarding ActionScript, we’re able to find new links loaded through ActionScript. We explore Flash like a website visitor does, we do not decompile the SWF file. Unless you're making ActionScript visible to users, Google will not expose ActionScript code.
  
  @dlocks: We respect rel="nofollow" wherever we encounter it in HTML.

• 

  What are your SEO recommendations?

  Posted: June 30th, 2008, 2:31pm CEST by Susan Moskwa

  Tags:  feedback and communication  

  Posted by the Webmaster Trends Analysts (John, Jonathan, Susan)

  You may have noticed that we recently rewrote our article on What is an SEO? Does Google recommend them? Previously, the article had focused on warning people about common SEO scams to look out for, but didn't mention many of the valuable services that a helpful SEO can provide.

  The article now notes some of the benefits of search engine optimization, and provides some guidance to site owners who are considering hiring an SEO. We'd also like to get your perspective: how would you define SEO? What questions would you ask a prospective SEO? What advice would you give to an inexperienced webmaster who's considering whether to contract an SEO? We'd like to hear your thoughts and incorporate your feedback if there's important advice that we should add.

• 

  One year of monitored European Webmaster Help Groups

  Posted: June 24th, 2008, 7:27am CEST by John Mueller

  Tags:  feedback and communication  

  Written by Kaspar Szymanski and the European Guides Team, Dublin
  
  A year ago Google Guides from the Search Quality Team started monitoring Google Webmaster Help Groups in European languages. It has been an interesting time and a great, rewarding experience for all those involved :o). We have enjoyed building and growing our communities with webmasters across Europe and we hope you have as well. The feedback we have been receiving since the beginning has been encouraging, with many webmasters responding positively to the opportunity to be able to solve their indexing issues in their native language. Still, we are aware that we have a lot of work ahead of us. I would like to thank everyone who has been active in our communities and invite those who have not joined yet to do so. Sign up, participate and help us make a difference.
  
  Taking advantage of this anniversary, I would like the European Google Guides to have a chance to say a few words about their communities.
  
  Polish Webmaster Help Group
  Guglarz, the Polish Google Guide
  Our group grew very fast, especially in the last few months and we are well on the way to hitting 600 users! But what I appreciate even more then our group's success, is the open, friendly spirit we have been cultivating in the last twelve months :o). I feel like this is a community that is able to help with indexing issues and beyond!
  
  I already pointed out some exceptional community members in the last blog post about communication with webmasters in a dozen languages. Since then, a few more have joined the list of tremendously committed, group-driving contributors. Bigu has been jumping in every time I've been traveling or very busy, which is the reason why I’d like to thank him a lot.
  
  Also I want to acknowledge Maciej Gluszek for escalating language issues we used to struggle with. You guys help to improve the Google experience for everyone.
  
  Lastly, I want to invite all group members to the chit chat section in order to introduce yourself and share some more information about who you are and what you do. I am really curious to find out more about the people I work with! I hope to see you in the group soon :o).
  
  ---
  Группа помощи Google для веб-мастеров
  Oxana, Tilek, Vitali, Mariya
  Our Russian Webmaster Help group is growing by the day and just reached an important milestone - a whole kilobyte of members: 1024 (-:! Thanks to all our users for the great questions, the wisdom, and the humour which they bring to our community. Special thanks go to Web-Master, Crazy LionHeart, Andrey Morozov and Lmd for their spot-on comments and prolific posts - we really appreciate your dedication in helping others. We're very happy to be part of such a savvy and lively community, we look forward to more interesting questions and discussions in the future, and to reaching one megabyte of members (-:! С днем рождения!
  
  ---
  Google Webmester Súgó Csoport
  Tibor, János
  Although the Hungarian Webmaster Help Group is one of our youngest, it already counts 300+ members, and has a pretty high number of lively discussions. The atmosphere is really great and loads of issues are resolved thanks largely to our dedicated and wise users. I would especially like to thank snomag for helping greatly in building the group up in the beginning (where have you disappeared to? We miss you! :)
  
  ---
  Google Site Yöneticisi Yardım Grubu
  Bahar
  The Turkish Webmaster Help Group has grown into a dynamic and effective community of fantastic webmasters in just four months. I want to thank to everyone in the group who has been eager to share his/her knowledge to help others. Special thanks to Erkan (man_blood) who has written a huge amount of posts in the last three months; Merve who has helped the group with her technical skills as well as calm spirit, especially when I was on vacation; Salih (SesVer) who used to help a lot when the group was first launched; and many more who are not mentioned by name here. It has been exciting and a pleasure to be a part of this rapidly growing family! Desteğiniz için teşekkürler :-)
  
  ---
  קבוצת העזרה למנהלי ומנהלות אתרים
  Alon, your Google Guide
  Our Hebrew group is maturing. To date we have more than 200 users and we are gaining visibility while more users are using the well-informed discussions we produce.
  
  It is a pleasure to see so many webmasters stopping by to ask, learn and help others. I would like to take the opportunity to thank all of you who make the group a place for discussion and collaboration. My special thanks go to Shoshanna, AliaG, Tomer and Seosos, for their great contribution.
  
  By browsing through the “tell us about yourself ” thread you can learn more about our community where many members introduce themselves. You're also welcome to watch my short video giving some specific tips for Israeli Webmasters.
  
  ---
  Google diskussionsgruppe for webmasters
  GoogleGuide (Jonas)
  The Danish group has been slowly gaining more subscribers since its launch last year, and we still hope to attract more people. We've grown almost 100% in the past year, and we hope to keep up the steam! (: This growth wouldn't have been possible without the help of Anders who contributes equal amounts of webmaster goodness in the form of sharing his tips on 301's and moving content to a new domain (which has been the hottest and most debated topic in the group), to issues on getting your site better indexed and crawled. So a big thanks to Anders, and all the others who have made this group useful for everybody.
  
  ---
  Ajuda a Webmasters do Google
  Pedro
  The Portuguese Webmaster Help Group is now above 1200 members. The success of the group is a reflection on the great community built by every single contributor. Special thanks to Carlos Lavieri, Rodrigo Soares, Leandro Leite e João Carlos (Orquiza) who used to help so much and recently Flávio Raimundo (M&S) and Ruben Zevallos Jr.
  
  Recently I had the chance of giving a few public interviews to Portuguese speaking webmasters and recognizing the group and its success publicly. I'd also like to take the opportunity to thank my interviewers for their great questions; if you speak Portuguese or you are a Google Translator fan you can read them: Entrevista com Pedro Dias (by Paulo Teixeira) and Entrevista a Pedro Dias (by Kazulo Webmarketing). It is a great pleasure to be a part of such an awesome community.
  
  ---
  Googlen Keskusteluryhmä Verkkovastaaville
  Anu
  The Finnish Webmaster Help Group will be celebrating its first birthday soon. During the first year, we've built a small but dynamic community and covered various topics from general webmaster issues to language-related toughies. A huge thank you to everyone who has taken part in discussions! I'm looking forward to starting our second year as a group, hope to see both familiar faces and newcomers join in our community!
  
  ---
  Groupe Google d'entraide pour les webmasters
  Alec, Nathalie
  
  The French Webmaster Help Group was launched one year ago and we've had a tremendous growth thanks to all our fantastic members. Special thanks to Paul-Arnaud and Cyrille who used to help so much and to Thierry who is now our bionic poster :-)
  
  Recently I had the chance to discuss the Webmaster Help Group with a French SEO I met at SES Paris '08. It was so great to connect again on the Internet and help him with his issues.
  
  ---
  Swedish Webmaster Help Group
  Hessam, the Swedish Google Guide
  The small Swedish webmaster forum, with currently almost 200 users, has almost tripled its size since its launch last July. During the past year we have seen a steady growth in user participation and a number of questions from our Swedish webmasters. I also had the chance to attend Search Marketing Expo (SMX) Stockholm 2007 which was definitely the highlight of the year. There I was able to meet a number of Swedish webmasters to talk about Google, our Webmaster Tools as well as the cold Swedish winter. During the coming year we are determined to further grow the community to better support our users. Lastly, a big thanks goes to everyone who has supported the group, especially Mattias Nordin and Widar Nord who have contributed with their knowledge and time. I am looking forward to hearing more from our regulars and would like to extend an invite to new members.
  
  ---
  Google Diskussionsforum für Webmaster
  Uli
  With more than 2700 members and almost a thousand posts per month, the German Webmaster Help Group has developed into a vibrant community. You will find many very savvy webmasters there who are always prepared to discuss all webmaster-related issues and offer advice to both beginners and advanced webmasters.
  
  I would like to take this opportunity to thank everyone who has contributed to this forum. Some webmasters like luzie, alpaka, beejees, Garzosch have been a crucial part of our group since this group began, and with the likes of Zombie joining in October, Mikki in January and Sistrix in March after having the pleasure of meeting him at SMX in Munich, we have an ever-growing pool of wisdom :-)
  
  ---
  Foro de Google para Webmasters
  Rebecca, Alvar
  The Spanish webmaster group now has almost 2500 members. We have some wonderful webmasters that offer great advice and support, people like Ricardo Antonio, Enrique de Mesa, Jesús Cáceres (chupi), Jose Antonio Nobile (nobilesoft), Carles Pastor, and many others who help make this forum work as well as it does. Recently we had one member visit whose site had been hacked with a cloaking technique and the community was able to help point out how to detect the hidden links and clear the hacked content. It’s great to see his website clean again and up and running.
  
  If you want to know who is behind this forum, have a look at the video where we introduce the forum and Webmaster Tools. Additionally, please give any feedback you may have in the thread where we present the video. By the way, it was great meeting some of you guys face to face at the SMX Madrid conference.
  
  ---
  Google Discussiegroep voor Webmaster
  Andre, Jos
  Our Dutch Webmaster Help Group now has well over 700 members. Some of our regular posters, like Marketsharer, Janus and Joop are doing a great job helping a lot of our less experienced (as well as more experienced!) webmasters with their questions. Recently, some Dutch webmasters have been dealing with the fact that their sites have been hacked. We've added a sticky post to the group to help webmasters clean up their sites after getting hacked, so if you are dealing with this issue, do have a look at the group! At this moment, the group is still steadily growing and we are really looking forward to reaching the 1000 members mark!
  
  ---
  Google Assistenza Webmaster
  Lella, Alessandro, Stefano
  The Italian webmaster group is about to turn one year old pretty soon and has been a great success. We have a bunch of fabulous witty power posters like Angelo Palma (Angelo), Marco (Customsoft) and others that keep on massively contributing to the success of the group day by day. Over the last year we achieved excellent results both in terms of traffic increase and new users posting frequently. Coordinating the localization of the post on paid links - later translated into 14 languages - was a turning point for the Webmaster Help team. So was Lella's blog post in English, German and Italian about the Search Engine Strategies Conference in London, where she represented Google as a speaker.
  
  ---
  
  Webmaster Help Group Guides and Domo-Kun, the team mascot
  
  Speaking on behalf of all our Google Guides, we are looking forward to continued collaboration with webmasters all over the world. Surely the upcoming 12 months are going to be exciting for the Google Help Groups as the communities continue to grow. Thanks again to all the great webmasters out there and see you soon in the discussions!
• 

  A new layer to Google Trends

  Posted: June 20th, 2008, 11:13am CEST by Maile Ohye

  Tags:  products and services  

  Written by R.J. Pittman, Director of Product Management - Consumer Search Properties
  
  Two years ago, we launched Google Trends, a tool that lets anyone see what the world is searching for, and compare the world's interest in your favorite topics. Last year, we added Hot Trends, which shows what people are searching for right now - the fastest rising search queries on Google, updated every hour. And just last week, we introduced normalized search volume numbers available for export in Google Trends.
  
  Today, we add a new layer to Trends with Google Trends for Websites, a fun tool that gives you a view of how popular your favorite websites are, including your own! It also compares and ranks site visitation across geographies, and related websites and searches.
  
  Let's take a look at one example, the release of Radiohead's In Rainbows album. As part of our annual Zeitgeist, we post the fastest rising search terms, and this past year, radiohead took the crown as the fastest rising search term in the last quarter of 2007.
  
  Using Google Trends, we can see the search volume for radiohead compared to in rainbows over the last 12 months.
  
  
  radiohead vs. in rainbows
  
  With Hot Trends, we can see that on October 10th (the release date of In Rainbows), people were most interested in downloading In Rainbows and reviews of the new release.
  
  Now, using Trends for Websites, this story can be viewed from another perspective: we can see how the number of unique visitors that visit radiohead.com and inrainbows.com has changed over the last 12 months, the countries where the sites are most popular, the top related sites and search terms.
  
  
  radiohead.com vs. inrainbows.com
  
  We can see that in October 2007, radiohead.com (the blue line) saw a huge surge in popularity. The release of In Rainbows clearly drove many people to visit the band's website. The album's website, inrainbows.com (the red line), saw an even more dramatic jump; probably because that is where you could actually download the new album. And, it looks like site traffic to radiohead.com has increased overall, even as inrainbows.com traffic declined.
  
  Keep in mind that Trends for Websites is a Google Labs product and that we are experimenting with ways to improve the quality of the data. Because data is estimated and aggregated over a variety of sources, it may not match the other data sources you rely on for web traffic information. For more information, be sure to check out our Website Owners FAQs.
  
  To start using Trends for Websites, head over to Google Trends.
• 

  Join us for another live chat - June 19, 2008

  Posted: June 16th, 2008, 9:46pm CEST by Adam Lasnik

  Tags:  feedback and communication  

  Written by Adam Lasnik, Search Evangelist
  When it comes to talking with webmasters, we just can't get enough.
  This past March we had the pleasure of connecting with hundreds of you online in our first-ever Webmaster Help Group online live chat, which included a presentation on Images in Google Search by Maile, lots of feedback from you on webmaster issues, a site clinic, and dozens of questions answered by folks from our Google Webmaster Central team.
  Given the success of this previous chat, we've decided to do it again.  We're hosting another free live chat (dubbed JuneTune), and we'd love to have you attend!Here's what you'll need:

  • About an hour of free time.
  • A phone, or ability to make a phone call online (e.g., with Skype or Gizmo5, etc.)
  • An internet-connected computer, and these additional specifications.
  • A URL for the chat, which you can only get when you register for the event  (you can likely do this in under 60 seconds)

  What will our JuneTune chat include?  

  • INTRO:  A quick hello from some of your favorite Help Group Guides
  • PRESO:  A presentation on Personalization in Google Search by our own John Mueller.
  • FAQs:  We're calling this "Three for Three," and we'll have three different Googlers tackling three different issues we've seen come up in the Group recently.  What will they be?  You'll just have to attend the chat to find out!
  • And lots of Q&A!  You'll have a chance to type questions during the entire session, and we'll pick as many as we can to answer in writing and in speaking during the chat.

  When and how can you join in?

  • Mark the date on your calendar now: Thursday, June 19, for about one hour starting at 2:00pm PDT / 5:00pm EDT / 21:00 UTC / 23:00 CET 
  • Register right now for this event. Please note that you'll need to click on the "register" link on the lefthand side.
  • Using the link e-mailed to you by WebEx (the service hosting the event), log in 5-10 minutes prior to 2pm PDT.

  
  We hope you can stop by, and look forward to chatting with you!  In the meantime, if you have any questions, feel free to post a note in this Groups thread.
• 

  Duplicate content due to scrapers

  Posted: June 9th, 2008, 5:40am CEST by Search Quality Team

  Tags:  crawling and indexing  

  Written by Sven Naumann, Search Quality Team
  
  Since duplicate content is a hot topic among webmasters, we thought it might be a good time to address common questions we get asked regularly at conferences and on the Google Webmaster Help Group.
  
  Before diving in, I'd like to briefly touch on a concern webmasters often voice: in most cases a webmaster has no influence on third parties that scrape and redistribute content without the webmaster's consent. We realize that this is not the fault of the affected webmaster, which in turn means that identical content showing up on several sites in itself is not inherently regarded as a violation of our webmaster guidelines. This simply leads to further processes with the intent of determining the original source of the content—something Google is quite good at, as in most cases the original content can be correctly identified, resulting in no negative effects for the site that originated the content.
  
  Generally, we can differentiate between two major scenarios for issues related to duplicate content:
  

  • Within-your-domain-duplicate-content, i.e. identical content which (often unintentionally) appears in more than one place on your site
    
  
  
  • Cross-domain-duplicate-content, i.e. identical content of your site which appears (again, often unintentionally) on different external sites
    

  With the first scenario, you can take matters into your own hands to avoid Google indexing duplicate content on your site. Check out Adam Lasnik's post Deftly dealing with duplicate content and Vanessa Fox's Duplicate content summit at SMX Advanced, both of which give you some great tips on how to resolve duplicate content issues within your site. Here's one additional tip to help avoid content on your site being crawled as duplicate: include the preferred version of your URLs in your Sitemap file. When encountering different pages with the same content, this may help raise the likelihood of us serving the version you prefer. Some additional information on duplicate content can also be found in our comprehensive Help Center article discussing this topic.
  
  In the second scenario, you might have the case of someone scraping your content to put it on a different site, often to try to monetize it. It's also common for many web proxies to index parts of sites which have been accessed through the proxy. When encountering such duplicate content on different sites, we look at various signals to determine which site is the original one, which usually works very well. This also means that you shouldn't be very concerned about seeing negative effects on your site's presence on Google if you notice someone scraping your content.
  
  In cases when you are syndicating your content but also want to make sure your site is identified as the original source, it's useful to ask your syndication partners to include a link back to your original content. You can find some additional tips on dealing with syndicated content in a recent post by Vanessa Fox, Ranking as the original source for content you syndicate.
  
  Some webmasters have asked what could cause scraped content to rank higher than the original source. That should be a rare case, but if you do find yourself in this situation:
  

  • Check if your content is still accessible to our crawlers. You might unintentionally have blocked access to parts of your content in your robots.txt file.
  
  
  • You can look in your Sitemap file to see if you made changes for the particular content which has been scraped.
  
  
  • Check if your site is in line with our webmaster guidelines.
    

  To conclude, I'd like to point out that in the majority of cases, having duplicate content does not have negative effects on your site's presence in the Google index. It simply gets filtered out. If you check out some of the tips mentioned in the resources above, you'll basically learn how to have greater control about what exactly we're crawling and indexing and which versions are more likely to appear in the index. Only when there are signals pointing to deliberate and malicious intent, occurrences of duplicate content might be considered a violation of the webmaster guidelines.
  
  If you would like to further discuss this topic, feel free to visit our Webmaster Help Group.
  
  For the German version of this post, go to "Duplicate Content aufgrund von Scraper-Sites".
• 

  Improving on Robots Exclusion Protocol

  Posted: June 3rd, 2008, 11:00am CEST by koppula

  Tags:  crawling and indexing  

  Written by Prashanth Koppula, Product Manager
  
  Web publishers often ask us how they can maximize their visibility on the web. Much of this has to do with search engine optimization -- making sure a publisher's content shows up on all the search engines.
  
  However, there are some cases in which publishers need to communicate more information to search engines -- like the fact that they don't want certain content to appear in search results. And for that they use something called the Robots Exclusion Protocol (REP), which lets publishers control how search engines access their site: whether it's controlling the visibility of their content across their site (via robots.txt) or down to a much more granular level for individual pages (via META tags).
  
  Since it was introduced in the early '90s, REP has become the de facto standard by which web publishers specify which parts of their site they want public and which parts they want to keep private. Today, millions of publishers use REP as an easy and efficient way to communicate with search engines. Its strength lies in its flexibility to evolve in parallel with the web, its universal implementation across major search engines and all major robots, and in the way it works for any publisher, no matter how large or small.
  
  While REP is observed by virtually all search engines, we've never come together to detail how we each interpret different tags. Over the last couple of years, we have worked with Microsoft and Yahoo! to bring forward standards such as Sitemaps and offer additional tools for webmasters. Since the original announcement, we have, and will continue to, deliver further improvements based on what we are hearing from the community.
  
  Today, in that same spirit of making the lives of webmasters simpler, we're releasing detailed documentation about how we implement REP. This will provide a common implementation for webmasters and make it easier for any publisher to know how their REP directives will be handled by three major search providers -- making REP more intuitive and friendly to even more publishers on the web.
  
  So, without further ado...
  
  Common REP Directives
  The following list are all the major REP features currently implemented by Google, Microsoft, and Yahoo!. With each feature, you'll see what it does and how you should communicate it.
  
  Each of these directives can be specified to be applicable for all crawlers or for specific crawlers by targeting them to specific user-agents, which is how any crawler identifies itself. Apart from the identification by user-agent, each of our crawlers also supports Reverse DNS based authentication to allow you to verify the identity of the crawler.
  
  1. Robots.txt Directives
  

  DIRECTIVE	IMPACT	USE CASES
  Disallow	Tells a crawler not to index your site -- your site's robots.txt file still needs to be crawled to find this directive, however disallowed pages will not be crawled	'No Crawl' page from a site. This directive in the default syntax prevents specific path(s) of a site from being crawled.
  Allow	Tells a crawler the specific pages on your site you want indexed so you can use this in combination with Disallow	This is useful in particular in conjunction with Disallow clauses, where a large section of a site is disallowed except for a small section within it
  $ Wildcard Support	Tells a crawler to match everything from the end of a URL -- large number of directories without specifying specific pages	'No Crawl' files with specific patterns, for example, files with certain filetypes that always have a certain extension, say pdf
  * Wildcard Support	Tells a crawler to match a sequence of characters	'No Crawl' URLs with certain patterns, for example, disallow URLs with session ids or other extraneous parameters
  Sitemaps Location	Tells a crawler where it can find your Sitemaps	Point to other locations where feeds exist to help crawlers find URLs on a site

  
  2. HTML META Directives
  

  DIRECTIVE	IMPACT	USE CASES
  NOINDEX META Tag	Tells a crawler not to index a given page	Don't index the page. This allows pages that are crawled to be kept out of the index.
  NOFOLLOW META Tag	Tells a crawler not to follow a link to other content on a given page	Prevent publicly writeable areas to be abused by spammers looking for link credit. By using NOFOLLOW you let the robot know that you are discounting all outgoing links from this page.
  NOSNIPPET META Tag	Tells a crawler not to display snippets in the search results for a given page	Present no snippet for the page on Search Results
  NOARCHIVE META Tag	Tells a search engine not to show a "cached" link for a given page	Do not make available to users a copy of the page from the Search Engine cache
  NOODP META Tag	Tells a crawler not to use a title and snippet from the Open Directory Project for a given page	Do not use the ODP (Open Directory Project) title and snippet for this page

  
  
  These directives are applicable for all forms of content. They can be placed in either the HTML of a page or in the HTTP header for non-HTML content, e.g., PDF, video, etc. using an X-Robots-Tag. You can read more about it here:X-Robots-Tag Post or in our series of posts about using robots and Meta Tags.
  
  Other REP Directives
  The directives listed above are used by Microsoft, Google and Yahoo!, but may not be implemented by all other search engines. In addition, the following directives are supported by Google, but are not supported by all three as are those above:
  
  UNAVAILABLE_AFTER Meta Tag - Tells a crawler when a page should "expire", i.e., after which date it should not show up in search results.
  
  NOIMAGEINDEX Meta Tag - Tells a crawler not to index images for a given page in search results.
  
  NOTRANSLATE Meta Tag - Tells a crawler not to translate the content on a page into different languages for search results.
  
  
  Going forward, we plan to continue to work together to ensure that as new uses of REP arise, we're able to make it as easy as possible for webmasters to use them. So stay tuned for more!
  
  Learn more
  You can find out more about robots.txt at http://www.robotstxt.org and at Google's Webmaster help center, which contains lots of helpful information, including:

  • How to create a robots.txt file
  • Descriptions of each user-agent that Google uses
  • How to use pattern matching
  • How often we recrawl your robots.txt file

  We've also done several posts in our webmaster blog about robots.txt that you may find useful, such as:

  • Using robots.txt files
  • All about Googlebot

  There is also a useful list of the bots used by the major search engines.
  
  To see what our colleagues have to say, you can also check out the blog posts published by Yahoo! and Microsoft.
• 

  How Google defines IP delivery, geolocation, and cloaking

  Posted: June 2nd, 2008, 1:06am CEST by Maile Ohye

  Tags:  crawling and indexing  

  Written by Maile Ohye
  
  Many of you have asked for more information regarding webserving techniques (especially related to Googlebot), so we made a short glossary of some of the more unusual methods.

  • Geolocation: Serving targeted/different content to users based on their location. As a webmaster, you may be able to determine a user's location from preferences you've stored in their cookie, information pertaining to their login, or their IP address. For example, if your site is about baseball, you may use geolocation techniques to highlight the Yankees to your users in New York.
    
    The key is to treat Googlebot as you would a typical user from a similar location, IP range, etc. (i.e. don't treat Googlebot as if it came from its own separate country—that's cloaking).
  
  
  • IP delivery: Serving targeted/different content to users based on their IP address, often because the IP address provides geographic information. Because IP delivery can be viewed as a specific type of geolocation, similar rules apply. Googlebot should see the same content a typical user from the same IP address would see.
    
    (Author's warning: This 7.5-minute video may cause drowsiness. Even if you're really interested in IP delivery or multi-language sites, it's a bit uneventful.)
  
  
  • Cloaking: Serving different content to users than to Googlebot. This is a violation of our webmaster guidelines. If the file that Googlebot sees is not identical to the file that a typical user sees, then you're in a high-risk category. A program such as md5sum or diff can compute a hash to verify that two different files are identical.
  
  
  • First click free: Implementing Google News' First click free policy for your content allows you to include your premium or subscription-based content in Google's websearch index without violating our quality guidelines. You allow all users who find your page using Google search to see the full text of the document, even if they have not registered or subscribed. The user's first click to your content area is free. However, you can block the user with a login or payment request when he clicks away from that page to another section of your site.
    
    If you're using First click free, the page displayed to users who visit from Google must be identical to the content that is shown to the Googlebot.

  Still have questions?  We'll see you at the related thread in our Webmaster Help Group.
• 

  Best practices when moving your site

  Posted: April 16th, 2008, 3:55pm CEST by Ríona MacNamara

  Tags:  crawling and indexing  

  Posted by Ríona MacNamara, Webmaster Tools Team
  

  Planning on moving your site to a new domain? Lots of webmasters find this a scary process. How do you do it without hurting your site's performance in Google search results?

  
  
  Your aim is to make the transition invisible and seamless to the user, and to make sure that Google knows that your new pages should get the same quality signals as the pages on your own site. When you're moving your site, pesky 404 (File Not Found) errors can harm the user experience and negatively impact your site's performance in Google search results.
  

  Let's cover moving your site to a new domain (for instance, changing from www.example.com to www.example.org). This is different from moving to a new IP address; read this post for more information on that.

  Here are the main points:

  • Test the move process by moving the contents of one directory or subdomain first. Then use a 301 Redirect to permanently redirect those pages on your old site to your new site. This tells Google and other search engines that your site has permanently moved.

  • Once this is complete, check to see that the pages on your new site are appearing in Google's search results. When you're satisfied that the move is working correctly, you can move your entire site. Don't do a blanket redirect directing all traffic from your old site to your new home page. This will avoid 404 errors, but it's not a good user experience. A page-to-page redirect (where each page on the old site gets redirected to the corresponding page on the new site) is more work, but gives your users a consistent and transparent experience. If there won't be a 1:1 match between pages on your old and new site, try to make sure that every page on your old site is at least redirected to a new page with similar content.

  • If you're changing your domain because of site rebranding or redesign, you might want to think about doing this in two phases: first, move your site; and second, launch your redesign. This manages the amount of change your users see at any stage in the process, and can make the process seem smoother. Keeping the variables to a minimum also makes it easier to troubleshoot unexpected behavior.

  • Check both external and internal links to pages on your site. Ideally, you should contact the webmaster of each site that links to yours and ask them to update the links to point to the page on your new domain. If this isn't practical, make sure that all pages with incoming links are redirected to your new site. You should also check internal links within your old site, and update them to point to your new domain. Once your content is in place on your new server, use a link checker like Xenu to make sure you don't have broken legacy links on your site. This is especially important if your original content included absolute links (like www.example.com/cooking/recipes/chocolatecake.html) instead of relative links (like .../recipes/chocolatecake.html).

  • To prevent confusion, it's best to make sure you retain control of your old site domain for at least 180 days.

  • Add your new site to your Webmaster Tools account, and verify your ownership of it. Then create and submit a Sitemap listing the URLs on your new site. This tells Google that your content is now available on your new site, and that we should go and crawl it.

  • Finally, keep both your new and old site verified in Webmaster Tools, and review crawl errors regularly to make sure that the 301s from the old site are working properly, and that the new site isn't showing unwanted 404 errors.

  We'll admit it, moving is never easy - but these steps should help ensure that none of your good web reputation falls off the truck in the process.
• 

  Crawling through HTML forms

  Posted: April 11th, 2008, 12:50pm CEST by Maile Ohye

  Tags:  crawling and indexing  

  Written by Jayant Madhavan and Alon Halevy, Crawling and Indexing Team
  
  Google is constantly trying new ideas to improve our coverage of the web. We already do some pretty smart things like scanning JavaScript and Flash to discover links to new web pages, and today, we would like to talk about another new technology we've started experimenting with recently.
  
  In the past few months we have been exploring some HTML forms to try to discover new web pages and URLs that we otherwise couldn't find and index for users who search on Google. Specifically, when we encounter a <FORM> element on a high-quality site, we might choose to do a small number of queries using the form. For text boxes, our computers automatically choose words from the site that has the form; for select menus, check boxes, and radio buttons on the form, we choose from among the values of the HTML. Having chosen the values for each input, we generate and then try to crawl URLs that correspond to a possible query a user may have made. If we ascertain that the web page resulting from our query is valid, interesting, and includes content not in our index, we may include it in our index much as we would include any other web page.
  
  Needless to say, this experiment follows good Internet citizenry practices. Only a small number of particularly useful sites receive this treatment, and our crawl agent, the ever-friendly Googlebot, always adheres to robots.txt, nofollow, and noindex directives. That means that if a search form is forbidden in robots.txt, we won't crawl any of the URLs that a form would generate.  Similarly, we only retrieve GET forms and avoid forms that require any kind of user information. For example, we omit any forms that have a password input or that use terms commonly associated with personal information such as logins, userids, contacts, etc. We are also mindful of the impact we can have on web sites and limit ourselves to a very small number of fetches for a given site.
  
  The web pages we discover in our enhanced crawl do not come at the expense of regular web pages that are already part of the crawl, so this change doesn't reduce PageRank for your other pages. As such it should only increase the exposure of your site in Google. This change also does not affect the crawling, ranking, or selection of other web pages in any significant way.
  
  This experiment is part of Google's broader effort to increase its coverage of the web. In fact, HTML forms have long been thought to be the gateway to large volumes of data beyond the normal scope of search engines. The terms Deep Web, Hidden Web, or Invisible Web have been used collectively to refer to such content that has so far been invisible to search engine users. By crawling using HTML forms (and abiding by robots.txt), we are able to lead search engine users to documents that would otherwise not be easily found in search engines, and provide webmasters and users alike with a better and more comprehensive search experience.
• 

  Speaking the language of robots

  Posted: March 27th, 2008, 2:27pm CET by Sean Harding

  Tags:  indexing and crawling  

  Written by Sean Harding, Webmaster Tools Team
  
  We all know how friendly Googlebot is. And like all benevolent robots, he listens to us and respects our wishes about parts of our site that we don't want crawled. We can just give him a robots.txt file explaining what we want, and he'll happily comply. But what if you're intimidated by the idea of communicating directly with Googlebot? After all, not all of us are fluent in the language of robots.txt. This is why we're pleased to introduce you to your personal robot translator: the Robots.txt Generator in Webmaster Tools. It's designed to give you an easy and interactive way to build a robots.txt file. It can be as simple as entering the files and directories you don't want crawled by any robots.
  
  Or, if you need to, you can create fine-grained rules for specific robots and areas of your site. Once you're finished with the generator, feel free to test the effects of your new robots.txt file with our robots.txt analysis tool. When you're done, just save the generated file to the top level (root) directory of your site, and you're good to go. There are a couple of important things to keep in mind about robots.txt files:

  • Not every search engine will support every extension to robots.txt files

  The Robots.txt Generator creates files that Googlebot will understand, and most other major robots will understand them too. But it's possible that some robots won't understand all of the robots.txt features that the generator uses.

  • Robots.txt is simply a request

  Although it's highly unlikely from a major search engine, there are some unscrupulous robots that may ignore the contents of robots.txt and crawl blocked areas anyway. If you have sensitive content that you need to protect completely, you should put it behind password protection rather than relying on robots.txt.
  
  We hope this new tool helps you communicate your wishes to Googlebot and other robots that visit your site. If you want to learn more about robots.txt files, check out our Help Center. And if you'd like to discuss robots.txt and robots with other webmasters, visit our Google Webmaster Help Group.