Article Library
Make your dynamic web site
search engine friendly

Bookmark This Page

Back to Article Library

Make Your Dynamic Web site Search Engine Friendly
by Brad S. Konia

Do you have a dynamic Web site or are thinking of building one? If so, failing to make your dynamic site search engine friendly can dramatically reduce your visibility in the search engines. Even though some engines spider dynamic URLs, others still do not. Those that do spider them may still place artificial limits on how deep they will travel within such links.

A dynamic site is one in which the pages are generated on the fly, usually from a database. You can often recognize one by seeing symbols such as a question mark, ampersand or other special symbols in the URL.

Let's say that you're a web designer and a client asks you to build a Web site for him to sell his products online. If the client has one or two products, then a static HTML site would be all that's needed. But what if the client has a database containing hundreds or thousands of products? To build a Web site like that using static HTML pages would involve creating a separate page for each product, meaning that you might have to create thousands of pages.

A dynamic Web site can solve this problem. Unfortunately, many Web sites make use of this technique without fully realizing the dangers relating to search engine visibility.

There are programs available that will automatically generate static template-based HTML pages from a database, but for a variety of reasons, this is usually not the best approach. Most designers prefer to work with active server technologies such as PHP or ASP to create truly dynamic Web sites. This saves time and simplifies maintenance.

The only problem with dynamic Web sites is that they're not nearly as search engine friendly as static sites. Some engines will not index them or will index only a limited number of pages. If you've already spent a lot of money to have a dynamic site designed and built, you're not going to want to scrap all that work and start over with a static HTML site.

So how can you retain the functionality of your dynamic Web site, but make it search engine friendly? First, let me give you an example of how a typical dynamic Web site functions.

Suppose that you had a real estate Web site in which customers could view all of your available properties online. Instead of creating a separate HTML page for each property, you could put all the information for your properties into a database. The database might contain fields like:

• Property ID
• Property Name
• Address
• City
• Asking Price
• Owner
• Pictures,etc...

Your Web site would most likely include some type of search form so that your users could search for a list of properties that fulfill various criteria. For instance, someone might search for a list of properties in Miami, Florida with asking prices below $300,000.

After conducting this search, the results page would contain a list of properties that meet the criteria and perhaps a thumbnail picture of each property. Then the user could click on the property name or picture to view more detailed information.

On a typical dynamic Web site, the hyperlinks to the property detail pages would contain URLs similar to the following:

http://www.xyzWeb site.com/property-detail.php?id=57

The property-detail.php page is a dynamic page that the server builds on the fly. The information after the question mark in the URL is passed to the server so it knows which property to display on the detail page. In this case, it will display detail information for property #57. Therefore, this single page can actually display detailed information for an unlimited number of properties. This is an extremely efficient way to manage your Web site, but unfortunately, some search engines will not index dynamic pages.

There are two basic problems in indexing a dynamic Web site:

1. Search engines will not utilize your search box.

   If the only way to access your dynamic content is by first conducting a query on your site, a search engine will miss this content entirely! You must provide access to all your Web pages by doing nothing more than navigating links. This means that you'll need to have one or more pages organized by product or category that eventually drills down to every page in your database that you may want indexed by a search engine. Without this, an engine will never "see" the vast content that your site has to offer.

   Offering both methods of locating information on your site will also help improve your site's navigation. Notice how Yahoo.com gives visitors an option to conduct a site search or to browse its catalog by traveling a series of links.

2. Some engines won't index pages containing a question mark or other special symbols. Or, they may limit how deep they will spider such sites.

   How can we get around this problem? The solution differs depending on what server software your Web site runs. Ask your system administrator or hosting service if you are unsure whether your site runs on Apache, Microsoft IIS, or another server.

   For Apache-based Web sites...

   Most Unix and Linux sites run a server application called Apache. For these people, the solution to the dynamic site dilemma is in the Apache module called "mod-rewrite." Mod-rewrite is a powerful scripting program that will translate URLs based on the patterns that you define. In layman's terms, this will allow you to feed the search engines URLs that appear to be static, but are actually dynamic.

   As an example, consider the dynamic URL: http://www.yourWeb site.com/yourscript.php?id=123.

   The above URL passes the variable called "id" to a script called yourscript.php. This script builds a dynamic page based on the product ID. How can we make this more search engine friendly?

   With mod-rewrite, you can get the same result using the following URL:

   http://www.yourWeb site.com/productid123.htm

   Notice how the "offending" question mark symbol has been removed from the URL. This second URL is much more search engine friendly. In addition, by using the mod-rewrite technique, it will function exactly the same as the first URL.

   When mod-rewrite sees "productid123.htm", it knows to translate that into "yourscript.php?id=123." This translation takes place behind the scenes, so the URL in the browser's address bar will continue to display "productid123.htm" while your database program sees the URL it is expecting to see. It is important to understand that there is no re-direction taking place. It's simply a URL translation.

   Rather than getting into a detailed technical description of how this all works, I'll give you step-by-step instructions on how to configure this on your Apache-based server. The main requirement is that your site be hosted on a server running the Apache web server software and that the mod-rewrite module is installed (it usually is).

   Apache is by far the most popular web server for Linux, so if your site is hosted on a Linux server, chances are it's running Apache. Apache is also available for Windows, but most Windows servers use Microsoft's IIS Web Server. If your site is hosted on a Microsoft IIS server, then see the section of this article addressing Microsoft servers.

   The instructions below assume that you're using PHP as your scripting language, but this could easily be adapted to any scripting language.

Below are the instructions to implement mod-rewrite. If you are not experienced in Web site development and scripting, I recommend you forward this article to someone with expertise in this area. For an experienced Web developer or programmer, these changes should not take long to implement. In many cases, your hosting service may be able to make the changes for you although they may charge a fee.

Instructions:

1. Open Notepad on your Windows machine (or a comparable text editor if you're using a Macintosh). Avoid using MS Word or any type of word processor because these programs add extra formatting characters by default that will cause problems.
2. Copy and paste the following text into Notepad:
   
   RewriteEngine on
   RewriteBase /basedir
   RewriteRule ^productid([^.]+).*$ yourscript.php?id=$1 [T=application/x-httpd-php]
   
   
3. Change /basedir to the name of the directory containing your dynamic pages. This will normally be just a "/" unless the pages are in a subdirectory, in which case it would be a "/" followed by the name of the subdirectory.
   
4. In the "RewriteRule" line, after the question mark, change "id" to whichever variable you're using to pass your product id in order to display the product detail page. Navigate to the link on your site that displays a product detail page from your