How to make your dynamic web site
search engine friendly. Techniques used by the experts to increase visibility in the search engines.

Bookmark This Page | Back to Article Library

Make Your Dynamic Web site Search Engine Friendly
by Brad S. Konia

Do you have a dynamic Web site or are thinking of building one? If so, failing to make your dynamic site search engine friendly can dramatically reduce your visibility in the search engines. Even though some engines spider dynamic URLs, others still do not. Those that do spider them may still place artificial limits on how deep they will travel within such links.

A dynamic site is one in which the pages are generated on the fly, usually from a database. You can often recognize one by seeing symbols such as a question mark, ampersand or other special symbols in the URL.

Let's say that you're a web designer and a client asks you to build a Web site for him to sell his products online. If the client has one or two products, then a static HTML site would be all that's needed. But what if the client has a database containing hundreds or thousands of products? To build a Web site like that using static HTML pages would involve creating a separate page for each product, meaning that you might have to create thousands of pages.

A dynamic Web site can solve this problem. Unfortunately, many Web sites make use of this technique without fully realizing the dangers relating to search engine visibility.

There are programs available that will automatically generate static template-based HTML pages from a database, but for a variety of reasons, this is usually not the best approach. Most designers prefer to work with active server technologies such as PHP or ASP to create truly dynamic Web sites. This saves time and simplifies maintenance.

The only problem with dynamic Web sites is that they're not nearly as search engine friendly as static sites. Some engines will not index them or will index only a limited number of pages. If you've already spent a lot of money to have a dynamic site designed and built, you're not going to want to scrap all that work and start over with a static HTML site.

So how can you retain the functionality of your dynamic Web site, but make it search engine friendly? First, let me give you an example of how a typical dynamic Web site functions.

Suppose that you had a real estate Web site in which customers could view all of your available properties online. Instead of creating a separate HTML page for each property, you could put all the information for your properties into a database. The database might contain fields like:

• Property ID
• Property Name
• Address
• City
• Asking Price
• Owner
• Pictures,etc...

Your Web site would most likely include some type of search form so that your users could search for a list of properties that fulfill various criteria. For instance, someone might search for a list of properties in Miami, Florida with asking prices below $300,000.

After conducting this search, the results page would contain a list of properties that meet the criteria and perhaps a thumbnail picture of each property. Then the user could click on the property name or picture to view more detailed information.

On a typical dynamic Web site, the hyperlinks to the property detail pages would contain URLs similar to the following:

http://www.xyzWeb site.com/property-detail.php?id=57

The property-detail.php page is a dynamic page that the server builds on the fly. The information after the question mark in the URL is passed to the server so it knows which property to display on the detail page. In this case, it will display detail information for property #57. Therefore, this single page can actually display detailed information for an unlimited number of properties. This is an extremely efficient way to manage your Web site, but unfortunately, some search engines will not index dynamic pages.

There are two basic problems in indexing a dynamic Web site:

1. Search engines will not utilize your search box.

   If the only way to access your dynamic content is by first conducting a query on your site, a search engine will miss this content entirely! You must provide access to all your Web pages by doing nothing more than navigating links. This means that you'll need to have one or more pages organized by product or category that eventually drills down to every page in your database that you may want indexed by a search engine. Without this, an engine will never "see" the vast content that your site has to offer.

   Offering both methods of locating information on your site will also help improve your site's navigation. Notice how Yahoo.com gives visitors an option to conduct a site search or to browse its catalog by traveling a series of links.

2. Some engines won't index pages containing a question mark or other special symbols. Or, they may limit how deep they will spider such sites.

   How can we get around this problem? The solution differs depending on what server software your Web site runs. Ask your system administrator or hosting service if you are unsure whether your site runs on Apache, Microsoft IIS, or another server.

   For Apache-based Web sites...

   Most Unix and Linux sites run a server application called Apache. For these people, the solution to the dynamic site dilemma is in the Apache module called "mod-rewrite." Mod-rewrite is a powerful scripting program that will translate URLs based on the patterns that you define. In layman's terms, this will allow you to feed the search engines URLs that appear to be static, but are actually dynamic.

   As an example, consider the dynamic URL: http://www.yourWeb site.com/yourscript.php?id=123.

   The above URL passes the variable called "id" to a script called yourscript.php. This script builds a dynamic page based on the product ID. How can we make this more search engine friendly?

   With mod-rewrite, you can get the same result using the following URL:

   http://www.yourWeb site.com/productid123.htm

   Notice how the "offending" question mark symbol has been removed from the URL. This second URL is much more search engine friendly. In addition, by using the mod-rewrite technique, it will function exactly the same as the first URL.

   When mod-rewrite sees "productid123.htm", it knows to translate that into "yourscript.php?id=123." This translation takes place behind the scenes, so the URL in the browser's address bar will continue to display "productid123.htm" while your database program sees the URL it is expecting to see. It is important to understand that there is no re-direction taking place. It's simply a URL translation.

   Rather than getting into a detailed technical description of how this all works, I'll give you step-by-step instructions on how to configure this on your Apache-based server. The main requirement is that your site be hosted on a server running the Apache web server software and that the mod-rewrite module is installed (it usually is).

   Apache is by far the most popular web server for Linux, so if your site is hosted on a Linux server, chances are it's running Apache. Apache is also available for Windows, but most Windows servers use Microsoft's IIS Web Server. If your site is hosted on a Microsoft IIS server, then see the section of this article addressing Microsoft servers.

   The instructions below assume that you're using PHP as your scripting language, but this could easily be adapted to any scripting language.

Below are the instructions to implement mod-rewrite. If you are not experienced in Web site development and scripting, I recommend you forward this article to someone with expertise in this area. For an experienced Web developer or programmer, these changes should not take long to implement. In many cases, your hosting service may be able to make the changes for you although they may charge a fee.

Instructions:

1. Open Notepad on your Windows machine (or a comparable text editor if you're using a Macintosh). Avoid using MS Word or any type of word processor because these programs add extra formatting characters by default that will cause problems.
2. Copy and paste the following text into Notepad:
   
   RewriteEngine on
   RewriteBase /basedir
   RewriteRule ^productid([^.]+).*$ yourscript.php?id=$1 [T=application/x-httpd-php]
   
   
3. Change /basedir to the name of the directory containing your dynamic pages. This will normally be just a "/" unless the pages are in a subdirectory, in which case it would be a "/" followed by the name of the subdirectory.
   
4. In the "RewriteRule" line, after the question mark, change "id" to whichever variable you're using to pass your product id in order to display the product detail page. Navigate to the link on your site that displays a product detail page from your database. Study the URL in your browser. Normally you'll only have a question mark followed by the product ID. However, if other parameters exist, you'll need to rewrite those too.
   
5. In the "RewriteRule" line, change yourscript.php to the name of your dynamic product detail page. This will be the script name seen in the URL often ending in ASP, PHP, or various other extensions. The script name will normally precede the question mark.
   
6. If your site does not use PHP, change "T=application/x-httpd-php" to the MIME type for the language that you use.

   If you have questions, the technical documentation for the mod-rewrite functions can be found at: http://httpd.apache.org/docs/mod/mod_rewrite.html
   

7. Save the file to your local computer. Name it htaccess.txt.

8. Upload the file to your Web site in ASCII mode.

9. Configure your FTP program to display hidden files. Hidden files are files that start with a dot on UNIX or Linux based operating systems. Most FTP programs have an option to display hidden files either on the preference screen or in the settings for the individual FTP site. Some FTP programs will allow you to add parameters to the list command that will display hidden files. In that case, the parameter that you would need to add to the ls command is "-al" (without the quotes).

10. Test your FTP program to make sure you can view hidden files. Try renaming a non-essential file on your Web site to a filename that starts with a dot. Make sure that it shows up in the directory listing. Then make sure you can rename it back to its original file name.

11. Rename htaccess.txt to .htaccess (starts with a dot, and no file extension). This will activate your changes.

CAUTION: If you have done something wrong in the htaccess file, your site may stop working after you rename the file. In that case, rename .htaccess back to htaccess.txt and the site should start working again. That's why it's important that your FTP program is configured to see hidden files.

If you set things properly, at this point your web server will automatically rewrite URLs of the format: http://www.yourWeb site.com/yourscript.php?id=123 to: http://www.yourWeb site.com/productid123.htm

The rewriting takes place behind the scenes, so the URL in the address bar will always display in the new format, without the question mark.

For Microsoft IIS Web sites...

If your Web site runs on the Microsoft IIS server, you can obtain the same basic functionality I described for Apache Web sites using a program called ISAP rewrite. In the interest of brevity, I'll simply point you to the Web site offering more information on this program.

Read this article regarding Apache servers and then visit the above link. In this context, you should understand how you could apply ISAP rewrite to accomplish the same goal as Apache's mod-rewrite.

Whether you're running a Microsoft or Apache server, test your changes by browsing and searching your site to see if it worked. Instead of having URLs containing question marks followed by the product ID, the product ID should become part of the page name. Ampersands and commas should also be eliminated whenever possible. Hyphens, underscores, and periods should be fine.

Once you have verified that the change worked as expected, you must go into your site and change the URLs that point to each of your detail/product pages so that they conform to the new format. This is critical since these are the links that the search engines will follow to find your product pages. If you leave them using the old syntax containing the troublesome symbols, then the search engines may still avoid spidering the links.

If the bulk of your dynamically generated pages are not being indexed today, then don't short-change yourself any longer. Implementing this one change on your Web server can improve your site's potential visibility by a hundred-fold or more! The next step is to make sure those detail pages are properly optimized so that they'll rank well. WebPosition Gold's Page Critic is the perfect match for this task.