A basic robots.txt

When a crawler visits your site, it first looks for a robots.txt file placed in the root of your domain that instructs it on which pages it should ignore. Such a file is made of one or more records, and each must contain a line to address a certain user agent followed by one or more Disallow lines. The syntax is therefore trivial – you don’t need to learn more than these two directives. For example, a robots.txt file made of

would tell Google Bot not to crawl neither http://yourdomain.com/login.php nor http://yourdomain.com/admin.

Wildcards – not so wild

User agents may be matched by a wildcard. Instead of having a User-agent: spidername section for each crawler, you can instruct them all to follow the subsequent Disallow lines by using User-agent: *. The star symbol matches any number of characters – so “spider*” can stand for “spider A”, “spiderFromSomeSite” or “spiderFromThatOtherSite”. A question mark would match one character, so “spider?” will work for “spiderA”, “spiderB”, but not “spider X”.

Follow the syntax strictly

Robots.txt can be picky with syntaxes, so make sure you follow the structure:

• Don’t mix and match. User-agent comes before Disallow, it won’t work if you put them the other way around.
• Don’t use more than one URL in the Disallow line. “Disallow: /path1 /path2 /path3” won’t work; you are to put each path under its own Disallow rule.
• Keep case sensitivity in mind. “/path1” and “/Path1” are two different URLs.
• Inline comments don’t work. You might be used to placing comments after a “#” sign at the end of the row from Perl, PHP or shell scripting. With robots.txt, each comment has to be on its own line. For example, “Disallow: /admin # don’t index /admin” might confuse some spiders who will be looking for a “/admin#don’t” folder.
• There is no “Allow” directive. As outlined in the beginning, the syntax of a robots.txt file is as easy as it can get. To allow spiders to index every file, simply place an empty Disallow line.

Beware of sensitive information

As a final note, remember that you shouldn’t use robots.txt to keep spiders from accessing overly sensitive information. Just because a standards-compliant web spider won’t access it, it doesn’t mean a malicious one (or even a human user) can’t or won’t. Such a setting is a no-no:

Sensitive information such as users and passwords should be placed outside your htdocs path. Just because Google won’t crawl your passwords.txt file it doesn’t mean a malicious user won’t open up /admin/passwords.txt in a browser and read your “hidden” content.

Back in the days of PHP 4.x reading an RSS stream would have involved opening a TCP connection to the URI, reading the content and parsing it with an endless stream of regular expressions. PHP 5 allows you to do it seamlessly in a few lines of code with the help of simplexml_load_file().

Prerequisites

You will need the libxml extension installed. To check for it, point your browser to the phpinfo page and look for something similar to the image below:

If the extension is not enabled, you will need to recompile PHP with the –enable-libxml argument (or ask your web hosting provider to set it up for you if you are on a shared host). Also, since you will be opening remote XML feeds, you will need allow_url_fopen = on in your php.ini file.

Reading the remote feed with simplexml_load_file()

Getting the RSS feed is as simple as

For this example, we will be using the Lifeline Design feed. Provided there are no errors, $rss is now a SimpleXMLElement object, with one property we are interested in: $rss->channel contains all information in the RSS feed. $rss->channel is a SimpleXMLElement object as well, with the following properties:

• title: The title of the blog—the one that you see at the very top of your browser window when you open the front page. In our case, it will be “Lifeline Blog.”
• link: The URL of the blog—”http://www.lifelinedesign.ca/blog”
• description: Description of the site. It might not be relevant for the blog, since a lot of WordPress themes don’t take it into account. With LLD, the description is the standard “Just another WordPress weblog.”
• lastBuildDate: The time and date of the last update, namely the latest post published.
• generator: Usually the name and version of the script the site is running on. With WordPress blogs, this string is usually in the form of “http://wordpress.org/?v=x.x.x”
• language: The language the site runs on, in locale format.
• item: An array containing the posts offered in the RSS feed.

To get the actual posts, you will need to loop through the $rss->channel->item object, which is also an instance of SimpleXMLElement, and read the following properties:

• title: the title of the article;
• link: the Permalink to the post;
• comments: the URL for comments on the article (with most WordPress installations this would be http://example.com/permalink-to-article/#comments)
• pubDate: self-explanatory, the date and time when the article was published;
• category: a SimpleXMLElement object with the category under which the article lies;
• guid: the URL of the article in standard canonical format (http://example.com/?p=article_id);
• description: depending on the settings on that site, this will contain either a preview or the full article.

The code

As outlined in the beginning, as little as a few lines of code can do the job:

Of course, there is still room for improvement (for example, if there is no feed at that URL, libxml will throw a set of nasty warnings), yet the code can be used to jumpstart your own RSS reader.

Yesterday evening, I decided to “brand” my phone to Lifeline a bit more, and made a custom wallpaper for the phone:

When two rows of icons are setup, the logo fits nicely between them. I’m pretty happy with my handiwork, but it got me thinking that maybe other people out there would like to create a background for their G1 with their company logo on it, so I’ve created a Photoshop source file along with some explanation below to show you how to create your own!

First of all, take a look at the image below, it’s a picture of the source file (not to scale):

In the actual source file, the coloured lines are just set up as guides, but for the example image I’ve coloured them to help make the explanation easier.

The red line is the middle of the entire wallpaper, so you’d want your logo to be centered horizontally on this line. The while lines represent the edge of the main screen. If you have an HTC-Dream, you’ll know that you can switch to alternate pages of icons on the left and right. For example, if I switched to the alternate icon page on the left side, then the phone would display from the left edge of the image all the way to the red line in the center.

The green lines represent the vertical space you’ll want to fit your logo in to allow a row of icons above and below your logo without obstructing or crowding it. So, you’ll want to fit your logo in the area shaded blue. The Photoshop file has a black background currently, but feel free to change it to anything you’d like! You can download it by clicking here.

Have fun making your own HTC-Dream wallpapers!