Tips on Using Search Engines

"The biggest library in the world and the most disorganised"

This is an oddball page for a security and privacy site, but it sure is useful.

There are three sections:
Easy searches
Difficult searches
Next to impossible searches

Easy searches

The results you get from different search engines vary wildly. For the best results, run your searches on the best engines. In my experience, the top three are Google, NorthernLight, and AltaVista.

Use multiple word search terms. Single words rarely give satisfactory results. Most times they'll return lots of junk. The more words you use, the more precise the results will be. Here's an example that clearly demonstrates the need for multiple search terms:

Genesis music
Genesis bible
Genesis town USA

A general search for Genesis alone would return plenty of junk, whichever meaning you were searching for.

And that's it for easy searches. Ninety percent of the time, these two simple guidelines will bring the result you're looking for. Now let's move on to difficult searches.

Difficult searches

Assuming you've followed the guidelines in easy searches but still haven't got a result, what next?

Try your search terms in three more search engines, AskJeeves, Hotbot, and Webcrawler.

It takes a while to follow the leads in the drop-down boxes of AskJeeves, but gems are often hidden there.

Still no luck? It may be worth giving the directories a try. Unfortunately they can take up a lot of time. If you're searching for a broad selection of information, perhaps background stuff, they can be very useful. But if you're looking for the answer to a specific question, your chances aren't good.

The two recommended directories are The Open Directory (dmoz.org, which is mirrored in many search engine directories, for example in Google) and Yahoo!.

Changing your search terms

If you've used the top six engines and still haven't found the specific information you're looking for, you need to change your search terms.

Here are some tips on search terms

If your search is country-specific, try including the country name or initials as your final search term. For example, online groceries uk.

Some words are commonly ignored by the search engines even when you ask for them to be included, such as - to, for, Internet, web. You may be able to find a way round this. One example is that instead of using Internet you can try online.

Force a word to be included in a search by putting a + sign in front of it. Do this if a search engine is returning lots of junk results that don't include one of your essential keywords. Examples: +web page designer, and online groceries +uk

You can force a word to be excluded by putting a minus sign (-) in front of it. Do this when search engines go down an obvious blind alley. Classic example: exotic chicks -sex -women. Chinese ducklings will otherwise come way down the results for exotic chicks.

Putting quotes around a string of words forces a search engine to look for that complete string. This can be very useful. "A Room With A View" should return more literary sites than apartment lettings. Google may drop words within quotes, but most search engines don't.

Try to avoid capital letters and plurals, though you might eventually turn to them if you've had no luck with lower-case singulars. There are some obvious exceptions such as trousers and groceries. Use these plurals early.

Sometimes, particularly with technical searches, you may not know the correct search words to use. In this case devote the first part of your search to tracking down the right words. Find pages that are close to useful and read them. Pick up new keywords from them and try these out. For example if you want highly technical info on Internet security you'll need to use the search term bugtraq, but you probably won't come across this word until you've already read a few deeply technical security pages.

Searches within large sites

If there are large sites that deal with the kind of information you're looking for, try searches within those sites. For example, I often search on specific Internet security questions within Wired, ZDNet and CNET rather than using a regular Web-wide search engine.

If you've spent around 30 minutes following these guidelines, and you still haven't found what you're looking for, it's probably time to move on to Next to impossible searches.

Next to impossible searches

If you've followed the guidelines for difficult searches and you're still stuck, it's time to slow down and recognise you're in for a marathon not a sprint.

There are two classic problems that make information hard to find on the Internet. The first is that it's buried deep in an area where the search engines don't look (they index only a small proportion of the Internet) and the second is that it's obscured by something close in meaning but far more popular. For example scientific sex research is obscured by pornography.

One of the joys of searching is cracking these tough nuts. It may take an hour and a half, but when you make the breakthrough it gives a pleasant feeling of achievement.

Human involvement

You've now exhausted the limits of collective computer intelligence and need to apply something far superior, collective human intelligence.

Ask yourself, is it likely that somebody else has already found the information source you're looking for and included it on their links page?

If the answer is no, your search just got harder. But if you can imagine the perfect link exists on somebody else's links page, this is what you should be searching for next. Not the information itself, but the link to it.

Take your best search terms and add on +link, or alternatively +links. You'll get people's link pages returned. It's a neat trick and often brings results. It's almost guaranteed to get you closer to your goal.

Reverse linking

Here's another curious trick, again based on human sorting. The starting point is the best sites and pages you've found so far, the ones that came closest to meeting your needs. What you need to do with these is find out who has linked to them, because there's a good chance they'll have linked to something even better.

To find out who has linked, copy the URL of a nearly-but-not-quite page or site, and paste it into the search field of a good search engine, preceded by link:

Don't forget the colon. Example - link:www.tinhat.com. With luck, the results will include some excellent sites that bring you a lot closer to your goal.

Guessing strings

This technique is especially suited to the kind of searches that are obscured by something close and immensely popular - anything to do with sex, sport and other human obsessions.

This is best shown by example. Let's say we want to find out how much money the online sex industry made last year. This is a tough search. Once you enter sex or adult into the search field of a search engine you can imagine the results, though they can be improved by switching on the family filter (or whatever it's called in that particular engine - it's usually linked from the opening page).

The best approach is probably to find sites specialising in industry statistics and search those, using the Web-wide engines only to identify those sites, or possibly ignoring the engines altogether and reaching the best sites via directories. But if you're lucky you might be able to get a speedy result using the Web-wide engines and very specific search strings.

We have to guess exactly how the information will be presented in the site that has the answer. Our guess might be that the author writes - The online sex industry made $X billion last year. So we run a string search (using quotemarks) on "the online sex industry made". If this doesn't work, we might try "revenue from adult sites", again anticipating the precise wording of the information we're looking for.

I've managed to get a result from this many times, but I have to admit that it's usually in desperation.

We've now covered all the tips I know. Hope you didn't need to read this far.