Google has never been good for academic research, but it's getting worse and bringing Bing along for the ride.
Let me give some background.
First, some stats. Statistics vary, but I'll just pick one source and you can go side-check it with your own research. As of February 2012, eBizMBA (http://www.ebizmba.com/articles/search-engines puts the battle like this:
- #1 Google with estimated 900,000,000 unique monthly visitors
- #2 Bing with estimated 165,000,000 unique monthly visitors
- #3 Yahoo! with estimated 160,000,000 unique monthly visitors
- #4 Ask with estimated 125,000,000 unique monthly visitors
- Everyone else 33,000,000 or fewer, dropping quickly to less than a million for #15 Alta vista.
Lesson #1: It pays to be number one. Google has WAY more visitors than ALL of their competition combined.
Second, Some history. In the early days of the Internet, there was no world wide web. That is to say, there was no graphical user interface to the Internet. We used tools like telnet and later FTP. A brilliant young man named Scott Yanoff compiled a list called Scott Yanoff's list. It was his attempt to list every resource on the Internet in a text file. That actually made sense when the Internet was small.
Most people received the list via email or Telnet from within their university. If you had access to a service called Finger, you could "finger [email protected]" and capture the list. By the time the Web came along and started to grow in the early 1990's, actual search tools had come into existence. Archie and WAIS allowed academics and other resourceful people to scour machines and find stuff.
True search engines came along with Excite, Alta Vista, and Yahoo. These search engines use "bots" to go find information and index it. That led to amazing attempts (mostly by the porn industry) to trick the search engines into higher rankings. That in turn led to search engines becoming more and more sophisticated about finding what people were really looking for. Later, everyone who claimed to be an SEO Wizard made the battle even more difficult.
Sidecheck: As an academic, trying to find the "reality" of what's going on in the world, search engines have never been very good. In these early days, if you could filter out the porn hits, you had a pretty good idea. But it was more likely than not that the information you wanted was not on the Internet! Now we assume almost everything is on the Internet (it's not). But back then you had to assume that most information was not.
Third, Enter Google. Google changed the game. Instead of trying to simply match "keywords" with search terms, Google did deep-dive analysis of what web sites were trying to say. For example, if I have a link that reads Best Sandwich in The World and I link to your sandwich shop, Google knows that I think your sandwiches are the best. That increases your ranking for the search term "Best Sandwich."
If I just link to your site and Cousin Larry's Sandwich Shop, that does you very little good. It increases your in-bound links. Good. It lists you by name. Good. But if people are not searching for you and only searching for the best sandwich, it does you no good.
Important lesson. When you ask people to post a link to your site, you MUST give them a recommended link format that maximizes your keywords. For example, I might ask you to link the term
"Managed Service Agreements"
Google interprets that and knows that someone considers that page to contain authoritative information on Managed Service Agreements. This is HUGE in the search business. It means that Google was actually using the evolving sense of subject matter authority to return search results.
Of course the SEO spammers continue to trick the system. Google keeps evolving. And the uniqueness that catapulted Google to the top has been eroded.
Sidecheck: In the academic search for "reality" Google plays an interesting role. If you're looking for who is the best, who knows the most, and who is considered the one true source of information, Google might get you there. But if you're looking for hard, cold facts, Google is a step in the wrong direction. It's more like a popularity contest than a tool for finding the truth behind the reflections of reality we see every day.
Fourth, Enter Bing. Bing took a different approach. Bing was introduced as a "decision engine." In other words, they assume that you're looking for something so you can make a decision. Where to live, which train to take, which shoes to buy, etc. So Bing returns a variety of results that are supposed to help you make a decision.
Clearly, this is not helpful in the search for who really invented the telegraph or how many people have died from Dihydrogen Monoxide abuse.
Luckily, Bing has evolved to be more or less just another search engine. So it returns much more useful results than it did three years ago. In the search for "reality," however, it has had no effect.
Google will now try, whenever possible, to gather up your activities from various Google products and throw them into consideration when you search. So if you searched for certain things in the past, that will affect your searches in the future. If you read certain blogs, post certain pictures, or post replies or comments on web sites, that's taken into consideration. The keywords you use in your Gmail, the Google+'s you plus, and the Google Docs you use will all affect your search results. All of your activity will be taken into consideration in order to tailor the advertisements you see and the search results you receive.
Google's goal is to give you a world wide web experience that includes the things you like, the things you enjoy, and the things you're familiar with. That allows them to give you unique and interesting results that no one else will experience. If the world is defined as spending money in order to express your interest, then Google wants to give you a very fulfilling experience.
The web will literally be molded to fit your needs and desires.
And that's cool, but it seems to me there there are two big reasons this is bad. First, it means that we will see fewer and fewer sites that are "interesting" because they are unexpected. Our experience will become more and more homogenized as we use Google products more and more. For many people, the fun of the Internet includes stumbling onto things that are new and fun and different. It sounds like we'll see less of that.
The second reason this change is bad is that reality will be even harder to find. Because our future search experience will be so completely colored by our past Internet experiences, Google will move further away from being a useful research tool. Now, granted, Google doesn't want to be a research tool. It's not their mission. They've never presented themselves as that.
When my daughter was little, I told her time and time again that Internet research needs to be backed up with real, hard core, facts. Anyone can put up any crap they want on the Internet. And if someone believes it, then it becomes just another competing version of reality.
Most of us are not academics. But we all do research from time to time. Every good anthropologist and social scientist knows that we change reality simply through the process of observing it. So maybe computer-aided search tools will never get us to the "truth" in the real world. Maybe reality will always go to the highest bidder or the most popular page.
But when we really need to do hard-core research, we need to choose the right tools for the job. Popular, commercial search engines are NOT the right tools for this job. When you're serious about serious background information and fact-seeking, there are many alternatives. Luckily, you have hundreds of search engines to choose from. Some of the best are highly focused on one specific topic or set of resources.
A great list of search engines can be found here: Noodle Tools: Choose the Best Search for Your Information Need. It is also a good idea to talk to a Librarian. They do this for a living.
Please don't assume that Google, Bing, or Yahoo will give you the information you need. For many of us in marketing, they are just right. But for real research, they are as far from reality as you can get.