Web search and enterprise search are coming together
Web search and enterprise search are in many ways fundamentally different problems. The biggest problem in web search is screening out pages that deliberately pretend to be relevant to a search. The second biggest problem is picking out the crème de la crème from a long list of essentially good hits. In enterprise search, on the other hand, the biggest problem is finding a single document, or single fact, that is lonely at best, and if you’re unlucky doesn’t exist in the corpus at all. Document structures are also completely different, as are linking structures and almost every other input to the ranking algorithms except the raw words themselves.
Even so, the businesses and technologies of web and enterprise search are beginning to combine. Google’s attack on the low end of enterprise search is well-known, of course, as are Microsoft’s increasingly well-publicized ambitions. But enterprise search companies are also reaching out to the Web. Convera has gotten the most press for this strategy, offering focused web search to the same customers (mainly intelligence law enforcement agencies) that bought its enterprise product RetrievalWare. This is a great fit for Convera, both in customers (a lot of what those agencies have done all along is filter news information) and technology (their key differentatior is their detailed taxonomies, and those can help in any kind of search).
But it’s not just Convera. FAST of course sold alltheweb.com, which is now owned by Yahoo, and is barred by non-compete agreement from getting back into the web search business. Even so, it is spidering and analyzing and perhaps filtering billions of web pages, and offering the results as a service to its enterprise customers. These customers then have a huge leg up in deciding which pages to spider themselves and index with FAST’s enterprise technology, and they have access to FAST’s metadata banks to help with the ranking of those pages once spidered. Clever!
I think that Autonomy is doing something along these lines too, but I’m devoid of any details.
EDIT: Actually, Convera later sold its search technology to FAST, and started OEMing FAST’s technology instead. Microsoft now inherits that relationship with its acquisition of FAST.
Comments
3 Responses to “Web search and enterprise search are coming together”
Leave a Reply
[…] Now the company offers RetrievalWare, augmented by some pattern-matching technology – e.g., what they think is a better form of fuzzy word tokenization, and some color/shape/texture image matching as well. They also have introduced a web search product. (This is confusingly called Excalibur, but they told me last week that a much-needed rebranding is underway.) Maybe this strategy will be the one that finally works out for them. • • • […]
[…] FAST, Convera, Google, and Microsoft all have the potential to introduce such a product package, although I’m not aware of any specific initiatives that exactly match what I have in mind. The closest may be Convera, which is providing a standard vertical-market-specific sub-Web designed for its government intelligence/law-enforcement customers. (I’ve forgotten whether this is on-site or on a SaaS basis.) […]
[…] Edit: But the separation isn’t absolute. • • • […]