Search engines
Analysis of search technology, products, services, and vendors. Related subjects include:
The text technologies market 2: It’s actually in disarray
The text technologies market should be huge and thriving. Actually, however, it’s in disarray. Multiple generations of enterprise search vendors have floundered, with the Autonomy/Verity merger being basically a combination of the weak. The RDBMS vendors came up with decent hybrid tabular/text offerings, and almost nobody cared. (Admittedly, part of the reason for that is that the best offering was Oracle’s, and Oracle almost always screws up its ancillary businesses. Email searchability has been ridiculously bad since — well, since the invention of email. And speech technology has floundered for decades, with most of the survivors now rolled into the new version of Nuance.
Commercial text mining is indeed booming, but not to an extent that erases the overall picture of gloom. It’s at most a several hundred million dollar business, and one that’s highly fragmented. For example, at a conference on IT in life sciences not that long ago, two things became evident. First, the text mining companies were making huge, intellectually fascinating, life-saving contributions to medical research. Second, more than ten vendors were divvying up what was only around a $10 million market.
If text technology is going to achieve the prominence and prosperity it deserves, something dramatic has to change.
| Categories: Enterprise search, Language recognition, Search engines, Speech recognition, Text mining | 2 Comments |
The text technologies market 1: It should be huge
From a number of standpoints, the market for enterprise technologies that explicitly* manage text SHOULD be huge. Consider:
1. The market for consumer text search is huge — think of Google.
2. The market for implicit* management of text is huge. Email management is a significant fraction of the IT budget, if you factor in the predominance of email in the use of networks and PCs. Now regulations are compelling email to be stored and managed at great expense. People spend hours per day working on email, word processors, etc.
3. The text mining market has recently boomed, and good ROI appears to be the norm.
*My implicit vs. explicit distinction here is meant to distinguish technologies that manage text as some sort of BLOB or other blob vs. technologies that take account of the fact that it is text, which contains words, phrases, synonyms, and so on.
If text technologies could live up to researchers’ dreams, typical knowledge workers would save hours per week and in many cases hours per day. The benefits would rival at least those of the whole PC/office productivity/messaging set of technologies. Thus, at least in theory, the market potential for these technologies is enormous.
| Categories: Search engines, Text mining | 3 Comments |
Autonomy + Verity — so what?
On some levels, the Autonomy/Verity merger makes total sense. The text search industry now has an unquestionably dominant vendor of shelfware. Somewhat less snarkily, I could say that it has a dominant OEM vendor of search technology. And while Verity’s management team has never recovered from the dizzying cycles of turnover in the 1990s, Autonomy’s obviously was quite effective. However, I see no obvious reason to believe that combined company will actually ship good products, or ones that lead to fundamentally greater adoption for enterprise search than the fairly marginal role it plays today.
Verity and Autonomy represent different philosophies of text search — Boolean vs. concept-based, basically. Neither works very well on its own, whether in the enterprise or on the web, with concept-based being the weaker of the two. That’s why Altavista et al. failed, and Excite failed yet more completely. It’s why Verity’s text search is generally more respected, and has more hardcore users, than Autonomy’s. (Being a vastly older company than Autonomy helps a lot too, of course.)
I hope that the merged company will soon introduce some new and/or synthesized approaches to search, significantly improving the overall quality of available products. If anybody has the resources and motivation, it will. The recent boom in text data mining, and the general increase of seriousness about ontologies, at least raises the possibility that concept-oriented search will evolve into something significantly useful. But I’m not holding my breath.
| Categories: Enterprise search, Ontologies, Search engines | 4 Comments |
Linkage among different text technologies
The first post in this blog describes its subject as “a group of interrelated, linguistics-based technology sectors, including text mining, search, speech recognition, and text command-and-control.” So I might as well kick off the discussion by summarizing some reasons why I think these sectors really are connected to each other. Very quickly:
IBM says so, and nobody (that I know of) is contradicting them. The essence of the UIMA story is that a lot of different pieces of technology need to be swapped in and out, not just among different brands of the same text applications, but among different kinds of text app. The vendors I checked with are uniformly skeptical about whether UIMA will have a real market impact, but none disputes UIMA’s underlying premise.
The tokenization chain. This general industry agreement is only one of three major reasons I believe in the general UIMA premise (while sharing the skepticism about that particular framework’s early adoption). A second dates back to when I was first learning about text search. At a Verity User Conference in, I think, April 1997, I had a very interesting conversation about Verity’s new architecture. (Probably with Phil Nelson, maybe with somebody else, such as Hugh Njemanze or Nick Arnett.) Basically, the system had been modularized, and the way it had been modularized was to create a flow of tokenization after tokenization after tokenization. The third reason is the observation that Inxight, so central to the tokenization strategies of text search vendors, plays pretty much the same role for the text mining companies.
The centrality of concept ontologies. I don’t currently have an opinion about the Semantic Web, but in a more limited sense it’s clear that ontologies will rule text applications. Whether for search, text data mining, or application command/control, it just doesn’t suffice to identify, find, weigh, or respond to individual words. Rather, you need to add other words indicating similar meaning – or a similar user “intent” — into the mix.*
This is a big deal, because simple minded ontologies don’t work. They can’t just be automatically generated, and they can’t just be hand-built. They can’t just be custom to each user or user enterprise, but they also can’t be provided entirely by technology vendors. Almost no large enterprises currently have a good system of ontology building and management, but in the near future most will have to. Evolution in this area will be a crucial determinant of how multiple text technology submarkets are shaped.
In particular, this is a big enough deal that I think search and text data mining and other text technologies will, for each enterprise, tend to use the same ontology.
*Note: There’s a whole other question as to how long we’ll be able to get by just looking at semantics, or whether syntactic analysis absolutely also should be in the mix. But first things first; without a good ontology, syntactic analysis is a pretty hopeless endeavor.
The use of text data mining in other areas. The automated part of the ontology building process involves a lot of text data mining. Large search engine companies generally do a lot of data mining to establish and validate tweaks to their search algorithms. The same goes for spam filters and more questionable forms of censorware. You can’t act intelligently without learning, and machines don’t learn well without doing statistical analyses.
I hope to post soon on each of these issues at more length, and I encourage comments on any of them as inputs to further work. But for now, I’ll just claim to have provided strong evidence for my initial point: Seemingly different text technologies are indeed closely related.
| Categories: Ontologies, Search engines, Text mining | 2 Comments |