Wolfram Alpha

I played with Wolfram Alpha a bit this morning. So far it mostly seems like a nice web service front end on Mathematica, example link1 and link2, and a good source for straightforward kinds of information lookup, e.g., facts on a city, travel, (but it won't disambiguate 'seattle washington' as a potential query on travel b/w Seattle and D.C.) company information and the like.

It did rather poorly on the following:

April Wine discography
Why are there ants in my sink drain?
wine to serve with tofu
kurt godel mental illness
toyota sales U.S. 2004 (this 'related input to try' was amusing though, interpreting 'sales' and 'toyota' as places)

To be fair, I've been evaluating it by posing questions in the same way that I'd posed them to Google. (with the exception of the mathematical queries). Unfortunately, I think that our experiences with Google shape our expectations and the way we test these new tools, we basically expect to be able to write natural language inputs. I think that if end users would be willing to open their minds a bit in terms of the way they submit input to a search/knowledge tool, the potential effectiveness would increase significantly and developers could spend more time on developing intelligent apps rather than solving the really really hard problem of understanding natural language input. I think about this frequently because the company I work for develop tools that are designed to exploit semantic search. We index documents using a knowledge base of concepts that gather synonyms and relations between those concepts. As a result we can understand, for example, that documents about aortic regurgitation are relevant to queries on 'heart valve disorder'. It's clear to me that our tools could be exploited so much more effectively if people would be able/ willing to use simple semantic relationships and variables in their queries. (Imagine being able to specify queries asking explicitly and fairly directly for, for example, articles about types of heart disease written by people affiliated with universities in the southwestern u.s. or the stock indices for stock markets located in Eastern Europe). But we're constrained, greatly IMO, by the fact that people expect / demand to use search engines only with natural language queries. It's like the rest of the world could be developing sophisticated robots capable of having sophisticated conversation in French, but we insist that we'll only use English when testing it.

UPDATE: BTW, should have mentioned, the 13 min. intro video is a good place to start (before experimenting) if you really want to have a sense of how to test it.