The Chinese Room experiment really does bother me. I don't see any reason for it to be impossible from a theoretical standpoint. If we can get a computer to really process language it seems to be an inevitability. Just print out the program, however it was obtained, and turn it into a set of instructions. The only question is whether a human could realistically follow them in a reasonable amount of time.
So given that we also have the fact that the human in the experiment has no knowledge of the language, no real understanding of anything that is happening. But if we define conciousness purely by outward characteristics, there must be a conciousness here that knows what is going on, but where is it? There are explanations, but they are all weird. It could be that the entire room, including the person as an "organ," has a literal conciousness. If we accept that than it seems to me that we must state in situations like when a person uses a computer program to help work out a problem there must actually be another conciousness present in the computer-human system, seperate from the human's (and perhaps the computer's, if it exists).
Then perhaps you could say that the intelligence is really in the computer that drew up these rules, or the human that programmed that computer if it really was all hard coded in. But then we have some weird situation where all the genunine thinking occurs in the past, and I can't see conciousness factoring in here.
It particularly seems important to me when I bring out my old 2XL robot, which if you don't know, is really a multi-track tape player with cleverly designed tapes. The intelligent behavior is all the result of a human pressing buttons in response to questions and a cleverly designed tape "program" (keeping in mind that the program does no action on its own.) Now no one thinks that 2XL is intelligent or concious, but they might ascribe to it characteristics that it doesn't have, like memory.
