Subject

Materials (*=available in pdf)

Introduction and organizational issues

Basic Bibliography

* Stein, B. E. & Meredith, M. A. (1993). The merging of the senses. Cambridge, MA: The MIT Press. Introduction (p. ix-xv); Chapter 1 (p. 1-19); Chapter 2 (20-37).

* Shimojo, S. & Shams, L. (2001). Sensory modalities are not separate modalities: plasticity and interactions. Current Biology, 11: 505-509

Multisensory integration revealed in the spatial domain

Basic Bibliography

Bertelson, P. (1998). Starting from the ventriloquist: The perception of multimodal events. In M. Sabourin, F. I. M. Craik, & M. Robert (Eds.), Advances in Psychological Science, Vol. 2: Biological and Cognitive Aspects, (pp. 419-439). Hove, England: Psychology Press/Erlbaum.

Further readings On the determinants of ventriloquists illusion and on how to measure it.

Welch, R. B. (1999). Meaning, attention, and the “unity assuption” in the intersensory bias of spatial andd temporal perceptions. In G. Ashersleben, T. Bachmann, & J. Müsseler (Eds.), Cognitive contributions to the perception of spatial and temporal events (pp. 371-383). Elsevier Science, B.V.: Amsterdam.

* Recanzone, G. H. (1998). Rapidly induced auditory plasticity: The ventriloquism after-effect. Proceedings of the National Academy of Science, 95, 869-875.

* De Gelder, B. & Bertelson, P. (2003). Multisensory integration, perception and ecological validity TRENDS in Cognitive Sciences Vol.7 No.10, 460-

Multisensory interactions in the domain of time

Basic Bibliography

* King, A. (2005). Multisensory Integration: Strategies for Synchronization. Current Biology, 15(9), 339-341

Further Readings:  Basic demonstrations of cross-modal temporal effects

^* Morein-Zamir, S., Soto-Faraco, S. & Kingstone, A. (2003). Auditory capture of vision: examining temporal ventriloquism. Cognitive Brain Research, 17(1):154-63.

^* Fujisaki W, Shimojo S, Kashino M, Nishida S. (2004). Recalibration of audiovisual simultaneity. Nat Neurosci,  7(7):773-8. 

^* Vroomen J, Keetels M, de Gelder B, Bertelson P. (2004). Recalibration of temporal order perception by exposure to audio-visual asynchrony. Cogn Brain Res., 22(1):32-5.

^* Shams, L., Kamitani, Y., Shimojo, S. (2000). What you see is what you hear. Nature, 406: 788 

Discussion about limitations based on neural correlates

* McDonald, JJ, Teder-Salejarvi WA, Di Russo F, Hillyard SA. (2005). Neural basis of auditory-induced shifts in visual time-order perception. Nat Neurosci. 2005 Sep;8(9):1197-202

How do they do it? Mechanisms and models of multisensory integration

Basic Bibliography

^* Stein, B. E. (1998). Neural mechanisms for synthesizing sensory information and producing adaptive behaviors. J. Exp. Brain Res., 123, 124-135.

^* Ernst. M. O. & Bulthoff, H. H. (2004). Merging the senses into a robust percept TRENDS in Cognitive Sciences, 8(4), 162-169

Further Readings: Extrapolating super-additivity from the cell to the organism and limitations of the super-additivity hypothesis

* Calvert, G. A. (2001). Crossmodal Processing in the Human Brain: Insights from Functional Neuroimaging Studies. Cerebral Cortex, 11(12): 1110-1123

* Murray MM, Molholm S, Michel CM, Heslenfeld DJ, Ritter W, Javitt DC, Schroeder CE, Foxe JJ. (2004). Grabbing Your Ear: Rapid Auditory-Somatosensory Multisensory Interactions in Low-level Sensory Cortices Are Not Constrained by Stimulus Alignment. Cereb Cortex. 2004 Nov 15; [Epub ahead of print] 

* Holmes, NP & Spence, C. (2005). Space, Time and Superaditivity. Curr Biol. 15(18), 762-764

Feedforward vs. feedback mechanisms

^* Driver, J. and Spence, C. (2000). Multisensory perception: Beyond modularity and convergence in crossmodal integration. Current biology, 10, R731-R735.

* Foxe, JJ.  & Schroeder, CE (2005). The case for feedforward multisensory  convergence during early cortical processing. Neuroreport 16(5):419-423

Extension of the optimal integration model

* Alais, D. & Burr, D. (2004). The Ventriloquist Effect Results from Near-Optimal Bimodal Integration. Current Biology, Vol. 14, 257–262, February 3, 2004,

Hearing lips and seeing sounds: Speech as a multisensory phenomenon

Bibliography

* Massaro, D. W. & Stork, D. G. (1998). Speech recognition and sensory integration. American Scientist

* Rizzolatti, G. & Arbib, M. (1998). Language within our grasp. Trends in Neurosciences, 21, 188-194

Further Readings: Basic demonstrations and brief definitions

* Sumby, W. H., & Pollack, I. (1954). Visual contribution to speech intelligibility in noise. Journal of the Acoustical Society of America, 26, 212-215.

* McGurk, H. & MacDonald, J. (1976). Hearing lips and seeing speech. Nature, 264, 746-748.        

* Campbell, R. (in press; 2004). Audiovisual speech processing. The Encyclopedia of Language and Linguistics (2nd edition) Keith Brown (Ed.). Oxford, Elsevier

Neural substrate for the integration of speech

* Colin, C. (2002). Mismatch negativity evoked by the McGurk–MacDonald effect: a phonetic representation within short-term memory. Clinical Neurophysiology, 113, 495–506

* Calvert, G. A., Bullmore, E. T., Brammer, M. J., Campbell, R., Williams, C. R., McGuire, P.K., Woodruff, P. W. R., Iversen S. D., & David, A. S. (1997). Activation of auditory cortex during silent lipreading. Science, 276, 593-596.

* van Wassenhove, V. Grant, KW & Poeppel, D. (2005). Visual speech speeds up the neural processing of auditory speech, PNAS, 102;1181-1186

Mirror neurons, animal calls, and the origins of language

^* Ghazanfar AA, Logothetis NK. (2003). Neuroperception: facial expressions linked to monkey calls. Nature. 423(6943): 937-938

* MELTZOFF, AN & MOORE, MK (1977). Imitation of Facial and Manual Gestures by Human Neonates, Science, Vol. 198. no. 4312, pp. 75 – 78

* Fadiga, L., Craighero, L., Buccino, G. & Rizzolatti, G. (2002). Speech listening speciÆcally modulates the excitability of tongue muscles: a TMS study. European Journal of Neuroscience, Vol. 15, pp. 399-402

Can I borrow this piece of brain? Development and plasticity of multisensory interactions

Basic Bibliography (one of the two references)

* King, A. (2004). Development of multisensory spatial integration (pp 1-24), In C. Spence & J. Driver (Eds.) Crossmodal space and crossmodal attention. Oxford, England: Oxford University Press.

* Lickliter, R. & Bahrick, L. E. (2004). Perceptual development and the origins of multisensory responsiveness. The Handbook of Multisensory Processes. MIT Press (pp. 643-654)

Further Readings Cross-modal integration in infancy

Kuhl, P. K., & Meltzoff, A. N. (1982). The bimodal perception of speech in infancy. Science, 218, 1138-41.

Cross-modal plasticity in the sensory deprived

^* Cohen LG, et al. (1997). Functional relevance of cross-modal plasticity in blind humans. Nature, 389(6647):180-183. 

* Bavelier, D. & Neville, H. (2002). Cross-modal plasticity: Where and How. Nat. Rev. Neurosci., 3, 443-452

Synesthesia

^* Grossenbacher, P. G., and Lovelace, C. T., 2001. Mechanisms of synesthesia: cognitive and physiological constraints, Trends in Cognitive Sciences, 5: 36- 41.

^* Dixon, M. J., Smilek, D., Cudahy, C., & Merikle, P. M. (2000). Five plus two equals yellow, Nature, 406: 365.

Where is my hand? Constructing the body schema

Basic Bibliography

^* Maravita, A., Spence, C. & Driver, J. (2003). Multisensory Integration and the Body Schema: Close to Hand and Within Reach. Current Biology, 13, R531–R539,

Readings Localizing tactile events in the body

* Yamamoto, S. & Kitazawa, S. (2001). Reversal of subjective temporal order due to arm crossing. Nature Neurosci., 4(7), 759-765

* Kitazawa, S. (2002). Where conscious sensation takes place. Consciousness and Cognition, 11, 475-477.

Libet B, Alberts WW, Wright EW Jr, Feinstein B. (1967). Responses of human somatosensory cortex to stimuli below threshold for conscious sensation. Science. 1967;158(808):1597-1600. [-]

Out of the body experiences and phantom limbs

^* Graziano MS. (1999). Where is my arm? The relative role of vision and proprioception in the neuronal representation of limb position. Proc Natl Acad Sci U S A. 96:10418-21. 

^* Botvinick, M., & Cohen, J. (1998). Rubber hands ‘feel’ touch that eyes see. Nature, 391, 756.

^* Pavani F, Spence C, Driver J. (2000). Visual capture of touch: out-of-the-body experiences with rubber gloves. Psychological  Science, 11, 353-9.

^* Blanke O, Ortigue, S, Landis, T, Seeck, M. (2002). Stimulating illusory own-body perceptions. Nature., 419, 269-70.

Breakdown of body representation

* Wolpert, DN, Goodbody, SM & Husain, M. (1998). Maintaining internal representations: the role of the human superior parietal lobe. NatNeuro, 1(6), 529-533

* Rapp, B. Hendel, SK & Medina, J. (2002) Remodeling of somotasensory hand representations following cerebral lesions in humans. Neuroreport, 13(2), 207-211

Extending the body schema with tools

^* Iriki, A., Tanaka, M. and Iwamura, Y. (1996). Coding of modified body schema during tool use by macaque postcentral neurones. Neuroreport 7, 2325–2330.

^* Maravita, A. & Iriki, A. (2004). Tools for the body (schema). Trends in Cognitive Neurosciences, 8, 79-86

Attention in a multisensory world 

Basic Bibliography

^* Driver, J., & Spence, C. (1998). Attention and the crossmodal construction of space. Trends in Cognitive Sciences, 2, 254-262.

Readings Electrophysiological correlates

Eimer, M. (2004). Electrophysiology of human crossmodal spatial attention (pp 221-245) In C. Spence & J. Driver (Eds.) Crossmodal space and crossmodal attention. Oxford, England: Oxford University Press.

* Talsma, D. & Woldorf, MG (2005). JoCN, 17(7):1098-114

Does multisensory integration need attention?

* Driver, J. (1996). Enhancement of selective attention listening by illusory mislocation of speech sounds due to lip-reading. Nature, 381, 66-68.

^* Alsius, A. et al. (2005). Audiovisual Integration of Speech Falters under High Attention Demands. Curr. Biol.