Cover not available

Article published In: Journal of Second Language Pronunciation
Vol. 4:2 (2018) ► pp.232–259

Get fulltext from our e-platform
JBE logo with link
PDF logo with download linkDownload PDF
authorization required

Visualizing the acoustic and gestural beats of emphasis in multimodal discourse

Theoretical and pedagogical implications

 | Michigan State University
Published online: 1 February 2019
DOI logo with link
https://doi.org/10.1075/jslp.17006.har
Abstract
Perceivers’ attention is entrained to the rhythm of a speaker’s gestural and acoustic beats. When different rhythms (polyrhythms) occur across the visual and auditory modalities of speech simultaneously, attention may be heightened, enhancing memorability of the sequence. In this three-stage study, Stage 1 analyzed videorecordings of native English-speaking instructors, focusing on frame-by-frame analysis of time-aligned annotations from Praat and Anvil (video annotation tool) of polyrhythmic sequences. Stage 2 explored the perceivers’ perspective on the sequences’ discourse role. Stage 3 analyzed 10 international teaching assistants’ gestures, and implemented a multistep technology-assisted program to enhance verbal and nonverbal communication skills. Findings demonstrated (a) a dynamic temporal gesture-speech relationship involving perturbations of beat intervals surrounding pitch-accented vowels, (b) the sequences’ important role as highlighters of information, and (c) improvement of ITA confidence, teaching effectiveness, and ability to communicate important points. Findings support the joint production of gesture and prosodically prominent features.
Article outline
  • 1.Introduction
    • 1.1Gesture and prosody
    • 1.2Theoretical perspectives
    • 1.3Prosody in Korean
    • 1.4L2 nonverbal communication
    • 1.5ITA Training
  • 2.Current study
    • 2.1Stage 1
      • 2.1.1Method
        • 2.1.1.1Participants
        • 2.1.1.2Recording
        • 2.1.1.3Selection of G-phrases for analysis
        • 2.1.1.4Speech annotations
        • 2.1.1.5Video annotations
      • 2.1.2Results
        • 2.1.2.1Example 1
        • 2.1.2.2Example 2
    • 2.2Stage 2
      • 2.2.1Participants
      • 2.2.2Materials and procedure
      • 2.2.3Results
    • 2.3Stage 3
      • 2.3.1Participants
      • 2.3.2Presentation skills program: Materials and procedure
        • 2.3.2.1Step 1: Raising visual awareness
        • 2.3.2.2Step 2: Raising gesture-prosody awareness
        • 2.3.2.3Step 3: Review and practice of speech features
        • 2.3.2.4Step 4: Review and practice of nonverbal communication
        • 2.3.2.5Step 5: Preparation for future classes
        • 2.3.2.6Follow-up
      • 2.3.3Results and discussion
        • 2.3.3.1ITA Example 1
        • 2.3.3.2ITA Example 2
        • 2.3.3.3Student survey on communication skills
        • 2.3.3.4ITA survey
  • 3.General discussion
  • Notes
  • References
References (65)
References
Allen, G. D. (1972). The location of rhythmic stress beats in English speech. Parts I & II. Language and Speech, 151, 72–100, 179–195. Google Scholar logo with link to Google Scholar
Bailey, K. M. (1982). Teaching in a second language: The communicative competence of non-native speaking assistants (Unpublished doctoral dissertation). University of California, Los Angeles.Google Scholar logo with link to Google Scholar
Beckman, M. E., Hirschberg, J., & Shattuck-Hufnagel, S. (2005). The original ToBI system and the evolution of the ToBI framework (pp. 9–54). In S. Jun (Ed.), Prosodic typology: The phonology of intonation and phrasing. Oxford: Oxford University Press. Google Scholar logo with link to Google Scholar
Beckman, M. E., & Pierrehumbert, J. (1986). Intonational structure in Japanese and English. Phonology Yearbook 31, 255–309. Google Scholar logo with link to Google Scholar
Bergeson, T. R., Pisoni, D. B., & Davis, R. A. O. (2003). A longitudinal study of audiovisual speech perception by children with hearing loss who have cochlear implants. The Volta Review, 1031, 347–370.Google Scholar logo with link to Google Scholar
Biau, E., & Soto-Faraco, S. (2013). Beat gestures modulate auditory integration in speech perception. Brain and Language, 1241, 143–152. Google Scholar logo with link to Google Scholar
Birdwhistell, R. (1970). Kinesics and context: Essay on body motion communication. Philadelphia, PA: University of Pennsylvania Press.Google Scholar logo with link to Google Scholar
Boersma, P., & Weenink, D. (2014). Praat: Doing phonetics by computer [Computer program]. Retrieved from <[URL]>
Bolinger, D. (1986). Intonation and its parts: Melody in spoken English. Stanford, CA: Stanford University Press.Google Scholar logo with link to Google Scholar
Bull, P., & Connelly, G. (1985). Body movement and emphasis in speech. Journal of Nonverbal Behavior, 91, 169–187. Google Scholar logo with link to Google Scholar
Chafe, W. (1994). Discourse, consciousness, and time: The flow and displacement of conscious experience in speaking and writing. Chicago, IL: The University of Chicago Press.Google Scholar logo with link to Google Scholar
Clark, J. M., & Paivio, A. (1991). Dual coding theory and education. Educational Psychology Review, 31, 149–210. Google Scholar logo with link to Google Scholar
Clayton, M., Sager, R., & Will, U. (2005). In time with the music: The concept of entrainment and its significance for ethnomusicology. ESEM (European Seminar in Ethnomusicology) CounterPoint, 11, 1–82.Google Scholar logo with link to Google Scholar
Condon, W. (1976). An analysis of behavioral organization. Sign Language Studies, 51, 285–318. Google Scholar logo with link to Google Scholar
de Ruiter, J. P. (2000). The production of gesture and speech. In D. McNeill (Ed.), Language and gesture (pp. 284–311). Cambridge: Cambridge University Press. Google Scholar logo with link to Google Scholar
Dimitrova, D., Chu, M., Wang, L., Özyürek, A., & Hagoort, P. (2016). Beat that word: How listeners integrate beat gesture and focus on multimodal speech discourse. Journal of Cognitive Neuroscience, 281, 1255–1269. Google Scholar logo with link to Google Scholar
Flecha-García, M. L. (2010). Eyebrow raises in dialogue and their relation to discourse structure, utterance function and pitch accents in English. Speech Communication, 521, 542–554. Google Scholar logo with link to Google Scholar
Flowerdew, J., & Tauroza, S. (1995). The effect of discourse markers on second language lecture comprehension. Studies in Second Language Acquisition, 171, 435–458. Google Scholar logo with link to Google Scholar
Fraisse, P. (1982). Rhythm and tempo. In D. Deutsch (Ed.), The psychology of music (pp. 149–180). New York, NY: Academic Press. Google Scholar logo with link to Google Scholar
Goldin-Meadow, S., & Alibali, M. W. (2013). Gesture’s role in speaking, learning, and creating language. Annual Review of Psychology, 641, 257–283. Google Scholar logo with link to Google Scholar
Gorsuch, G. J. (2003). The educational cultures of international teaching assistants and U.S. universities. TESL-EJ, 71. <[URL]>
Gregersen, T., Olivares-Cuhat, G., & Storm, J. (2009). An examination of L1 and L2 gesture use: What role does proficiency play? The Modern Language Journal, 931, 195–208. Google Scholar logo with link to Google Scholar
Gullberg, M. (2011). Multilingual multimodality: Communicative difficulties and their solutions in second language use. In J. Streeck, C. Goodwin, & C. LeBaron (Eds.), Embodied interaction: Language and body in the material world (pp. 137–151). Cambridge: Cambridge University Press.Google Scholar logo with link to Google Scholar
Hardison, D. M. (2003). Acquisition of second-language speech: Effects of visual cues, context and talker variability. Applied Psycholinguistics, 241, 495–522. Google Scholar logo with link to Google Scholar
(2004). Generalization of computer-assisted prosody training: Quantitative and qualitative findings. Language Learning & Technology, 81, 34–52. <[URL]>
(2005). Contextualized computer-based L2 prosody training: Evaluating the effects of discourse context and video input. CALICO Journal, 221, 175–190. Google Scholar logo with link to Google Scholar
(2012). Second-language speech perception: A cross-disciplinary perspective on challenges and accomplishments. In S. Gass & A. Mackey (Eds.), The Routledge handbook of second language acquisition (pp. 349–363). London: Routledge.Google Scholar logo with link to Google Scholar
Hincks, R., & Edlund, J. (2009). Promoting increased pitch variation in oral presentations with transient visual feedback. Language Learning & Technology, 131, 32–50. <[URL]>
Inceoglu, S. (2016). Effects of perceptual training on second language vowel perception and production. Applied Psycholinguistics, 371, 1175–1199. Google Scholar logo with link to Google Scholar
Jenkins, S., & Parra, I. (2003). Multiple layers of meaning in an oral proficiency test: The complementary roles of nonverbal, paralinguistic, and verbal behaviors in assessment decisions. The Modern Language Journal, 871, 90–107. Google Scholar logo with link to Google Scholar
Jones, M. R. (1986). Attentional rhythmicity in human perception. In J. R. Evans & M. Clynes (Eds.), Rhythm in psychological, linguistic, and music processes (pp. 13–40). Springfield, IL: Charles C. Thomas.Google Scholar logo with link to Google Scholar
Jun, S. -A. (2005). Korean intonational phonology and prosodic transcription. In S. -A. Jun (Ed.), Prosodic typology: The phonology of intonation and phrasing (pp. 201–229). Oxford: Oxford University Press. Google Scholar logo with link to Google Scholar
Kellerman, E., & Van Hoof, A. -M. (2003). Manual accents. International Review of Applied Linguistics, 411, 251–269. Google Scholar logo with link to Google Scholar
Kendon, A. (1972). Some relationships between body motion and speech: An analysis of an example. In A. Siegman & B. Pope (Eds.), Studies in dyadic communication (pp. 177–210). New York, NY: Pergamon Press. Google Scholar logo with link to Google Scholar
Kipp, M. (2001). Anvil – A generic annotation tool for multimodal dialogue. In Proceedings of the 7th European Conference on Speech Communication and Technology (pp. 1367–1370). Aalborg, Denmark: Eurospeech.Google Scholar logo with link to Google Scholar
Krahmer, E., & Swerts, M. (2007). The effects of visual beats on prosodic prominence: Acoustic analyses, auditory perception and visual perception. Journal of Memory and Language, 571, 396–414. Google Scholar logo with link to Google Scholar
Large, E. W., & Jones, M. R. (1999). The dynamics of attending: How people track time-varying events. Psychological Review, 1061, 119–159. Google Scholar logo with link to Google Scholar
Leonard, T., & Cummins, F. (2011). The temporal relation between beat gestures and speech. Language and Cognitive Processes, 261, 1457–1471. Google Scholar logo with link to Google Scholar
Levelt, W. J. M. (1989). Speaking: From intention to articulation. Cambridge, MA: The MIT Press.Google Scholar logo with link to Google Scholar
London, J. (2012). Hearing in time: Psychological aspects of musical meter (2nd ed.). Oxford: Oxford University Press. Google Scholar logo with link to Google Scholar
McCafferty, S. G. (2002). Gesture and creating zones of proximal development for second language learning. The Modern Language Journal, 861, 192–203. Google Scholar logo with link to Google Scholar
McClave, E. (1994). Gestural beats: The rhythm hypothesis. Journal of Psycholinguistic Research, 231, 45–66. Google Scholar logo with link to Google Scholar
McCroskey, J. C., Richmond, V. P., Sallinen, A., Fayer, J. M. & Barraclough, R. A. (1995). A cross-cultural and multi-behavioral analysis of the relationship between nonverbal immediacy and teacher evaluation. Communication Education, 441, 281–291. Google Scholar logo with link to Google Scholar
Mehrabian, A. (1969). Methods & designs: Some referents and measures of nonverbal behavior. Behavior Research Methods and Instrumentation, 11, 203–207. Google Scholar logo with link to Google Scholar
McNeill, D. (1992). Hand and mind: What gestures reveal about thought. Chicago, IL: University of Chicago Press.Google Scholar logo with link to Google Scholar
(2006). Gesture and communication. In the Encyclopedia of language and linguistics (2nd ed.) (pp. 58–66). Amsterdam: Elsevier. Google Scholar logo with link to Google Scholar
McNeill, D., Levy, E. T., & Duncan, S. D. (2015). Gesture in discourse. In D. Tannen, H. E. Hamilton, & D. Schiffrin (Eds.), The handbook of discourse analysis (2nd ed.) (pp. 262–289). Hoboken, NJ: Wiley & Sons.Google Scholar logo with link to Google Scholar
Meltzoff, A. N., & Kuhl, P. K. (1994). Faces and speech: Intermodal processing of biologically relevant signals in infants and adults. In D. J. Lewkowitz & R. Lickliter (Eds.), The development of intersensory perception: Comparative perspectives (pp. 335–369). Hillsdale, NJ: Lawrence Erlbaum Associates.Google Scholar logo with link to Google Scholar
Merker, B. H., Madison, G. S., & Eckerdal, P. (2009). On the role and origin of isochrony in human rhythmic entrainment. Cortex, 451, 4–17. Google Scholar logo with link to Google Scholar
Mori, J., & Hayashi, M. (2006). The achievement of intersubjectivity through embodied completions: A study of interactions between first and second language speakers. Applied Linguistics, 271, 195–219. Google Scholar logo with link to Google Scholar
Munhall, K. G., Jones, J. A., Callan, D. E., Kuratate, T., Vatikiotis-Bateson, E. (2004). Visual prosody and speech intelligibility: Head movement improves auditory speech perception. Psychological Science, 151, 133–137. Google Scholar logo with link to Google Scholar
Munhall, K. G., & Tohkura, Y. (1998). Audiovisual gating and the time course of speech perception. Journal of the Acoustical Society of America, 1041, 530–539. Google Scholar logo with link to Google Scholar
Neu, J. (1990). Assessing the role of nonverbal communication in the acquisition of communicative competence in L2. In R. C. Scarcella, E. S. Andersen, & S. D. Krashen (Eds.), Developing communicative competence in a second language (pp. 121–138). Boston, MA: Heinle and Heinle.Google Scholar logo with link to Google Scholar
Port, R. F. (2003). Meter and speech. Journal of Phonetics, 311, 599–611. Google Scholar logo with link to Google Scholar
Rosenblum, L. (2010). See what I’m saying: The extraordinary powers of our five senses. New York, NY: W.W. Norton & Co.Google Scholar logo with link to Google Scholar
Rusiewicz, H. L., Shaiman, S., Iverson, J. M., & Szuminsky, N. (2013). Effects of prosody and position on the timing of deictic gestures. Journal of Speech, Language, and Hearing Research, 561, 458–470. Google Scholar logo with link to Google Scholar
Skipper, J. I., van Wassenhove, V., Nusbaum, H. W., & Small, S. L. (2007). Hearing lips and seeing voices: How cortical areas supporting speech production mediate audiovisual speech perception. Cerebral Cortex, 171, 2387–2399. Google Scholar logo with link to Google Scholar
Sueyoshi, A., & Hardison, D. M. (2005). The role of gestures and facial cues in second-language listening comprehension. Language Learning, 551, 661–699. Google Scholar logo with link to Google Scholar
Summerfield, Q. (1979). Use of visual information for phonetic perception. Phonetica, 361, 314–331. Google Scholar logo with link to Google Scholar
Tajima, K., & Port, R. F. (2003). Speech rhythm in English and Japanese. In J. Local, R. Ogden, & R. Temple (Eds.), Phonetic interpretation: Papers in laboratory phonology VI (pp. 317–334). Cambridge: Cambridge University Press.Google Scholar logo with link to Google Scholar
Treffner, P., Peter, M., & Kleidon, M. (2008). Gestures and phases: The dynamics of speech-hand communication. Ecological Psychology, 201, 32–64. Google Scholar logo with link to Google Scholar
Tuite, K. (1993). The production of gesture. Semiotica, 931, 83–105. Google Scholar logo with link to Google Scholar
Volterra, V., & Erting, C. J. (Eds.) (2002). From gesture to language in hearing and deaf children. Washington, DC: Gallaudet University Press.Google Scholar logo with link to Google Scholar
Wennerstrom, A. (2001). The music of everyday speech: Prosody and discourse analysis. Oxford: Oxford University Press.Google Scholar logo with link to Google Scholar
Cited by (13)

Cited by 13 other publications

Lelandais, Manon
2025. Gestural strategies in questions during monological discourse. Linguistics Vanguard DOI logo
Algana, Mashael & Debra M. Hardison
2024. Variable effects of speakers’ visual cues and accent on L2 listening comprehension: A mixed-methods approach. Language Teaching Research DOI logo
Kim, Yoo Lae, Chen Liu, Pavel Trofimovich & Kim McDonough
2024. Is nonverbal behavior during conversation related to perceived fluency?. TESOL Journal 15:3 DOI logo
Bobkina, Jelena, Elena Domínguez Romero & María José Gómez Ortiz
2023. Kinesic communication in traditional and digital contexts: An exploratory study of ESP undergraduate students. System 115 ► pp. 103034 ff. DOI logo
Tsunemoto, Aki, Rachael Lindberg, Pavel Trofimovich & Kim McDonough
2022. VISUAL CUES AND RATER PERCEPTIONS OF SECOND LANGUAGE COMPREHENSIBILITY, ACCENTEDNESS, AND FLUENCY. Studies in Second Language Acquisition 44:3 ► pp. 659 ff. DOI logo
Hardison, Debra M & Martha C Pennington
2021. Multimodal Second-Language Communication: Research Findings and Pedagogical Implications. RELC Journal 52:1 ► pp. 62 ff. DOI logo
Hardison, Debra M.
2019. Technology-Based Communication Success for Second-Language Learners. In Encyclopedia of Educational Innovation, ► pp. 1 ff. DOI logo
Hardison, Debra M.
2020. Technology-Based Communication Success for Second-Language Learners. In Encyclopedia of Educational Innovation, ► pp. 1 ff. DOI logo
LaSCOTTE, DARREN & ELAINE TARONE
2019. Heteroglossia and Constructed Dialogue in SLA. The Modern Language Journal 103:S1 ► pp. 95 ff. DOI logo
Levis, John M.
2019. Teaching-oriented research. Journal of Second Language Pronunciation 5:1 ► pp. 1 ff. DOI logo
Levis, John M.
2024. Key issues in L2 pronunciation research. Journal of Second Language Pronunciation 10:3 ► pp. 293 ff. DOI logo
Pennington, Martha C.
2019. ‘Top-Down’ Pronunciation Teaching Revisited. RELC Journal 50:3 ► pp. 371 ff. DOI logo
Tarone, Elaine & Colleen Meyers
2018. The Mirroring Project. In Speaking in a Second Language [AILA Applied Linguistics Series, 17], ► pp. 197 ff. DOI logo

This list is based on CrossRef data as of 13 november 2025. Please note that it may not be complete. Sources presented here have been supplied by the respective publishers. Any errors therein should be reported to them.

Mobile Menu Logo with link to supplementary files background Layer 1 prag Twitter_Logo_Blue