講座題目：Multimodal Analysis of Video-based process Data
講 座 人：鄭東萍 教授（夏威夷大學）
地 點：北京外國語大學 東院圖書館三層報告廳
鄭東萍博士現任教于美國夏威夷大學二語研究系，博士生導師。美國康涅狄克州大學教育心理學博士。國際互動關系、語言及認知學會常任理事，廣東外語外貿大學外國語言學及應用語言學研究中心兼職研究員，吉林外國語大學白山學者。學術論文發表于世界著名的語言與跨文化交流SCI 、SSCI期刊，如Modern Language Journal, Language Sciences, Linguistics and Education等。
課題研究包括：1. 怎樣通過設計虛擬仿真游戲空間環境擴展其跨文化交流與協調的功能性，及跨文化雙向互動對語言認知和文化交流技巧的影響；2. 用移動無限通訊技術為平臺的設計App，檢測并培養對地方處所環境及他人的覺知性，從而讓學習者達到即時的學以致用，擴展認知、學習和文化交流空間，激發社會文化參與和實現價值的潛能；3. 創新立足于國學、適于亞洲文化社會的語言學習理論與實踐。主要提出行動協商理論(Negotiation for Action)、言語行動的價值實現理論(Languaging as Values-realizing) 、關護行動語言學習理論 (Language Learners as Caretakers)、從語言學習者到跨文化場所體驗者（From Language Learner’s identity to Cross-Cultural and Place-Taking Persons)。
With the increasing prominence of digitally-mediated communication, it has grown increasingly simple to incorporate a variety of modes, such as speech, writing, and visual images, into a single text. This increased potential for multimodality has called for a shift in focus from any one single mode to a framework which can incorporate all modes equally. Kress (2003) argues for an approach based on semiotic analysis, treating all elements of a text as signs which are purposefully chosen by and reflect the interests, perspectives, positions, and values of the sign-maker. Thus, the concept of literacy becomes not just the use of the resources of writing, but redefined through the broader scope of multimodal literacy in which the making and interpreting of signs is constrained by the affordances of the modes and resources involved, and simultaneously transforms both the semiotic resources at hand and the inner dispositions of those performing the meaning-making. The multimodal world of communication, Kress argues, now calls not simply for acquisition or competence but for design, the creative agency of the sign-maker to assess the affordances of the situation and incorporate their own interests and desires in the creation of new signs.
Kress’s approach demonstrates an increasing relevance for an ecological, dialogical and distributed (EDD) thinking in research on multimodality and new literacies, treating communication and interaction as ecological, agentive, and holistic. From an EDD perspective, cognition and learning are embodied, situated, dynamic and values-realizing. In tracing the development of EDD views applied to second language (L2) learning with technologies that afford avatar embodiment and place-based interactivity, this talk first will review four trends of multimodal analysis traditions (work done in collaboration with Kristyn Martin). Integrating EDD and selected analytical framework, this talk promotes an action-oriented sociocultural orientation to cognition and L2 learning. Such holistic learning requires new research methods to reveal how learning occurs and new learning results. Such results may include cultivated team relationships, friendships, coordination skills, cultural awareness. Multimodal analysis is one of the inclusive methods that can partially answer new research questions under the EDD views.
While Kress and many approaches to multimodality focus on the historical trends of literacy and the increasingly multimodal format of representation and communication, distributed cognition can offer a deeper look at the dynamic action-based processes involved in multimodal sense-making. In this paper, we will briefly review the trends of multimodality and new literacy research, and discuss how distributed cognition can bring the concepts of ecological values-realizing and the distributed cognitive system into the picture for more holistic analytical and pedagogical approach. Using multimodal data from student interactions in online virtual worlds and from Japanese television shows, we will demonstrate the need for analytical and pedagogical methods which are integrative and account for the agency and creativity of the sense-maker, particularly in the field of applied linguistics. We will then provide brief example of the application of the distributed cognitive view to multimodal analysis, and close with a discussion of how this integration of analytic approaches might be developed into a more holistic framework.