The formal analysis and semantic interpretation of pictorial representations
A 2-day tutorial and forum discussion
Guest speakers: Gabriel Greenberg and Neil Cohn
Date: March 16-17, 2018
Times: 10.15 am – 4.45 pm
Location: Humanities Institute, Library 4th floor
Organization and Contact: Harry van der Hulst
Registration required (write to: harry.van.der.hulst@uconn.edu)
This event is made possible due to support from UConn’s Humanities Institute (UCHI) and from the Connecticut Institute for the Brain and the Cognitive Sciences (IBACS).
Speakers
Gabriel Greenberg (UCLA, Philosophy)
Professor Gabriel Greenberg’s research interests include iconic representation, modality, and computational theories of mind. In ongoing work, he hopes to develop the foundations for formal theory of the semantics of pictures, using techniques from vision science, linguistics, and philosophy. He is also part of a research group studying discourse-based and dynamic account of film interpretation.
Neil Cohn (Tilburg University, The Netherlands)
Professor Neil Cohn’s area of expertise is the formal analysis of drawing systems and sequential graphic narratives (‘comics’, ‘graphic novels’) based on linguistic and cognitive neuroscience methods. His contribution would be his proposal for integrating the study of drawing and graphic communication into the linguistic and cognitive sciences, which was the topic of his 2013 monograph (The Visual Language of Comics) and over 60 articles.
Websites
Professor Greenberg: http://gjgreenberg.bol.ucla.edu/
Professor Cohn: http://www.visuallanguagelab.com/
Objectives
The work of both scholars is complementary in the sense that a (formal) semantic analysis can or must be based on a formal analysis of the graphic display. Greenberg’s and Cohn’s work has been developed independently and a major objective of this event is to connect and discuss two different strands of research.
Structure of the event
On the first day, both Greenberg and Cohn will discuss their research on pictorial representation, leaving ample room for questions and discussion. On the second day, both scholars will share their findings on visual sequences, such as film and comics. At the end of each day we will have a forum discussion, which includes researchers from various disciplines at UConn.
Friday March 16
10.15 Introduction
10.30 – 12.45 Professor Greenberg: “The Semantics of Visual Signs”
12.45 – 1.45 Lunch
1.45 – 4.00 Professor Cohn: “The Linguistics of Visual Lexicons”
4.00 – 4.45 Discussion
5.00 Informal gathering Bousfield, Psychology bld.
Saturday March 17
10.15 Arrival
10.30 – 12.45 Professor Greenberg: “Discourse Structure in Visual Narrative”
12.45 – 1.45 Lunch (provided)
1.45 – 4.00 Professor Cohn: “The Structure and Cognition of Visual Narrative Grammar”
4.00 – 4.45 Discussion
Talk summaries
Greenberg’s first talk: “The Semantics of Visual Signs”
In this talk, I’ll introduce participants to the emerging field of “formal semiotics” through the lens of my own research on the semantics of pictures. The fundamental problem of pictorial semantics is explaining how flat, marked surfaces manage to express the rich, three-dimensional spatial content for which pictures are so valued. Here I’ll focus on three key ideas which are essential for making progress on this complex puzzle. The first is the idea that it is possible model pictorial content using formal methods; here I’ll compare different strategies for thinking about pictorial content using tools borrowed from linguistics and perceptual science. Second is the idea that pictures are governed by systems of depiction, the pictorial analogues of languages; I’ll focus especially on variation in the treatment of spatial geometry and line between different systems. Finally, I’ll suggest that we must adopt a notion of visual reference, along with related concepts of visual anaphora and visual quantification, as part of a complete account of pictorial representation. I’ll conclude by reflecting on open questions and future directions for this nascent research program.
Greenberg’s second talk: “Discourse Structure in Visual Narrative”
Films are made up of individual shots strung together in sequences over time. Though each shot is disconnected from the next, combinations of shots still convey coherent stories that take place in continuous space and time. How is this possible? In this talk, I’ll outline a partial answer to this question, in which overall spatial coherence in film is facilitated by a set of visual conventions that govern spatial relations between viewpoints. I’ll focus on several examples, including the well-known “180° Rule”, POV-editing, and a few rules that I and my colleagues have discovered ourselves. These rules, I’ll argue, are analogous to relations of discourse coherence that are widely recognized in the linguistic domain. I’ll outline a dynamic account of film interpretation, in which such rules help secure the spatial interpretation of each shot as it is presented, against the backdrop of the space already established by the film thus far. The result is a new framework for understanding the representation of space in film.
This talk is the product of joint work between myself, Sam Cumming (UCLA Philosophy), Rory Kelly (UCLA Film, Theater, and Television), and Elsi Kaiser (USC Linguistics).
Cohn’s first talk: “The Linguistics of Visual Lexicons”
I will first orient participants towards a linguistic analysis of visual information, and make the case that drawing and graphic representations draw on similar cognitive machinery as language. This cognitive and linguistic orientation is contrasted against notions of drawing as grounded in the articulation of perceptual phenomena alone (i.e. “life drawing” or “drawing from memory”). Rather, I will argue that drawing uses “visual lexical items” which operate similarly to vocabulary in the verbal modality. Such an approach is sensitive both to the question of How do we comprehend pictures? but also to How do we produce pictures? This latter question is particularly important, because despite being a fundamental and basic ability of human expression, many—if not most—people feel that they “can’t draw” and have a fairly low drawing proficiency. I will thus reanalyze the evidence of drawing development through a psychological perspective that will illustrate how deficits in drawing ability parallel those found in language acquisition and individuals with impoverished language exposure (ex. home signers and individuals with language deprivation). Finally, I will posit cultural frames which push conditions for learning to draw away from individuals’ biological instincts, thereby perpetuating the notion of “I can’t draw.” Overall, this work will align graphic creation with other expressive modalities in a domain-general approach sensitive to cognition and development.
Cohn’s second talk: “The Structure and Cognition of Visual Narrative Grammar”
While the first day focused on the structure and development of individual images, I will then turn to examine the cognition of images in sequence, like those in visual narratives like comics. First, I will describe how approaches to juxtaposed relations are insufficient for describing the cognition of sequential images, and will discuss cross-cultural and developmental research establishing that a fluency is necessary to understand sequential images. Then, building on the linguistic orientation of the first day, I will argue that image sequences use a “narrative grammar” that organizes semantic information. I will show that this structure, based on linguistic construction grammars, packages meaning into categorical roles and hierarchic constituents, and thereby can account for phenomena like long distance dependencies and structural ambiguities. In addition, drawing on results from measurements of brainwaves (EEG), I will show that this grammar is independent of meaning (e.g., N400), and engages similar neurocognitive processing as syntax in language (e.g., anterior negativities, P600). Finally, I will show that sequential image processing is modulated by a person’s fluency in the specific narrative grammars found in different “visual languages” of the world. Altogether, this work introduces emerging research from the linguistic and cognitive sciences that challenges conventional wisdom with a new paradigm of thinking about the connections between language and graphic communication.