Subtitles & Captions:
Working Title

00 Abstract

Subtitles are a form of visual language that originate from the gap between native and foreign languages. Subtitles overlay the main content - usually visual media - and serve as a secondary explanation of the content. As a form of typography, subtitles are also an interesting medium in which the rules of graphic design apply, such as visibility, legibility, and explicitness.

In addition to the nature of these subtitles, my understanding of them tells a story about my life as an outsider. Like the subtitles, I have been floating in Dutch society and hoping to be interpreted by someone. Like a mental exile in a city of strangers, I have been trying to run away from somewhere. My fragmented thoughts have been recorded in my diaries, which are mainly written in Korean and then translated back into English because of the desire to be read by more people in a foreign language. Like the hierarchy of subtitles and main content, I often feel like I am an ancillary entity; this gap sometimes motivates me to flog myself, but many times I felt lonely among the different subjects. Can I, who speaks nonsense in a foreign language, ever get out of the subtitles and become the main content? If my thoughts were subtitles, what was I musing about to illustrate a topic?

In this research, the gap between the new and old worlds is metaphorised or explained through the symbiosis of microorganisms, Freudian psychoanalysis or political dynamics, Egon Schiele's self-portraits, and Cha Hak Kyung's diasporic life, with subtitles as a symbolic medium to connect these themes. I will subvert the hierarchy of contrasts and ambivalences, and at the same time speak about the many strangers who have left their comfort zone and are living without the protection of familiar community norms and native languages.

01 Introduction

I remember when I first started learning English in earnest to study abroad. My conversation partners were mostly Koreans who could only read English, and we practised speaking by stuttering over words. I remember feeling very anxious, even though I knew that when I spoke to a native speaker, they would listen to me better and be patient with my slow English. English for communication, as opposed to English for reading and testing, was unfamiliar territory for me.

The language barrier I experienced after arriving in the Netherlands, a foreign land, proved to be a bigger obstacle than I expected. Not only was I a stranger here, but I felt like no one understood me because I couldn't communicate perfectly, and it was hard to find Europeans who were friendly to Asians because the outbreak of the COVID-19 pandemic brought on a surge in anti-Asian racism worldwide. Historically, when human beings feel threatened, they try to find a sense of security by attacking another group, and in the case of Corona, it was Asians. I didn't know whether to laugh or cry when I went to get my vaccine confirmation and saw a Chinese couple wearing orange jumpers with the Dutch flag on them. I don't know if they were scared, but it was easy to infer what they were experiencing during this time. At the time, I thought I was being marginalised because I couldn't speak English, but looking back, I think I was experiencing a clash of cultures, power structures, and a new mainstream culture. not being able to speak English fluently makes an individual in this society socially inferior, which is an inevitable fate as a stranger suddenly plunged into another society.

In this context, I had no choice but to get used to subtitles. When I came home, I would turn on subtitles on YouTube and watch videos of native speakers to practice my listening skills. As I became quieter than I was in Korea, I spent more time thinking. I got fascinated by this medium that encapsulates the desire to be interpreted by others. I aim to explain my life as a stranger by exploring the typographic nature of subtitles, their origins and history, their political implications and, above all, their role as an inward gaze.

02 Typographic study of subtitles

Subtitles were first used during the silent era of cinema. They were not written directly on the screen, and if you watch old silent films, you can see that they go like this: scene - subtitle - scene - subtitle. The earliest form, of course, was just showing the screen, and then the dialogue was written on a few cards and handed out to the audience, who would then look at the cards along with the screen. This ultra-analogue method did not survive in film Naturally, the card dialogue is not included in the subtitles, so what follows the screen is the subtitles. In thesedays, illiteracy was high, and there was a profession called ‘dictator’, who explained what was happening on screen and spoke the dialogue on behalf of the subtitles in real time. Subtitles and temporality have been inextricably linked since the beginning of time. This is where the main difference between simple interpretation and captioning/subtitling lies. Subtitling is a media that replaces transcribed text with time-coded chunks.

We tend to use the terms subtitle and caption interchangeably because there is no difference in the technical implementation. However, there is a need to distinguish between them for different purposes. Subtitle refers to the common subtitle, which is the subtitle that is needed by the general public. They are usually visible (open) and are embedded in the video, not in a separate file. Caption is a verbatim transcription of the spoken words of the programme, so it's usually not visible (Closed) and you have to press the CC (Closed Caption) button to make it appear. Recently, social media has been offering a feature called Auto Caption, which uses AI technology. Auto captions automatically generate subtitles, allowing viewers to read or listen to content. As creators make content, they can select auto captions in the editing page after they've uploaded or recorded a video so that text is automatically transcribed and displayed on their videos.

However, since the settings of the media player or receiver determine the font, size, and background colour of the captions, there is a possibility that the captions may not match the content, so some programs embed the captions into the video (Open). For simplicity, we will refer to these as subtitles and distinguish between subtitles and captions when necessary.

Subtitles usually have a place and a purpose. Their primary purpose is to maximise readability in a limited area. As such, most of the rules of typography apply to them. In the recent development of social media and multimedia ^[1]OTT (over-the-top media service) , subtitles are not only used to explain the video, but also as ^[2]visual art. However, subtitles are still an ancillary media, and as such, they are still largely limited to legibility and readability. The most prominent modern forms of subtitling are found on TikTok and Netflix - two platforms that are the focus of my research here for their prominence in popular culture, with distinct differences. Netflix was launched on August 29, 1997, initially as a DVD rental service before transitioning to streaming in 2007 while TikTok was launched in September 2016 by the Chinese company ByteDance. It was later released internationally as TikTok in September 2017 after merging with the app Musical.ly in 2018. While Netflix is using ^[3]SDH to expand the scope of traditional subtitling, TikTok's subtitles combine free-form and automatic subtitle generation with the social and playful nature of TikTok to create a new form of subtitling. As the fastest growing social media in history, TikTok is making social and political content history.

03 The Politics of Subtitling

What's interesting is the close interaction between TikTok's captions and its machine learning algorithms. TikTok recently launched the ability to automatically live-caption users' content, giving them the opportunity to create more interesting algorithms and reach a wider audience. However, TikTok's data collection has been publicly questioned, and its developer, Byte Dance, is headquartered in China, so all content is subject to censorship by the Chinese government. But also outside of the scope of governmental censorship, issues that are politically sensitive and that cause users to be shadowbanned have come from different directions. It's not just China, censorship of politically sensitive topics is an open secret in the United States. The United States is a country known for its freedom of expression, but if you create content with a pro-Palestinian stance on a platform like Meta, you will be heavily ^[4]shadowbanned.

Some TikTok users have gone so far as to pronounce words that are politically sensitive differently when discussing topics to avoid censorship by live-captioning, and thus avoid having their content transcribed by the AI flagged or shadowbanned. In addition to these censorship issues, TikTok has recently been threatened with being shut down by the ^[5]US government , which is understandable given that it is a Chinese company under strict government censorship. However with the first one ending just 14 hours after Trump's re-inaugration, TikTok said it was gradually lifting the restrictions it had put in place for US users. TikTok is in one of the most politically contentious positions of any short-form video hosting service where subtitles are predominantly used.

TikTok isn't the only example of social media being used for political and social emancipation, but one interesting example is the ^[6]Silly Girl Project. The project is a movement that seeks to escape the violence of gender hegemony through the immature silliness of young teenage girls' behaviour and emotions, paradoxically. They reject the violent outside world and its rules through the personal, unsettling, and at times overly cute content of girls created and shared in digital environments. The idea is to send a message by covering up and caricaturing the shackles and social conventions of being a woman with cuteness, giggles, and the nostalgia of the early days of the internet - like patching up a hole in your car with a pink band-aid, or sharing girls' thoughts and questions about society through the #imjustagirl hashtag. By resisting language that devalues girls like girl logic, girl maths, and girl science, they are creating their own culture, where captioning is no longer a secondary role in sily girl content, but a primary role in sending a message. The silly girl trend is considered to be one of the positive women's movements as it plays with the digital space and creates empathy at the same time. As such, the message of the subtitles does not have to be a secondary explanation of the video.

In the podcast ^[7]How to speak bad English from 2021, we learn about how native speakers communicate with people from non-English speaking countries. The accents of non-natives are often treated as something that must be reduced to facilitate communication, but is it really necessary? It is argued finally, that it's a myth that so-called ‘bad English’ is a barrier to communication, and that from a global perspective, accents should in fact be seen as something to be understood, not shamed. It's not uncommon to have your opinion distorted or your logic negated by someone who is fluent in a language you don't speak without an audible accent. It is natural to memorise words and practice fluency in order to learn a language, but it is not natural to be passed over for a promotion or penalised simply because you have a particular accent and your fluency is not fast enough to keep up with the rhythm of the conversation. Given the two-way nature of communication, this is why members of an increasingly globalised society need to take a fresh look at the legitimacy of bad English. In subtitles, non-native speaker's accents disappear—leaving only the meaning behind the words. This highlights how communication is about mutual understanding, not perfection.

Tony Cokes (born1956, United States) is an artist who has explored the relationship between black culture and popular culture through the temporality, transformation, and placement of text. He mixes text with film and images. His work, which often consists of colourful text on a black background, seems to represent a non-mainstream voice whose identity has been deconstructed. The outsider is always being excerpted and reassembled by mainstream culture. He explores how language and images can interact and reinforce a message. Subtitles are often used in his films to emphasise social and political messages, or to present new perspectives to the audience, which pushes them beyond mere translation to act as a visual language.

An another artist who has talked a lot about herself as a foreign woman speaking a “strange language” is Cha Hak Kyung(1951 – 1982). As a Korean-American, and in her book, Dictee , she expresses the issues of translation elaborately.

" I mumble inside. I mumble. Inside is the pain of speaking, the pain of trying to speak. There is something greater than that. Greater is the pain of not speaking. To not speak. Nothing is said about the pain of trying to speak. It bubbles up inside. The wound, the liquid, the dirt. It needs to burst. It must be excreted."

Dictee reads like a avant-garde narrative poetry collection. Her book is less like a poetry collection and more like a manifesto of her thoughts. Reading her thoughts as she writes, one is invited to imagine the difficulties she must have faced when she moved from South Korea to the United States in the sixties, but the pain she felt when she was forced to express in another language the thoughts she wanted to excrete, the thoughts she had to excrete. In the book, she describes herself as a clumsy imitation of another country's language, trying to speak through fragmented text but not being interpreted. It is not difficult to find common ground in the fragmented and fragmented thoughts of people living as strangers in the face of language barriers. Weaving together the voices of immigrant Asian women and the stories of her family, she uses English, French, Chinese characters, and photographs to guide the reader through a sense of unfamiliarity.

04 As a Means of Social Convergence

Inspired by Cha's fragmented works of thought, I began to think about the relevance of subtitles to her life: life as a foreigner abroad is not only fragmented and transient. Perhaps, like the subtitles, foreigners are superimposed on top of mainstream society and live as an afterthought. I have been living in the Netherlands for four years as an international student, trying to get a good education and get used to their culture. Especially when I arrived when COVID was just breaking out, I saw and heard first-hand the Asian hatred that I had only seen in the news and despaired, thinking that I would never belong to this society. It was only after a long time that I finally got used to life in the Netherlands. Because upon reflection, this ran parallel to the development of the fluency of my English. When I looked at the country with a balanced perspective, I started to see the lives of the foreigners around me that were invisible before. Some complained about their lives in the Netherlands, complaining about being discriminated against as foreigners, while others were happy with their lives as strangers, fulfilling needs that were not being met in their own country. Perhaps we can see this as a symbiotic relationship, where one is not simply beneficial to the other, but can sometimes work in a way that one exploits or assimilates the other.

In her book Lovebug, Daisy Lafarge likens these human interactions to the cannibalism of microbiology. She draws on a range of disciplines, including microbiology, mysticism, and psychoanalysis, to speak about the complex relationship between the human body and the bacteria, viruses, and parasites that inhabit it. I was particularly intrigued by the relationship between microbiology and human relationships among them.

When microbes live together in ambiguity, it's not just a ^[9]harmonious symbiosis: they walk a fine line, making choices that, in evolutionary terms, are most likely to survive. Sometimes they eat each other, sometimes they become part of each other, and sometimes it's hard to tell which one is dominant anymore. Sometimes the dominant entity changes its personality to be more like the smaller one when it eats the smaller one. They are too interdependent to call it mere cannibalisms. Human relationships are the same way. People have the potential to change each other, or sometimes become part of each other, just as one can cannibalize the other, or a parasitic relationship can form under the guise of symbiosis. Microbes have evolved in this way, and it is this dynamic that seems to have enabled them to survive as long as they have. It is almost superstitious to take a romantic and optimistic view of the integration of strangers into any mainstream society and imagine that they can be perfectly integrated into it. They fight and argue and make the best choices they can to live together.

Just as I, a cynic living as an Asian in the Netherlands, eventually found myself wanting to settle down in this country. Relationships are always changing dynamics. It grows by asking ourselves questions about how we exist in relationships and how much of ourselves we are. It's as if these questions are hidden behind the form of subtitles. The subtitles and the film are pointing in the same direction, but they are different mediums, and words translated into a foreign language are inherently difficult to convey fully and clearly. The nuances, the context and usage of a particular language cannot be translated. The foreign language and the native language will always remain two different concepts in the dictionary. But we at least try. We try to understand each other, we try to interpret. I think this inescapable fate of being an alien, and the attempts to overcome it, are embedded in the cold aesthetic of the subtitles.

05 Inward Gazing and Meditation

No matter how you look at the world, subtitles are going to be on your screen, and one of the reasons they're such a fascinating medium is that they appear on the screen at all times, like the thoughts in your head. I love the inner monologue of a protagonist in a good documentary or film, and when that monologue appears on a black screen, accompanied solely by the form of subtitles, I get goosebumps all over my body. The yellow, san-serif letters stare back at you, blandly saying, I am well interpreted. On the surface, subtitles are just a tool for interpreting the world, and you might be inclined to dismiss their fascination. But if you're someone who enjoys wondering about the relationship between subtitles and video, your relationship to the world, and asking existential questions, then you can see the connection between subtitles and meditation. We live in a phenomenal world, but we are more than just animals, with a complex mental system, and it is clear that ^[10]the soul is something that cannot be satisfied by mere physical pleasure.

Pensive Bodhisattva refers to all forms of Buddhist statues depicting Maitreya seated in a half-reclining position, meditating on the suffering of sentient beings in this world. One of Korea's national treasures, the Pensive Bodhisattva is seated in a chair, right foot resting on the left knee, jaw set in a mysterious smile. Looking at the statue, which seems to stare into thin air, I often realize that when we stare at something, it is also observing us. Have you ever felt that when you read subtitles, they are reading you? When I read SDH, or subtitles for the people who are hard of hearing, on screen, I feel as if the person telling the story is talking to me. Its unique voice, its quiet message, rising between the brackets, makes me feel more connected to the film content. I find it ironic that English-speaking Europeans usually treat subtitles as an intrusion, and I imagine leaning into this secondary medium to feel closer to the world, and I try to extrapolate their feelings more deeply. I'm not sitting in a chair pondering the truth like the Pensive Bodhisattva, but I do sometimes think about my existence as a stranger through subtitles. This is only possible because I am not part of the mainstream society here. It is a blessing in a way to be able to examine myself through 3rd person’s perspective.

06 Conclusion

In brief, subtitles are media that exist in context. Language, like subtitles, constitutes a context with time and space, and if it lacks any of these conditions, it means that your meaning has been lost. I think this is also the reason why our existence as expats is often decontextualised.

People often ask me a question as I am about to graduate: Do you want to continue living in the Netherlands after you graduate? Whenever I’m asked to answer this, I slowly sink down to the bottom of the screen as if I am the subtitle. I will never be part of the mainstream culture in the Netherlands. However, its is not particularly a sad thing. I just accept my fate as a outsider in the Netherlands. I and Dutch society live in symbiosis, sometimes in conflict while creating new contexts. Then I finally answer the question: Yes, I think I can continue to live in the Netherlands.

Bibliography