When Language Becomes Art

By Katie Marie

Can humans communicate through a single unified language? 

Yes, according to Haytham Nawar, associate professor and chair of the Department of the Arts, who is using artificial intelligence and machine learning to develop a universal pictographic language that could bridge divides and enhance cross-cultural understanding.

Why pictographic? 

“The idea is that the machine creates a new language based on how humans in the past created pictographic languages, starting from hieroglyphics all the way to emojis,” said Nawar, who has long been interested in scripts and pictographic languages, such as Egyptian hieroglyphs, cuneiform scripts used in the Near East and ancient Chinese writing systems. These image-based languages were humanity’s earliest communication systems, the precursors to the script-based languages we use today.

With his expertise as an artist and designer, passion for linguistics and fascination with technology, Nawar is exploring the interrelationship between humans and technology, particularly how the advent of new technologies transforms the ways we communicate and produce art, for example. 

In his book, Language of Tomorrow: Towards a Transcultural Visual Communication System in a Posthuman Condition (The University of Chicago Press, 2020), Nawar theorized about the possibility of using tools such as AI and machine learning to create this universal pictographic language. In his current research, Nawar is deepening his understanding of the subject by putting theory into practice. In short, he is using AI to develop the new language. “The project explores this challenge between human and machine intelligence,” said Nawar. “Can a machine understand the logic of ancient communication systems and create new characters or even new languages?”  

“The way it works is that you actually train the AI to learn the logic of writing and drawing other languages. You feed data to the machine, and it begins to identify patterns and predict new ones. When you input more information, you get more accurate results,” said Nawar. 

To build the program that would create his language, Nawar enlisted the help of Ahmed ElGazzar, an AI engineer completing his PhD at the University of Amsterdam in the Netherlands. 

Nawar compiled a database of more than 180 writing systems, ranging from ancient pictographic scripts to alphabets and abjads, which he is using to train the AI. The training process has been long and complicated, full of various stages of trial and error that have taken weeks, sometimes months. Overall, Nawar has been working on the project now for more than two years. 

“The machine generated new hieroglyphs based on ancient Egyptian grammar, new characters that look like they are ancient Chinese or cuneiform, but they are not — and they are machine-made.”

He first attempted to train the AI to recognize shape patterns among the different languages, only to realize there was not sufficient representation of specific images within the database. Other trials teaching the AI to recognize the meaning of different characters and identify patterns proved to be too labor-intensive. “Later, I came around to the idea of educating the machine in the logic, or rather the grammar, of writing a pictographic language. I taught the machine to understand how the ancient pictographic languages were actually formed,” Nawar reflected. 

Using this approach, Nawar began to see some success. “Our AI commanded the machine to generate new characters based on what it had learned from the ancient scripts. It started to generate a new language based on the logic of the old.”

So far, his work has produced not one universal pictographic language, but the beginnings of new communication systems based on the old. “The machine generated new hieroglyphs based on ancient Egyptian grammar, new characters that look like they are ancient Chinese or cuneiform, but they are not — and they are machine-made,” Nawar said.

The machine has generated new characters from Arabic script as well, though these are less visually dynamic, as they are based on the letters of the alphabet, representing only 29 inputs (even with their letter variations), rather than the thousands of images included in pictographic scripts, Nawar explained. He is using technology to replicate the complicated and messy process typically undertaken by entire civilizations to develop a new communication system. 

Another challenge of Nawar’s work includes illustrating the technologization of this human process through artistic representation. “The machine learning process is very interesting, but it is not really visual. It takes place within the computer itself. This is why I created a short animated film that simulates the process through which the new pictographic characters are generated,” Nawar said. 

He also chose a more archaic artistic tool to illustrate the machine’s process, a flip book that details the transition between the original character from an ancient script and the new computer-generated one. 

A third representation renders AI-generated characters as three-dimensional objects. “We always perceive written communication in a two-dimensional format, but why can’t we also play with the idea of communicating in a three-dimensional format?” Nawar asked. 

Nawar selected a hieroglyphic symbol, a Chinese character and an Arabic letter — all generated by AI — to present in three dimensions using polylactic acid plastic, which is natural, renewable and non-toxic. 

It was at this stage in his work that Nawar felt he was ready to share his progress publicly. In August 2021, he presented his work at an exhibition celebrating the cultural value of Arabic calligraphy, which was hosted by the Saudi Arabian Ministry of Culture at the National Museum of Saudi Arabia in Riyadh. 

The newest phase of Nawar’s work continues to play with the idea of machines replicating and transforming human communication processes. One component involves rendering the characters three-dimensionally again using different materials. It was inspired, in part, through his collegial relationship with Hanadi Salem ’83, ’87, professor and chair of AUC’s Department of Mechanical Engineering. Salem is also founder and director of AUC’s Additive Manufacturing Centennial Lab, the first Wire Arc Additive/Subtractive Manufacturing (WAAM) integrated robotic system in Egypt and the Middle East.

 The WAAM integrated system is a type of 3D printer that uses an industrial robotic arm to generate parts by depositing layers of molten metal wire. Its primary use is in the manufacture and repair of metallic medium-to-large industrial parts. However, since the opening of the lab, Salem has been interested in exploring more diverse applications of the technology. 

“I approached Professor Nawar about working together to use the WAAM robotic system in design contexts because I felt that this represented a valuable integration of both engineering and the arts,” Salem recalled. “Interdisciplinary collaboration in research, development and education challenges us to become better and more creative critical thinkers.” 

The project to generate the metal objects is well underway. “The major challenges for my team have been preparing the complex pictographic shapes for interpretation by the robotic software and creating an optimal tool path for building the objects,” said Salem. 

“We want to see if a robotic arm can engrave the characters. We are challenging the machine not only to create a new language but to also apply the earliest methods that humans used to communicate with images: writing on walls.”

Another collaboration between Nawar and Salem that is under development will continue to play with the relationship between humans and machines, between the past and the future. This step will involve engraving the characters in stone or metal using a mechanical arm. It is a seemingly simple process that is again highly complex in its execution. 

Nawar noted, “We want to see if a robotic arm can engrave the characters. We are challenging the machine not only to create a new language but to also apply the earliest methods that humans used to communicate with images: writing on walls.”

Using AI and the tools of mechanical engineering to create new modes of communication and artistic representation may seem like a revolutionary project, and in some ways, it is. As artists like Nawar collaborate with coders like ElGazzar and mechanical engineers like Salem to apply new technologies in creative ways, they are breeding innovation in both practical and artistic terms.

“AI is already being widely used daily in human life, but there is a sense of liberty in using it for artistic experimentation,” said Nawar. “It is this laboratory feeling that you can really do more with the technology without industrializing it. In artistic contexts, you can experiment, exhibit your work in progress and get feedback. You are able to see how an audience, a community, interacts with what you have created.”

Salem echoed this notion that experimentation with technology is helping to fuel artistic innovation. “We are witnessing exponential growth in the applications of machine learning and AI in all domains of art, including visual, performing, spatial, transmedia, audiovisual and narrative,” she explained. “I believe that machine creativity in arts and design represents an evolution of ‘artistic intelligence.’”

“AI is already being widely used daily in human life, but there is a sense of liberty in using it for artistic experimentation,”

These collaborators are also engaged in an ongoing process of discovery that began in ancient times. It is the humble process by which humans identify, experiment with and adapt new technologies in their work and daily lives. In aggregate, these small efforts bring about radical transformations within the human condition. 

“Technology has always been there, and we have always piloted new tools through artistic and creative practices. Mixing colors and identifying new materials with which to make them is also a form of technological experimentation,” said Nawar. “Da Vinci and Michelangelo, for example, were always experimenting with the technology of their time, and now we are using different ones, such as coding, programming and artificial intelligence. Technology is the tool we use to create art. Soon, maybe very soon, there will be new technologies to work with.”

For Nawar, engaging in this experimental process is also a way to understand how incorporating ever-evolving technologies into our lives changes the very definition of humanity. He is looking at how humans have evolved their modes of communication and relationship with technologies in the past while simultaneously engaging in this process with the technologies of today. In doing so, as Nawar puts it, he is “questioning what is human and trying to understand what communication will be like, what the world will be like as this process continues. What is the human of the future?”