Helen Papagiannis
Photo credit: Dave Gillespie

Helen Papagiannis

Dr. Helen Papagiannis is an internationally recognized expert in the field of Augmented Reality. Her work has taken her around the globe as a researcher, designer, author and speaker focusing on storytelling and creating meaningful experiences in AR. Her book Augmented Human: How Technology is Shaping the New Reality was published in 2017 by O’Reilly Media and provides readers a look into our virtual future, how AR is evolving, where the opportunities are and where we’re headed. We sit down and talk about art, tech, yoga and the transformative power of Augmented Reality.

An interview with Ivonne Karamoy

Let’s start from the beginning.

You got your BA in Fine Arts, Art History and English Literature. What were you hoping to be then?

I want to go back even a little further first. I was very lucky to attend a fantastic public high school, Wexford Collegiate School for the Arts, in Scarborough [Ontario, Canada] that had a phenomenal Visual Arts program: Printmaking, Photography, Sculpture, Painting, Drawing, and the list went on. I would often stay after hours in the studio making. I knew I wanted to be in the Arts, but I also loved tech. We had access to a computer in the Arts department that ran a super early version of Photoshop. I was intrigued by the possibilities of using the computer to make art, yet there was something I loved so much about the physicality of making art by hand: the materials, the textures, the smells. I began to combine the two: scanning my physical artwork and continuing the process of mark-making digitally. I didn’t know what I was hoping to be, but I knew what I was interested in and I followed my curiosity. My peers and mentors at Wexford inspired me and I owe a lot to my highschool experience. The community and uplifting environment at Wexford was outstanding. It only made my love for the Arts stronger.

I continued down the path of art and tech in my undergrad degree at the University of Toronto. I worked in galleries. I always loved being surrounded by artists. And I also loved digital media. I remember taking a course called “Research and the Arts” where I designed and built my first website (using Dreamweaver) about artist Cindy Sherman. At the end of the course, my professor, who also happened to be the Digital Media program director, asked if I wanted a job designing websites for the university and I gladly accepted.

I was focused on the graphic design side of things, but I really wanted to learn to code. When I finished my undergrad degree, I attended Sheridan College, and enrolled in a program called Interactive Multimedia. I learned HTML, Lingo, Actionscript 1.0, Flash, ASP, PHP, Javascript. We also did a lot of audio visual production, built CD-ROMs, and even made QuickTime VRs. Keep in mind, this was all the latest tech at the time. I enrolled at Sheridan because I felt like a better understanding of the programming side of things would empower my design work. In a way, looking back at that experience, it’s how I approached working with Augmented Reality when I first came across it: getting under the hood and tinkering. I began to break demos to understand how they worked.

Art has always stayed with me, as has the desire to combine my love for tactility and materials with the digital. I think that’s what drew me to Augmented Reality — it combines that physicality with the virtual. I love how those two worlds come together in AR.

Helen Papagiannis Cinemagician Exhibit
Visitors interacting with "The Amazing Cinemagician", an interactive installation that Helen exhibited at the Ontario Science Center in 2010.
Helen Papagiannis Cinemagician Exhibit
Photo Credit: Pippin Lee.
So you started as an artist, then went into web design–how did you discover augmented reality?

I had always admired Bruce Mau, and was in awe of the studio’s latest book at the time, Life Style, which was released in 2000. While I was a student at Sheridan, I pitched Bruce Mau Design and Bruce agreed to be our client for a school project. We filmed and produced a short documentary, designed and built a website, and of course, a CD-ROM! Everybody wanted a CD-ROM, it was all the rage back then.

After I graduated from Sheridan, Bruce called me in for a meeting one day and presented Massive Change to me. In the back of my mind I was thinking, “Does he want a website, or a CD-ROM, or maybe both?” To my surprise, he didn’t want either. Bruce asked me to lead the project and work with him on the initial design and research for Massive Change, including a 20,000 square foot exhibition commissioned by The Vancouver Art Gallery, a book to be published by Phaidon Press, and starting a post-graduate program called The Institute without Boundaries in collaboration with George Brown College. I was in awe. It was an incredible opportunity and of course, I seized it! I pinched myself going to work each day.

Massive Change was a bold look at the power and promise of design to change the world through new inventions, emerging technologies, and ideas posing the question, "Now that we can do anything what will we do?"

I led the research on Massive Change, identifying different areas of design for us to explore and that’s where I first came across references to Augmented Reality (AR). AR returned for me in grad school and I became fascinated by it. I remember seeing my first demo: it was very simple consisting of a blue 3D virtual cube, but it had me mesmerized! There was something in my space that wasn’t really there. It was magical. I began to ask, how can we use this new technology to tell new stories and to create compelling experiences? And these are the questions that have defined my work over the past 12 years.

I went into mad scientist mode after seeing that first demo. I made things and broke things. I began to work with constraint as a design principle. When I started working with AR, there was no sound, and there were only two or three AR tracking markers, max. Rather than using these parameters as a way to limit what I was doing, I looked at it as more of a challenge. I thought, "Okay. Within these constraints, what can I create that is compelling?""

Inspired by AR psychotherapy studies for the treatment of phobias, Helen created “Who’s Afraid of Bugs?”, the world’s first Augmented Reality pop-up book, in 2011-2012.

Augmented reality was very new at the time. What made you want to pursue it?

Because it was new and I love exploring unchartered terrain. Probably because there were also no rules or clear paths, it was ripe for experimentation. Being an artist at heart, having a blank canvas like that, and being able to make and to define new things, all of that was super exciting to me. That curiosity, that wonder—figuring out how this magical thing was possible, and what I could make with it as a new medium, that’s what made me want to pursue it.

You’ve been working in the field for over 12 years. How has augmented reality changed and evolved in that time?

Aside from the evolution of the user experience from primarily desktop-based AR experiences, to mobile, and to untethered AR headsets like HoloLens, we’re seeing a major shift in the way the industry is thinking about AR. The first wave of AR asked, "Can we do this?" and was focused primarily on the technology. Now that we know we can do this technologically (and of course there is more to be done, we’re just getting started), the big question is "What are we going to do with it?"" We’re seeing a greater emphasis placed on content creation and meaningful experiences. And this is absolutely key, because if we don’t have incredible and memorable experiences, the hardware is nothing more than a paperweight. We need artists, designers, and creatives of all types to help tackle this. There’s a major creative gap here that needs to be filled and I think the industry is coming around, but a big part of this will be not only hiring engineers, we need to look to the humanities as well: artists, filmmakers, musicians, performers, writers, poets, anthropologists, sociologists. AR needs storytellers and critical thinkers from multiple disciplines.

Helen wearing HoloLens
Helen using Microsoft HoloLens, an Augmented Reality Headset.
What do you think designers, artists, and storytellers can bring to the table that others can’t?

It’s the ability to see across disciplines, and to see relationships that are not otherwise apparent. Artists do that very well.

I often ask, “Does the technology drive the storytelling, or does the storytelling drive the technology?” By bringing in artists and creatives who are not necessarily programmers or computer scientists, but who are curious about the medium, we can allow storytelling to lead and define what the technology can become.

I also think that designers, artists, and storytellers experience the world differently, with all of their senses, and that’s something we need to explore in AR. We often think about AR as a visual medium, but it’s not limited to that. Reality isn’t just visual, and not everyone is sighted.

Helen Papagiannis book
Helen's new book Augmented Human published by O'Reilly Media is "not just about augmenting reality, it's about augmenting humanity".
Let’s talk about your book, Augmented Human: How Technology is Shaping the New Reality. There’s a chapter in the book that talks about each of the human senses and how technology will extend those capabilities. Can you tell us about one of those ideas?

Yes, an example of this is vision technologies that explore how the human brain can be trained to see with the other senses, like hearing, effectively learning how to see with sound. The vOICe system [OIC = “Oh, I See”] developed by Peter Meijer, translates images from a camera into audio signals to assist people who are congenitally blind to see. The vOICE consists of a pair of sunglasses with a camera, connected to a computer and pair of headphones, and it can also be used on a smartphone by downloading the software and using the phone’s built-in camera. The camera scans your environment from left to right and the vOICe software converts pixels into sound, transforming your surroundings into a soundscape for you to see with your ears. Frequency is used to indicate an object’s height, and volume is used to represent brightness. With training and practice, wearers begin to identify different sounds with objects.

One of the technologies you discuss in the book that fascinates me is haptic technology. Can you explain what it is?

Haptic technology provides tactile feedback from a virtual object, as though you can touch the virtual. Touch in physical reality is something very intriguing to me. It tends to verify that something exists, that something is real. In AR, for the most part, we can see, or hear, virtual information, but if we reach out to touch it, there’s nothing there, or we’re left touching the glass screen of the device we’re looking through. Haptics changes that.

The first time I experienced haptics was in 2011 at the Magic Vision Lab in Adelaide at The University of South Australia. Wearing a head-mounted display I was able to see an AR fish appear, and holding a special stylus, I was able to reach out and feel the individual scales of that fish.

You actually felt it?

Yeah, it blew my mind! It was pretty unreal. It threw me off because I knew that fish was virtual, yet I could feel it and verify it’s existence through a sense of touch.

That’s so exciting! I can wrap my head around augmented reality as it pertains to sight. But touch, feel and taste…

Taste, smell, hearing. There’s so much that can be done. I think we’re limiting ourselves by focusing on just the visual.

Helen speaking on Advancing AR for Humanity at Augmented World Expo (AWE) 2015.

In Augmented Human, you give us some examples of how AR has helped to advance artificial intelligence and machine learning allowing technology to augment human experiences even more. With all this advancement, is there a danger of becoming too reliant on technology to the point that we lose our physical and emotional connections with our reality and with other people? How do we design responsibly?

Designing responsibly is absolutely key. We do this by engaging all disciplines. The Humanities are particularly good at asking critical “Why?” questions, and we need a lot of that critical thinking to understand the potential social, cultural, and ethical implications, alongside solving the hard technical engineering challenges and informing those design decisions.

Helen Papagiannis talking at O'Reilly conference
Augmented Human talk at O’Reilly Solid Conference, San Francisco, 2015.
In addition to your work you also do a fair amount of speaking engagements and have done so for a few years now. How do you find confidence to speak in front of an audience, or what gives you confidence?

I think confidence comes from believing in what you’re doing. AR was completely new to me, and there’s certainly something scary about that. It’s about having faith in yourself, in what you’re doing, in where it’s all going. That gives you the power to define your own rules.

We all experience self-doubt. It’s human. But, in those moments, leaning on incredible people to help lift you up. Confidence, for me, definitely comes from the incredible support system I’m grateful to have.

And of course working your butt off. Confidence comes from hard work. Practice builds confidence.

Helen Papagiannis by Ally Chadwick
Photo credit: Ally Chadwick
I read that you do yoga. How does that heighten your sense of reality?

I’m interested in mindfulness and an awareness of our bodies in space. I think that all feeds into AR and virtual environments as well.

I spend so much time in front of a screen, so I think it’s very important to return to the body and to remember our breath. How many times a day do we have a deep belly breath? For me, when I have free time, I do yoga, or go for a walk in nature.

Also, with aerial yoga, where you’re hanging upside down, it can be really disorienting. But it allows you to see your world in an entirely new way. I think AR can help us do that too, to gain a fresh perspective.

Helen Papagiannis
We Are Wearables, 2017. Photo credit: Billy Lee
Have you had any mentors in your career?

So many, and I am incredibly grateful to be surrounded by such amazing people. In high school, my art teachers were fantastic and super inspiring. I’ll never forget the mentorship of Anne Jones and Peter Marsh. Dr. Caitlin Fisher, my Ph.D. supervisor, is an incredible woman. Extremely talented, and above all, an amazing human being. My parents have been my best mentors. They are my superheroes.

Who have you either mentored or advised, and what was that like? How did that originate? What were the benefits of being a mentor or advisor to someone?

Helping others is so important. We’ve all been mentored, so I think a big part of that is giving back. I get a lot of questions on and off stage.

The benefits of mentorship are simple: you have more hands on deck for the bigger cause. I like to see people engaging with and thinking about the medium in new ways. You get to see others exploring and taking care of the thing you love so much, and being able to contribute to that.

The more people that get involved in a certain field, the more ideas you have and the possibilities become magnified and the solutions too.

Exactly. And that’s the other big part of it. Having new people work on solutions and discover new questions. Look, how often does a new medium come around? The more people you have involved early on to help define what that can be, the better.

Helen Papagiannis
Camp Festival, 2014. Photo credit: Bram Timmer
What do you see, or hope to see, for the future of AR?

My hope for the future of AR is that it enriches and enhances our reality, not supplanting it or replacing it. I want to see AR create good in the world, uplifting and empowering people in their daily lives and igniting curiosity and creativity. This is what we must design for. May these new realities be deeply fulfilling and greatly benefit humanity.

Dr. Helen Papagiannis will be hosting a book signing for her latest book, Augmented Human, this Saturday, December 2, 2017 between 7pm and 8pm at House of VR in Toronto located at 639 Queen St W. Bring your copy of the book or purchase one at the event. Register for this free event.

with love from the Women&&Tech team