Beyond Screens - Mixed Reality User Experience Design
In the framework of my master’s degree thesis I created a set of design guidelines, based on the concept and principles of universal design, for the development of multidimensional interfaces in mixed reality environments. It was then possible to put into practice those guidelines in a field that has always welcomed favourably this type of technology, the surgical one.
Following a user-centered design process and the proposed ux design guidelines, it was possible to create a proof of concept interface, with an interaction design study on natural gestures , that could solve some of the gaps within the intra-operative process due to the contextual disabilities that are affecting the surgical operators.
The Challenge
It seems quite clear in recent years that the direction we are taking regards a progressive departure from those that a while ago were an essential prerequisite for the interaction with digital devices: the screens.
Traditional interfaces are increasingly evolving, becoming more and more invisible and integrated within a multidimensional environment, towards a condition ‘beyond screens’. In this regard, mixed reality offers a field of experimentation also supporting the human need to embody their own technologies.
Starting from these premises, through an intensive research, analysis and a user-centered design approach, I developed my master’s degree thesis project called Beyond Screens - Designing universal experiences within mixed reality environments.
Preliminary Research and Analysis
Historical-Anthropological Analysis
Starting from a historical-anthropological analysis on the relationship between the man and his extensions, through concepts linked to the world of cybernetics and transhumanism, I have defined how this connection has strongly evolved towards a progressive incorporation.
TECHNOLOGY
Mixed reality, through the creation of multidimensional environments, is able to bring to extremes concepts such as internet of things and ubiquitous computing, making traditional interfaces more and more evanescent.
ACCESSIBILITY
Because the perception of the frame that separates the organic and the virtual reality is becoming more and more thin (reality-virtuality continuum), the issue of accessibility assumes an increasingly important value. I decided to deal with this issue through universal design principles that takes accessibility to an extreme.
UX Design Guidelines for Mixed Reality Environments
After tracing the initial premises through an intensive preliminary research and analysis, I presented 8 design guidelines for the development of interfaces for multi-dimensional environments. Those best practice were developed taking into account the current evolutionary trend of humans in relation to digital technologies as well as the idea of universal design and its principles.
These guidelines introduce the idea of a holistic design that requires to consider various elements in synergy, in this case: user, environment and technology. And to understand how those three elements are interacting with each other.
Proof of Concept
I then decided to develop a proof of concept interface in order to validate and iterate the proposed guidelines. The P.O.C concerns the development of a Microsoft Hololens interactive application to be introduced in a field were contextual disability is high: the surgical field.
The whole project was developed through a user-centered design approach, following the design thinking process.
Empathize - User Research and Anslysis
Preliminary Research
To choose the primary audience to focus on it was necessary to identify the roles within the surgical team. To do this I carried out a desktop research and read some introduction to surgery books.
There are two main roles that need to always have their eyes focused on the operating table and therefore limited opportunities to observe and interact with external monitors: surgeons and nurses. I decided at this early stage to focus on the surgeon role.
Netnographic Research
To better investigate the role of the surgeon and to be able to prepare the interviews effectively I conducted a netnographic research on some discussion forums dedicated to surgeons.
Interviews
I then proceeded to interview two surgeons and also two nurses to get an external view on that role. This is what emerged:
Contextual Inquiry and Task Mapping
Due to logistical problems, it was not possible to carry out a real contextual analysis. But online I was able to view some audio-visual documents that allowed me to understand how surgical operators work in the field. The contextual analysis confirms the things that emerged from the interviews.
Through task mapping I was able to define user tasks in the form of a storyboard. I also added the variables emotions and thoughts, typical of the user journey map, to get a complete picture.
Segmentation and User Modeling
I then could proceed with the segmentation of the 'surgeon' target into three clusters, through the variable defined by Kotler P. 'lifecycle', in this context renamed more properly ‘careercycle'. Below there are the three clusters that I identified. For each cluster I developed a personas model and an empathy map in order to explore needs and wants:
User Stories and User Needs
After defining some user stories I was able to put together all the insights collected previously and to define the needs of the target:
Goals and Experience Strategy
Make surgical work more efficient and provide the surgeon with a "sixth sense” that allows him to overcome the limits of his 5 senses during surgery
HOW?
By developing a system that allows surgeons to consult medical documents, real time images and 3D scans autonomously
Let the surgeon iteraci with the system while keeping their hands free and sterile
HOW?
By leveraging the potential of mixed reality and by designing interactions that reflects human natural gestures and the ones related to the surgical field
IDEATE
Logical Design
Within the logical design I defined the sections, the architecture/flow, the labeling and the general semiosis (how the system should talk to the user).
In this phase I defined the system metaphor which would lead the entire project. Since this is a project that aims to allow consultation and navigation, the metaphor which is the most familiar to this type of context is the archive.The elements that characterize the real clinical archiving environment must therefore be translated into the idea of a digital archive:
At this stage I also started thinking on how the interface would be. The general structure has been designed to be modular in order to allow total freedom of interaction and management by the user.
Interaction Design
Starting from the work done by Wobbrock J. O., Morris M. R. and Wilson A. D., I then decided to conduct a simulation session with some users, in order to identify recurring patterns of interaction to be applied into mixed reality context as natural gestures.
I have provided users with a scenario and some tasks. They were asked to manipulate and perform some operations against physical objects considered as virtually generated elements in order to identify some interaction patterns.
Visual Design
Within the visual design phase colours, typography, iconography and shapes have been designed to fit the user, the technology and the environment.
Colours: needs to fit an operating room without getting mixed up in the background or faded by the strong lights. Therefore the colours needs to be in contrast with the surrounding environment.
Typography: the use of sans serif fonts is preferable in order to ease the reading within an environment that is characterised by movement. The size of the typography depends on the distance and on the physical ability. I conducted a flash test in order to determine a safe font size based on surgery working-distance which is approx. 50 cm.
Iconography: the use of solid icon is preferred in order to ease the reading and reduce cognitive load. After selecting all the icons I tested them out to evaluate: findability, recognition, information scent, attractiveness.
Shapes: in order to be consistent with the idea of modularity, I favoured the use of geometric shapes with clear cuts, which also helps defining the areas of content more clearly.
High Fidelity Mockup
