Maps convey information in a graphical form. There is no general recipe to create a map and its symbols; each map has its unique workflow and next to the cartographic conventions, there is a space for creativity and new ideas. Map-making is an iterative process where a cartographer interacts with both data and the author’s ideas and seeks the most suitable ways of depicting information for specific purposes. Interacting with data and symbolisation, composed of different visual variables (such as colours, shapes, sizes, etc.) makes the cartographer familiar with the visualisation and therefore subjective. In other words through the creator’s eyes developed symbols can work perfectly and intuitively. However, the final visualisations are intended to be used by larger groups who might interact with such kind of visualisation for the first time. These users should be able to easily decode the information which the map maker has visually coded into the visualisations. But how do we make sure that a map meets its main aim and can communicate with a larger audience? This is where user studies are great assistants to a cartographer. With user studies, a map maker can directly communicate with potential map readers already at the early stage of map-making, get more objective reflections and useful feedback about the visualisation, which later can be implemented and integrated into the iterative process of map creation.
Designing the user study
Within our interdisciplinary project DigiKAR (‘Digitale Kartenwerkstatt Altes Reich: Historische Räume neu modellieren und visualisieren‘) we are dealing with complex historical datasets. We develop new concepts of collecting, modelling, and visualising early modern spatial data from the Holy Roman Empire. In the period between 1500 and 1800, not only did data availability rise but also the way of governing space and executing power changed dramatically. This has affected people’s perception of space in daily life.
Our main task, in IfL, is to develop and test new concepts and multivariate visualisations for historical data. For this purpose, we created seven different point-based symbol variants (Figure 1). Each symbol stands for a certain location, has different visual complexity and shows multivariate historical attributes. To evaluate the perceptual aspects of the developed symbols we decided to conduct a user study. From our point of view, the sample for a study dealing with perceptive aspects is preferably diverse (different age groups, backgrounds, map usage, etc.). Therefore, as our sample members, we happily invited IfL colleagues from all departments and several external guests to contribute, evaluate and give their feedback regarding developed symbols.
As the user study was focused on visual aspects of developed symbols, we exchanged the complex historical data with a test dataset including fictitious place names and a simple storyline to make the experience more enjoyable for our potential users. Each symbol was representing a town. The storyline combined five types of events (market, coffee and cake, concert, dancing class, wine tasting) and five different organisers (baker, mayor, pastor, pensioner, teacher) of those events. The events had fixed positions at the edges of symbols, while organisers were presented with assigned colours (red, yellow, blue, orange, purple) (Figure 2). Everything seemed to be ready for user study, but even the best-planned user study needs to be verified using pre-tests. The pre-test is an initial stage of the study to evaluate the planned research method(s) (in our case a questionnaire). With pre-tests, we integrated individual interviews which provided valuable information about the developed symbols (modifying colours and sizes, adding labels and explanatory texts), ambiguous tasks and questions. After implementing all the changes and finalising the research method instrument, the user study was ready to be carried out.
What we tested
The map symbols were presented to participants through a questionnaire (in German or English). Each questionnaire included only three point-based symbol variants (symbol A, B, and C) to avoid both overwhelming a user and possible learning effects by interacting too long with similar data and tasks. The questionnaire was designed in different thematic sections combining general questions, map-related tasks, and evaluation. With the general questions, we collected demographic data and information on how often participants were using maps. The map-related tasks were designed to determine the accuracy of interpreting the tested symbols, i.e. whether users could solve the given tasks correctly. The questions were related to how the participants understand the symbols in general, and how they find specific information such as hetero- and homogeneity of data and different combinations of attributes. In addition to accuracy, it was important for us to know whether users were confident while solving the tasks. To find this out, we asked the participants to self-evaluate their confidence level after solving each task. Other important criteria in evaluating the symbols were readability, overall performance, and aesthetics. Based on these criteria, users were asked to evaluate each symbol on a given Lickert scale. Our last evaluation criterion was speed, i.e. the time a user needed to complete given tasks. For time monitoring, participants were self-tracking the start and the end time of completing the tasks related to each symbol. At the end of the questionnaire, users also had an opportunity to write comments on the symbols. The comments are always difficult to quantify but important to take into account. Many of them gave valuable information and suggestions about visual aspects of developed symbols; for example, colours were perceived differently regarding the shapes and the sizes they were applied to, difficulties in differentiating certain colour hues placed next to each other, uncertainty in distinguishing grey scale levels, etc.
Outcomes in numbers
As a result of our user study, we have collected quantitative and qualitative data from 60 participants. As the study was paper-based, we digitised the data, structured it, and coded the quantitative and qualitative parts. For illustrating insights and identifying trends we used descriptive statistical analysis. Different coefficients and generated charts showed that for specific tasks some symbols performed better than others. Therefore, for the final assessment, we decided to use a coefficient combining selected criteria: accuracy, certainty, readability, overall performance, appeal, and speed (order is based on the importance and weight of each criterion in the overall coefficient). Out of all the symbols, the ‘Snowflake’ symbol showed the best performance (Figure 3). Accordingly, we will continue to develop this symbol further for our interactive visualisations within the DigiKAR project.
To conclude
User studies are exciting parts of research which facilitate communication between the map maker and the map reader. However, while designing a user study, it is challenging to take into account all possible factors which might affect results and therefore interpretation. In our case, pre-tests were mainly carried out in English whereas most user tests were in German. It resulted in ambiguity for certain tasks. Also, we tested symbols and their perception on paper, yet, they will be eventually implemented in a web-based visualisation. Despite these issues, we believe it is important to test visualisations at early stages (and preferably frequently), as they can point to weak spots in the visualisation and provide ideas on how to improve them or even how to visualise things differently. I would like to use the opportunity and thank each of our colleagues and participants for this pleasant experience, for their valuable contribution and curiosity which was present throughout the whole process of the user study at IfL; and for those who are interested in further developments, I would recommend keeping an eye on our visualisations, maybe soon you will find historical maps full of ‘Snowflakes’.
Mariam Gambashidze is a member of the Geovisualisation Research Team at the IfL.