02: Discover
Low Light Ruck March
We started our Discovery phase with perhaps the toughest research study ever, especially for college students. We accompanied the ROTC cadets on a low light ruck march as part of an ethnographic study, to understand how they navigate in the dark.
During the exercise, we were able to experience first hand with cadets, what it is like to stumble up a mountain, through rocks and brush in almost complete darkness.
Following the end of the march, we had the opportunity to speak freely with cadets in impromptu, unstructured interviews about things that we had observed during the march.
02: Discover
Semi-Structured Interviews
Following the field study, 7 semi-structured interviews were conducted with cadets and training officers of the ROTC, to supplement our research and understanding of the challenges and procedures of low light training and navigation.
Some of the research questions we looked to get answers were:
Tasks Undertaken
- What gets the cadet to start their task?
- How they will know when the task is complete?
- What do they actually need to know to complete the task?
Artifacts and Skills in Use
- What tools or information do they use in the course of the task?
- Sensory / Motor / Cognitive Skills Required for the task?
Conditions Under Which These Tasks Are Done
- How do usual ROTC ruck marches look like?
- How are ROTC squads / units / commands structured and how do they communicate.
- Will these forms of communication change at different stages?
02: Discover
Prior Art Reviews and Research
In parallel to user research, we began secondary research and a literature review to understand what the current landscape of research in the ROTC and Military community looked like.
We looked at field manuals, guidebooks, research papers on similar efforts, and existing legacy systems.
03: Define
User Description
After our initial research, we converge as a team and laid out the Roles, Needs, Wants and Goals of the different players we had identified from our research:
03: Define
An ROTC Squad
As part of our research analysis, we identified what's the structure of an ideal ROTC squad, and the following aspects of the squad:
- What are the various roles within an ROTC squad.
- What is the chain of command and what communication models are used.
- Current ecosystem and opportunities.
An ROTC squad has three different roles:
Line Leader: This is the person in command who controls where and how the squad is supposed to move. The line leader is the only person in contact with other ROTC squads, usually via radio.
Line Cadets: These are regular cadets who follow orders and walk in a single file. They are young, athletic, and are often carrying their rucksacks, guns, and helmets. They have no access to radio and communicate using hand and arm signals while walking.
Line Anchor: A Line Anchor is usually an experienced person whose job is to ensure that no one is left back and maintains the integrity of the ROTC unit.
How does a unit communicate?
An ROTC unit communicates using hand and arm signals in situations where total silence and stealth is required. These situations are also encountered in the dark most of the time and thus are hard to follow as well.
The unit uses hand and arms signals in a method called "Daisy Chaining" where the line leader gives a hand signal for say, "Stop" which is observed by the person behind the line leader and repeated by him, and subsequently by the people behind each other. Thus the communication flows from one person to the other and in larger squads, it is several moments before the message reaches the last person in line.
03: Define
Affinity Mapping To Identify Patterns
After the field study and initial interviews, we organically flowed into the Define Phase, meeting as a team to debrief and share our experiences and findings from the field study, interviews and secondary research.
We used Affinity Mapping to collect thoughts and start exploring trends that were emerging. These sessions prompted more questions by our team that we used to direct further interviews with our population.
This particular step helped in consolidating all the data gathered to find the most required features of the product and the motivation behind it.
04: Ideate
Brainstorming
The Ideation phase consisted of several brainstorming sessions, where we used a collaborative “Yes and…” culture to rapidly form dozens of ideas. Later, we would take time to flesh out the ideas a bit further, before evaluating them based on feasibility, and if they truly fit the constraints we had identified earlier.
Ultimately, we were left with three viable ideas and hammered these out completely into different concepts:
- A bread crumb device much like the Hansel and Gretel fairy tale,
- A pressure sensitive sleeves worn by all cadets that communicated based on gestures,
- An accelerometer glove that could read and translate hand and arm signals to a haptic vest worn by all cadets.
We created a poster of our 3 ideas along with the progress and process till that point and prepared to showcase it at a poster session to get unbiased feedback from our peers, advisors, Teaching Assistants, and faculty.
04: Ideate
Peer Feedback and Participatory Design
We presented these designs to our peers, receiving third-party, unbiased feedback on our methods. Their criticism reassured us that our methods were truly user-centered as well as helped us identify other technologies that could help support our solutions.
However, their feedback could not move us in the best direction for our intended population. So, we presented our ideas to the Commanding Officers. Because their feedback found good and bad aspects in each design, we held a Participatory Design Session with them later to design a solution that had better buy-in and feasibility from all.
03: Define
Final Design
With the help of peer feedback and participatory design with members from the ROTC, finally converged on our final idea which was:
A hat with vibrational motors worn by all cadets that could receive directional commands and translate them into vibrational cues
Traditionally, the ROTC cadets used "Daisy Chain" method to communicate which resulted in longer times before everyone in the complete squad had received the command.
With the proposed design, the line leader could give a command, which would be instantly transmitted and received instantly by the vibrational motors in the line cadets' helmets. The vibration pattern in the motors would indicate what command it is, and leveraging different directions and patterns, multiple commands could be implemented.
How Does The Proposed Design Measure Against The Design Principles?
Simplicity
Using simple patterns and directional commands was easier to understand than complex hand signals.
Hardiness
Keeping the motors in the helmet meant that they would not be damaged, since the helmet is one of the hardest pieces of equipment.
Fail Proof
The forehead/temple area is one of the most sensitive area to vibrations, this meant that the cues were hard to miss by the the cadet.
Undetectable
Since vibrations can only be felt, and not seen by unwanted entities, this made them a secure form of communication.
Multi-Modal
By using different directions and patterns, we could accommodate multiple commands.
06: Evaluate
Hiccups before Evaluations
1) Testing with the ROTC cadets requires DOD authorization
It’s important to note that during discovery and development, we were submitting paperwork to the IRB for ethical approval for this study. Because the ROTC program falls under the Department of Defense, there are special government clearances we needed to get approved. At the time of testing, we had not yet gathered that approval.
Work Around:
In anticipation of this challenge, we submitted a second study that used generic college students as our intended population and pivoted the device towards one for a hiking-in-the-dark-like scenario.
2) IRB states that you cant blind-fold participants in a user test
The initial test plan included blindfolding the participants to simulate darkness, while still being able to observe them clearly as they navigated through the task, using the proposed solution. On discussing with the program Research Advisor, we realized that it's against IRB standards to blind-fold participants and simply put, we could not do it.
Work Around:
Instead of completely blindfolding the participants, we decided to spray tint a pair of ski goggles, so that they were almost opaque, but still translucent. This workaround was acceptable, and would very closely simulate darkness.
06: Evaluate
Heuristics Evaluation
While waiting for ethics board approval, we performed a heuristic evaluation with four experts so that we could evaluate the usability of our prototyped hat.
We reviewed established wearable device heuristics and realized that the published information was either overly general or too specific to a particular kind of wearable. So, we pieced together our own, focusing on the following heuristics:
Detectability
Should be simple to operate.
Distinguishability
Should be as tough as the cadet's tasks
Learnability
Should be reliable with low error
Wearability
Should be a discrete method to use
Efficiency
Should cater to different messages
Results:
Our experts found that the device wasn't uncomfortable and that you could indeed feel the vibration motors and interpret their correct directional meaning.
However, a major challenge in the heuristic evaluation was helping our experts get in the mindset of our target group. Our experts didn’t know the nuances and learned behaviors of a highly trained ROTC group; they could only imagine them.
This led to poor scores in wearability and efficiency, however the received critique was documented for further consideration when higher fidelity prototypes could be developed.
06: Evaluate
User Testing
Gaining approval for the generic hiking in the dark device, we proceeded with a first pass user-test of the proposed solution with 6 participants through an obstacle course followed with an exit-survey.
We used a within-subjects experiment, timing each participant through the course as they navigated under two different navigational cue conditions – line-of-sight hand and arm signals, and haptic signals using our prototype.
6 Participants
Navigated the course:
- Hand and arm signals
- Vibrational Cues
Exit Survey for SUS Score
From a quantitative perspective, haptic vibrations performed faster movement times than hand and arm signals. This was expected - hand and arm signals performed the poorest at mean times of over 92 secs, while vibrational cues performed at a mean times of around 74 seconds.
Overall, completion times from start to finish suggest that the Vibrational Cues resulted in more than 25% faster completion times than hand and arm signals.
However, from a qualitative perspective, the SUS Scores suggest that subjects found the vibration hat easier and more intuitive to understand than the hand and arm signals because the left and right directions were already mapped to the left and right sides of the head, eliminating the time needed to process the command.
This feedback suggested that the research is on the right track and that further revisions of the evaluation method and using the intended population could provide stronger support for moving forward with the entire communication system.
07: Ending Thoughts
Looking Forward and Lessons Learned
Our semester ended before approval to test with ROTC cadets could be gathered. We believe there is merit for moving forward, but as of now do not intend to pursue this direction.
Overall, though, this project challenged us in many aspects from paperwork and red-tape headaches to mind exercises for how to explore and design for new and rapidly changing technologies in ambiguous landscapes.
The lesson I take to heart is that evaluation is just as important as discovery. I've been an avid champion of user research for informing design (after all it's my favorite part - curiosity & talking to people). The culmination of our work though is hard to support, due to the hiccups with user testing constraints and the resulting low sample size. I don't mean to suggest that we didn't plan this part, because we did - building scenarios and steps, asking nearby classmates to be our guinea pigs, and revising the plan - but we didn't think to explore the constraints of the technology used to actually measure evaluations!
In hindsight, I'll take that less for granted and be sure to consider those angles in the future.
