"Sonar"

Sonar

This “Sonar” functionality allows users to test distances to the nearest objects. This helps users determine their distance from important objects. This function mimics the use of reverberation to calculate the proximity of objects. In the virtual world, we can convey this information even when the object is very far away.

Use case:

User is navigating room, attempting to follow wall. The user loses track of wall on right-hand side. The user continues navigating but becomes lost. The user suspects that they are moving towards the North wall, which is where the exit is. They query around themselves but no objects are within range. The user tries to find the wall by sending a “sonar” signal forward. After a short delay, they hear a “knock” coming back from the front. From experience, they associate this “knock” with an interior wall. The user estimates their distance from the wall based on the length of the delay. Comfortable that they have made progress towards the north wall, the user continues until they reach the wall, takes a right turn, and follows the wall to the exit.

Calculating Delay

Calculating the sound delay is a simple matter of determining the distance between the user and the object and accounting for the speed of sound. Where a and b are 3D points representing user and object positions:


        double dist = Math.sqrt( Math.pow((a.x - b.x ), 2) + Math.pow((a.y - b.y), 2) +                    Math.pow((a.z - b.z), 2) );;

        double delayAsSeconds = dist/speedOfSound * 2;
        double delayAsMilliseconds = delayAsSeconds * 1000;


The play thread is then qued with the appropriate delay and sound. The sound is always the one associated with the relevant object.

UI for Android: Touchscreen

I have createda gesture listener class to handle these.

You have to manually set what constitutes a "swipe"-- how far the user has to move their finger (both min and max). You have to set the speed at which they move their finger, so as not to conflate with a "scroll":
  private int swipe_Min_Distance = 100;
    private int swipe_Max_Distance = 800;
    private int swipe_Min_Velocity = 40;
Then you catch the probable swipe as a "fling." You make sure it's a swipe and determine its direction--manually setting the leeway.
@Override
    public boolean onFling(
        MotionEvent e1,
        MotionEvent e2,
        float velocityX,
        float velocityY)
    {
        final float xDistance = Math.abs(e1.getX() - e2.getX());
        final float yDistance = Math.abs(e1.getY() - e2.getY());

        if(xDistance > this.swipe_Max_Distance || yDistance > this.swipe_Max_Distance)
         return false;

        velocityX = Math.abs(velocityX);
        velocityY = Math.abs(velocityY);
              boolean result = false;

        if(velocityX > this.swipe_Min_Velocity && xDistance > this.swipe_Min_Distance){
         if(e1.getX() > e2.getX()) // right to left
          this.onSwipe(Globals.SWIPE_LEFT);
         else
          this.onSwipe(Globals.SWIPE_RIGHT);
         result = true;
        }
        else if(velocityY > this.swipe_Min_Velocity && yDistance > this.swipe_Min_Distance){
         if(e1.getY() > e2.getY()) // bottom to up
          this.onSwipe(Globals.SWIPE_UP);
         else
          this.onSwipe(Globals.SWIPE_DOWN);
         result = true;
        }
         return result;
    }
Have to differentiate between single taps, long presses and double taps:

 @Override
    public boolean onSingleTapConfirmed(MotionEvent e)
    {        float scaledX, scaledY;
        gs.touchCell(scaleX(e.getX()), scaleY(e.getY()), false);
        return true;
    }

 @Override
    public boolean onDoubleTapEvent(MotionEvent e)
    {
        gs.startListening();
        return true;
    }

IRB stuff

Purpose of the research study: The biggest barrier to the independence of the visually impaired is often the problem of mobility. Mobility training is difficult and intimidating.

The project uses publicly available 3D models of buildings, towns and cities to automatically generate audio-based “maps” for the blind. The visually impaired can then use these “maps” to explore environments virtually. These “maps” use 3D audio and text-to-speech to convey information to the user.  With this software, users become accustomed to the layout of the environment and memorize routes they can use in real-world navigation. While this technology is most beneficial to the visually impaired, this study is also open to sighted individuals.
These maps can offer varying level of detail; the purpose of this study is to discover whether or not more detailed audio information leads to better recall of layouts and routing.

What you will do in the study: Using an Android tablet and headphones, you will explore a 3D audio environment for a limited amount of time. To encourage focus, you will be blindfolded while exploring. After a tutorial, you will be expected to explore independently. When the time limit has passed, you will be asked a series of questions about the environment you explored.

Time required: One to two hours continuously.

Design: Functional Modules

Functional Modules

Explorer

Knows about: Player, input manager, move manager, query manager

→ Receives sanitized, platform-agnostic input from input manager

→ Makes appropriate calls to query/move manager

Movement Manager

Knows about: Player, Room

→ Collisions/Bounds

→ Movement w/ wall-following (auto-correction)

→ Turning

→ Forward movement

→ Makes calls to produce footsteps & collision noises

→ Handles exceptional movement (e.g. steps, elevators, room-to-room via doors)

QueryManager

Knows about: Player, room

→Makes call for Sonar

→ Makes calls for Reverb

→ “nearby” queries (left/right/front) (also as TTS)

→ Semantic info (TTS)

→ Later: makes calls to routing

SoundManager

Knows about: OpenAL

• OpenAL info (Listener position)

→ Sets and updates OpenAL info

→ Produces 3D sound

→ Produces “Sonar”

→ Produces Reverb

Design: Object Classes

Object Classes

Player (Global Singleton)

• Room
• Position
• Orientation (e.g., 90 degrees, relative to North)
• Preferences (height, stride length)

Room

• Entrances/Exits (and associated “starting positions”)
• Polygon representing room bounds
• Walls
• Floor (material)
• Dimensions
• Stray Objects (furniture, etc.)
• Room name
• Purpose (semantic info)
• Parent building

→ Given collision coordinates, returns object collided w/

→ Given step/query coordinates, returns object queried

Building is a Room

• Entrance/exits
• Exterior walls
• Rooms
• Name
• Type/purpose (semantic info)
• Environment (exterior)
• Lat/Long

Wall - - Mimic tree structure of gml

• Windows
• Doors
• Angle (relative to North)
• Adjacent rooms (N/S or E/W)
• Material (wood, brick, etc.)
• Color

→ Produce angle

Door

• Material
• Type (sliding door, automatic door, etc.)
• Adjacent rooms

StrayObject (furniture, computer terminals, etc.)

• Description
• Position

Custom Engine

I will be rolling my own custom engine.

Needs
* Search, routing
* Free-form navigation
* 3D audio output
* 2D mapping output
* GPS recognition

Implementation
* Java
* CityGML4J
* Android Platform
* Possibly C++
* MongoDB

Challenges
* Good 3D audio with effects (requires C++)
* Fast routing
* Variety of searches and ways to handle data
* Reasonably handling huge GML files
* Splitting up landscape
* Interface: supplying info to user (semantic and physical) purely through audio.
* Interface: Touch-screen that is blind friendly
* (These interface problems are partly addressed by my prototype)

User Study Continued

Optimal User Study
* Large volume of visually impaired persons
* App automatically converts model based on real-world version
* One group uses app to practice new space; control group does not
* Both groups navigate new space: Previously explained & surprise routes
* Study intends to show that automated conversion produces a map which does assist in real-world navigation.

Compromises:
* Sighted individuals
* Questionaire instead of real-world navigation
* Model is based on hypothetical building

User Study: Permission to use Human Subjects

Instructions are here:

https://www.wm.edu/offices/sponsoredprograms/preaward/policies/compliancereqs/humansubjects/instructions/index.php
"
A properly completed protocol will include a brief rationale for the study, full procedures,  description of the participants,  copy of all tests, questionnaires, all interview questions, the informed consent form, and other pertinent information.
Please allow sufficient time (3-4 weeks) for review
"

How does CityGML handle it?

(see posted CityGML tutorial). 

* Stores geometric and semantic info.

* xml-based

 * Geometric "bounding boxes"

 * Has a tree structure 

    But we are concerned not only with membership (a window is a member of a wall, a pc terminal is a member of a room) but also with proximity! We might have a closer, more desirable match in a different room, within a different sub-tree.

Model Obligations

Model Obligations / Functions

* user can "bookmark" spaces; important spaces are labelled--two-way searches (by location, by place-name) (e.g. "nearest rooms" or "(nearest) bathroom."

* simple, step-by-step traversal (user navigation; primary function) "Common case" (make common case fast)

* Model needs to understand function of elevators, stairs and doors.

Is speed important?

Not right now, but:

* Large scale representations--campus, town or even huge building -size -- could cause problems. We can't have n^2 or n^3 searches. Searches for unhashed/untabled spaces could be disastrous. So hash everything? could look up all "classrooms" in a table, find the nearest one.

* Also, CityGML files can get really big – there’s a lot of info here.

* Need reasonable "nearest" searches. Distance may be so great that non-verbal aural feedback is useless; we still want to provide information like "the closest comuter terminal is one floor up, 100 yards south"

* Routing -- we will often want to be able to quickly provide a route for these places. Long-distance routing, too.

Free v. Gridded Movement

    The prototype, which is essentially a 2D floorplan (although it does use standard heights to generate trully 3D sound), operates as a grid. The floorplan is split up into step-size squares that form a grid. Is this appropriate for a final product—could we have step-size “cubes”? Should we have finer detail? This also connects to the question of how the user should move through the space. There are pros and cons to both free-movement and gridded movement.

Free Movement

• Pro: More like real life. More fluid. More user freedom.
• Con:  More difficult to represent & implement programmatically. More difficult for user to track routing & orientation. Lose some “step-counting” functionality. Steeper learning curve for user.

3D Space

How to represent 3D space?

What data structure(s) ?

Need both metadata (semantic info) & 3D geometry (this is why CityGML is a good fit).

 (Why do we need 3D space?)

* need to provide spatial information via audio cues (how big a space is, reflected sounds, echos) -- requires coarse but variable height

* need to represent multiple stories/floors -- doesn't require variable height; floors could be separate entities.

    For the coming user study, we don't need "true 3D," because we may not emulate advanced aural feedback--it is likely that our study will be with sighted persons, who probably can't take advantage of this. We will use OpenAL to generate basic 3D sound, neglecting echos, etc. This is to save time.

However, we have to parse the CityGML anyway, and we might as well store all the information; if we don't use it now, we'll want it later.

    So how do we concisely and simply represent our 3D space? Most relevant studies are concerned with 2D floorplans.

CityGML Tutorial and Example

http://www.citygml.org/fileadmin/citygml/docs/CityGML_Tutorial_Kolbe_Internet.pdf


citygml4j   Java interface

Relevant Paper Abstracts

Using Virtual Environment to Improve Spatial Perception
 by People Who Are Blind
ORLY LAHAV, Ph.D.

ABSTRACT
Mental mapping of spaces, and of the possible paths for navigating these spaces, is essential
for the development of efficient orientation and mobility skills. Visual ability is a crucial
component to effective mental mapping. People who are blind consequently find it difficult
to generate useful mental maps of spaces and navigate competently within them. The research studies reported in this paper assume that the supply of appropriate spatial information through compensatory sensorial channels as an alternative to the visual channel may
contribute to the anticipatory mental mapping of unknown spaces and consequently may improve spatial performance for people who are blind

Usability of a Multimodal Video Game to Improve Navigation Skills
for Blind Children
JAIME S´ ANCHEZ and MAURICIO SAENZ

This work presents an evaluative study on the usability of a haptic device together with a soundbased video game for the development and use of orientation and mobility (O&M) skills in closed,
unfamiliar spaces by blind, school-aged children. A usability evaluation was implemented for
a haptic device especially designed for this study (Digital Clock Carpet) and a 3D video game
(MOVA3D) in order to determine the degree to which the user accepted the device, and the level
of the user’s satisfaction regarding her interaction with these products for O&M purposes. In
addition, a cognitive evaluation was administered. The results show that both the haptic device
and the video game are usable, accepted and considered to be pleasant for use by blind children.
The results also show that they are ready to be used for cognitive learning purposes. Results
from a cognitive study demonstrated significant gains in tempo-spatial orientation skills of blind
children when navigating in unfamiliar spaces.



BATS: The Blind Audio Tactile Mapping System

ABSTRACT
The BATS project focuses on helping students with visual
impairments access and explore spatial information using
standard computer hardware and open source software. Our work
is largely based on prior techniques used in presenting maps to the
blind such as text-to-speech synthesis, auditory icons, and tactile
feedback. We add spatial sound to position auditory icons and
speech callouts in three dimensions, and use consumer-grade
haptic feedback devices to provide additional map information
through tactile vibrations and textures. Two prototypes have been
developed for use in educational settings and have undergone
minimal assessment. A system for public release and plans for
more rigorous evaluation are in development.

Use Case (long-term prototype)

Use-Case (Long-term goal)

A blind student has registered for courses at a campus or center for the blind. Before setting foot on campus, they use their text-to-speech enabled device (adroid/iOS phone) to navigate to the app. Using a search function (they could use GPS location if they were on-site), they select the relevant institution from the list (e.g., College of William and Mary). Then, they might choose to practice navigating over the entire campus from a chosen starting point; or, they can select an individual building they want to practice in. A brief speaking description explains how the game works. The user then navigates through the environment using audio cues. She walks along until she is stopped by a wall. She does an action towards this wall, and the app informs her it is the east wall of the welcome center. The user follows the east wall until she hears a knocking sound, which indicates a door on her left. She taps her phone to perform a localized action. The speaker informs her that this is the door to the kitchen, but that it requires a key to be opened. The user continues to follow the east wall. She picks up one of several “jewels” required to complete the level. She is notified of another door. This time her localized informs her that this door leads to classroom A, and allows her to open the door. She enters and explores the room until she finds the key for the kitchen.  The user then navigates back to the beginning of the level. Now, her localized action opens the door. She continues until she finds every jewel and completes the level.

Surpassing the State of the Art

Assistive Technology has made great strides in recent years. Maps for the visually impaired has been a research subject for more than ten years; however, there is a huge gap between the exciting research being done and deliverables.

Need
*  More than eight million Americans over the age of fifteen have difficulty seeing (http://www.newyorker.com/online/blogs/elements/2013/12/where-are-the-games-for-disabled-players.html)
* working age adults with significant vision loss: ~37% were employed in 2011
* Navigation of new spaces is a serious limit on independence


Research
* Awesome video games for the blind:
* 3D audio tech
* Audio maps (large-scale)
* Covers city-scale and interior
* chips in floor tiles and on walls (impractical)
* Increasing amount of public 3D models of real-world places: http://www.3d-berlin.com/

Products / Deliverables
* Expensive GPS equipment (GPS canes)
* Mobile audio map apps (coarse, less accurate)
* Covers city-scale only

While mobile audio maps provide a great service for street-level navigation, the accuracy limits of GPS limit its usefulness--and these apps provide no help in navigating interiors. Studies have produced audio maps/games which are proven to help blind users navigate. However, these projects are limited to the studies themselves--only one individual building. The purpose of this project is to move towards providing audio maps of public and privately owned institutions. Visually impaired users could use these maps to explore buildings and cities, memorizing layouts and routes, and making navigation in these spaces easier and increasing independence.

Batch, 2D, User Study

Batch Processing -- While the model used in the user study will be hand-crafted, and output convenient CityGML, we want to be able to handle a wide variety of 3D models--the kind that are published on the web that might not be so pretty, or which might have certain idiosyncratic attributes. At some point, I'd like my converter to handle large-scale CityGML (towns, cities, etc.) rather than just individual buildings. This effort will require gathering data, testing, adjusting, testing, etc.
2D -> 3D -- Take common-format 2D floorplans/blueprints and convert them to 3D models. The 3D models can then be converted into an audio map.
User Study -- Study will be conducted at the College of William and Mary using blindfolded sighted students. The students will attempt to navigate a real space blindfolded after having used our audio map.

Month-by-month rundown

February	Parallel
Create 3D model Implement 3D->Audio conversion
	Design User Study User Study logistics (ethics board, etc.)

March	Parallel
Conduct User Study Start paper Batch Processing (building-scale)
	Audio model improvements 2D floorplan -> 3D model (time permitting)

April	Parallel
Finish Paper Batch Processing (city-scale)
	Deploy to server (conversion logic) Interface improvements