Goal

Capture photographs in three dimensional space and transfer photo collaging workflow between the immersive MR environment and computer.

Keyword features

Bring photography into three dimensional space.

Face your photograph heads on, as the frame is positioned closer to your viewpoint and your body, rather than being cropped into a fixed 2 dimensional screen.

Arrange the photo textures in three dimensional.

Use cases

1. Experimental immersive photography

2. Experimental immersive collaging

3. Quick sampling from real world to be used as texture in artistic collage or productivity workflow


Technology

1. clientState information on button, finger, pinch, point.

1. Using shared state to communicate between computer client and VR client:

- sending VR camera feed to computer

- sending cropped camera feed to VR environment

- sending positions of texture within the computer canvas marker in the VR environment to computer canvas

- sending positions of texture within the canvas in computer to the computer canvas area in VR environment

1. getDisplayMedia to access VR camera feed
2. preprocess the camera feed upon capture trigger, crop out image inside the quad border that is currently hovered on.
3. use window.open to render computer canvas in a new window
4. Transfer logic between textures in immersive to and from computer canvas using Canvas API

User Experience

User Interface

The following instructional UI will be assigned to controller/hand. The modes are detected based on window.handtracking.

Controller. All of them is hardcoded to right controller as of now.

- Select button -> create point

- Grab button -> move point or move surface (depending on where the controller is hovering at)

- Button A -> Save the VR camera view

- Button B -> toggle between activate calibration mode

Hand

- Pinch -> toggle between activating calibration mode & confirming the calibrated canvas scale.

- Pinch & hold for x seconds -> confirm calibration if on calibration mode, move point or surface if hovering over point or surface.

- Point & hold -> create point

Major issues

1. Texture are still rotated randomly on immersive, due to the point location that may not lie in the same 2 dimensional axis. Probably can be solved by forcing them into the nearest flat axis but the reason i'm not doing that is because I want user to have the flexibility to create point in non flat surface. It is not quiet realistic to always have flat surface or be able to view a surface through a camera viewfinder in a flat way.

2. Cannot move the texture on the computer partner canvas.

Improvements

1. Tutorial and UI/UX polish

2. More audio feedback on actions for hand mode

3. Allowing for accessibility to use left hand, give user option to user to switch dominant controller or hand.