Show them what they’re missing
Capturing Hololens Experiences from a Third Person POV
Browse any of the Microsoft produced promotional materials concerning Hololens, and you will see that the experiences are typically presented in third person view:
Unfortunately, this view isn’t easy to recreate for your own illustrative purposes without more than one Hololens. It appears that the official way to achieve this would be to use a camera rig that Microsoft has created which captures holograms and can be seen in the following video:
While this hardware hasn’t been made available to the public, I have a (somewhat) quick and dirty method for producing this effect for your own Hololens applications. This will allow you to document how a user interacts with your application with a fuller view of the space, and in motion.
Concept
This is presented as a follow-up to my last post: https://medium.com/truth-labs/how-to-share-your-reality-with-the-rest-of-the-world-22cea52ec07#.3hace05a1.
We will use OBS (again) to mix the Holograms over the live video feed.
The live video will once again come from your own Video Capture Device. However, this time, we will use the Hololens Emulator as the source for the holograms layer (via Window Capture).
Essentially you run the same application on both your Hololens and the Emulator, position the Emulator’s camera view to the same position as your video camera, then use the Color Key Effect Filter to remove the black from the Emulator’s view.
Set-Up
The majority of the set-up for this process is described in detail by Michael Washington at hololenshelpwebsite.com. I suggest you refer to those guides for more detailed instructions.
Unfortunately, there isn’t a global / environmental solution to this yet. Each experience that you want to capture in this way will need to be specifically configured to allow two ‘devices’ to communicate.
First, you will need to implement the Sharing Service in your application. The best guidance for this is the Holographic Academy: Holograms 240 Walkthrough.
Second, you will need to ensure that your Hololens and Emulator are capable of communicating with each other on the same local network, and with the Sharing Service. Again, Mr. Washington covers this here.
tip: When deploying to your Emulator, Windows / Hyper-V frequently changes your network adapter. I had to reassign the Emulator External Switch almost every time I launched the Emulator.
Third, to ensure that your Emulator has the same understanding of the physical space, you will need to provide it with the same room as your actual Hololens device. The process to do this can be found here.
tip: If you position your Hololens in the approximate location that your video source will be filming from, and then capture the room, it will be much easier to reposition it in the future, as it uses that location as the reference point.
tip: think ahead of where your camera will be positioned. You may not want to capture certain walls, ceilings, or objects which may occlude your camera. Think of it as building a Box Set.
#MixedReality
At this point you now have a holographic experience running on both your Hololens and Emulator, and they are able to communicate with each other and the Sharing Service with the same general understanding of the physical space that the Hololens will be used in.
Set the scene
Now you are ready to position your camera. Refer to my previous post on how to set up the camera in OBS. Here are some tips on how to frame your scene:
- You will want to frame your scene with a consideration of how the user and the Holograms will be captured, including the user entering and existing the space and potentially ‘deploying’ the hologram. I prefer wide angles.
- In this scenario, the Holograms are composited on top of the video feed. This means the user can walk behind the Holograms, but not in front of them.
- You may also want to consider how the room is lit, and if it matches the lighting of your 3D scene.
Mix it up
Again, going by my previous guide, you will have your scene set-up with your live video sources (including a first-person view of your Hololens, if desired). To blend your Hololens Emulator (your holograms) into the scene, follow these steps:
First, ensure a hologram is visible to the Emulator (i.e. it’s not just the cursor on a black background).
Next, in OBS, add a Window Capture source, and choose your Emulator window. For the Hololens Emulator, you want to make sure you choose the [XDE.exe]: XDE source:
Then, ensure your Window Capture source is placed on top of the Video Capture source, and is the same size as that source (Fit or Stretch to screen):
Finally, add a Color Key Filter to your Emulator Window Capture source and change your settings to:
- Key Color Type: Custom Color
- Key Color: #00000000
- Similarity: 1
- Smoothness: 200–500
You will want to fiddle with your Similarity and Smoothness settings to get an acceptable result.
The worst part
Now that you have positioned your camera and have the Emulator feed blending on top of it correctly, you will need to match the Emulator’s view to the camera’s position and angle. This is a tedious and frustrating task.
The only way to do this is moving it via trial-and-error. Here are the best tips I can provide to make it easier:
- Use an Xbox controller for more fine control of the view.
- Keep one window open with the Emulator’s 3D view to have an understanding of it’s physical location. Have another window open with the composited view to compare the results.
- Use markers in your physical space to match objects in the virtual space. For example, in our space, I was able to use the grid pattern on the floor, and the position of the bookshelf to center the main feature of the Holograms 240 app:
Future Considerations
Ideally, this is a temporary solution and Microsoft will release a comprehensive and integrated method for achieving this effect with much less effort. With the news that Windows Holographic will be available on many more devices, that time may be coming sooner than expected.
A quick stopgap measure would be to allow the Holograms feed to be broadcast by the Emulator (with Alpha Channel?) via the device portal, in the same way as the MRC feed is.
For more technical exploration, I recommend reading these links:
Locatable camera: https://developer.microsoft.com/en-us/windows/holographic/locatable_camera
Locatable camera in Unity: https://developer.microsoft.com/en-us/windows/holographic/locatable_camera_in_unity
Mixed reality capture for developers: https://developer.microsoft.com/en-us/windows/holographic/mixed_reality_capture_for_developers
Working with accessories: https://developer.microsoft.com/en-us/windows/holographic/Working_with_accessories
