This sprint was to take another run at upgrading the Avatar System to utilize the latest System Development Kit from Ready Player Me (RPM). We had tried to do this a few months back and were not able to. We are happy to report that this time we succeeded with the help of the RPM Unreal Engine SDK developer. A few sprints back we abandoned using the Poser/Daz derived Avatars and decided to go with RPM Avatars going forward. This will allow us to have what is quickly becoming an industry standard Avatar in the Music Oasis Game.
The new Avatar SDK from Ready Player Me added a number of features that are now in myMO.
The first is that now when you load a new RPM avatar into myMO to use, it will cache it locally on your computer. This means that when you enter the game the next time your Avatar load will be much faster than it has been. It not only caches your avatar from RPM, but, also any other Avatars that have been in any myMO session or event that you have attended. This makes their loads into the game much faster too and improves the startup of the event you may be hosting or attending load that much faster.
The second major features that were implemented had to do with integrating the Oculus Lip Sync as a plug in and the writing some code to use the Oculus Lip Sync program which basically listens to microphone audio and outputs a set of probabilities of which visemes it thinks that it recognized. From this probability matrix you can select the ones that go over a threshold you set and then have a pretty good idea of what was just spoken by an player in the game. We wrote code that then used these detected visemes to drive RPM Avatar face morphs (called blend spaces) that moved the mouth which resulted in the Avatars lips synching up with voice being captured. This is also replicated to all the clients in the game session so that other players see your avatars mouth/lips move appropriately when you talk in the game using voice chat. This lip sync software from Oculus works in Desktop, so their is not need to wear VR goggles to see mouth movement.
The third new feature we added was the addition of the Oculus Audio Library into the game which is in the same SDK as the Oculus Lip Sync. This plugin allowed us to spatialize sound in the game. We were able to get it working for sounds in the game like music, footsteps, doors or any other audio added to a scene. We are still working on being able to attach it to our voice module so that people speaking will also be spatialized. It is a big step forward, but we still have some work to do in a future sprint.
The fourth item from Ready Player Me was we were able to capture a render of our RPM avatar from the RPM hub. We may incorporate this as an image in the future attached to the Avatar Selection Menu to have a reference to the Avatar entry in the menu as an image on the button for that avatar. At this point, we just know that we have image available, but, have not incorporated into the menu or saved it to a database or local cache. Again, something to work on in a future sprint when we look at Avatar upgrading.
The fifth item was reworking the load of any custom avatar you make in the RPM website hub into the game. We became a partner site with Ready Player Me and now have our own subdomain that has the RPM Avatars that you might be utilizing in our game. Our subdomain is https://musicoasis.readyplayer.me/ and here is a video showing how easy it is now to create a new RPM Avatar and get it into myMO. Once you have loaded it stays in the game locally and you no longer need to go to the RPM hub unless you are creating a new avatar or changing their outfits. Here is a video showing how to create a new avatar using RPM hub and the import feature in myMO:
We also explored the possibility of embedding the RPM web based hub within our game. They only have it working for UE 5 and our game is still UE 4.27 so we were not able to create it using their example. They do not support trying to do this unless you are in UE5, but, we have an upgraded web browser plug-in we bought that supports an updated Chromium Embedded Framework (CEF) that may be sufficient for us to try doing this in the UE 4.27 engine we are using in myMO. Again this sprint we got it working in a demo project in UE 5, but didn’t try to use the same type of logic in that demo project and reworking it to work in UE 4.27 of our game. We will first make a decision about when or if we are going to do a near term upgrade to UE 5 or should spend time trying to make it work in UE 4.27. We should make that decision some time this month.
We also began doing some more documentation on our Wiki site. We documented how to get access to the game for Alpha Users as we are starting to open the game up for Alpha testing this month too. The next couple of sprints will have more emphasis on testing and documentation as other members of our overall Music Oasis community start getting access and playing with myMO. We plan on having some of our own meetings and events in the game as part of our Alpha Testing.
