Difference between revisions of "Unreal Engine: OptiTrack Live Link Plugin"

(Minor edits to title names)
 
(37 intermediate revisions by 2 users not shown)
Line 1: Line 1:
{{Warning|'''Note:''' Contents in this page are still preliminary and the instructions are subject to be changed}}
+
[[Category:Plugin]]
 +
[[Main Page]] → [[OptiTrack Unreal Engine 4 Plugin]] → {{PAGENAME}}
 +
----
  
 
'''Overview'''
 
'''Overview'''
:This page provides instructions on how to use OptiTrack UE LiveLink plugin.
+
:This page provides instructions on how to use the OptiTrack Unreal Engin Live Link plugin. The plugin communicates with Unreal's built-in Live Link system by providing a Live Link source for receiving tracking data streamed from Motive. This plugin can be used for controlling cameras and objects in virtual production applications. When needed, the [[OptiTrack Unreal Engine 4 Plugin]] can also be alongside this plugin.
  
=Plugin Setup=
+
 
 +
=Setup=
 
----
 
----
 
<div class="padded">
 
<div class="padded">
 
====1. [Motive] Setup rigid body streaming in Motive.====
 
====1. [Motive] Setup rigid body streaming in Motive.====
 
<div class="padded">
 
<div class="padded">
Get Motive streaming with at least one rigid body. Make sure the [[Data Streaming|Streaming]] settings are set up correctly, and the rigid body is active under the [[Assets pane]].
+
Get Motive streaming with at least one rigid body. Make sure the [[Data Streaming|Streaming]] settings are configured correctly, and the rigid body is active under the [[Assets pane]].
 
</div>
 
</div>
  
 
====2. [UE] Install the OptiTrack plugins in Unreal Engine (UE).====
 
====2. [UE] Install the OptiTrack plugins in Unreal Engine (UE).====
 
<div class="padded">
 
<div class="padded">
You can install the OptiTrack - NatNet Streaming Client plugin by putting the plugin files into one of the following directories:
+
You can install the OptiTrack Unreal Engine plugin by putting the plugin files into one of the following directories:
 
*A global engine plugin can be placed in <code>C:\Program Files\Epic Games\[Engine Version]\Engine\Plugins</code>
 
*A global engine plugin can be placed in <code>C:\Program Files\Epic Games\[Engine Version]\Engine\Plugins</code>
 
*A project-specific plugin can be placed in <code>[Project Directory]\Plugins</code>
 
*A project-specific plugin can be placed in <code>[Project Directory]\Plugins</code>
Line 21: Line 24:
 
====3. [UE] Enable the plugins in UE project.====
 
====3. [UE] Enable the plugins in UE project.====
 
<div class="padded">
 
<div class="padded">
Go to ''Edit &rarr; Plugins'' and enable the OptiTrack - LiveLink plugin as well as the Built-In LiveLink plugin in UE.
+
Go to ''Edit &rarr; Plugins'' and enable two of the required plugins. First one is the '''OptiTrack - Live Link''' plugin under ''Installed'' group, and the second one is the built-in '''Live Link''' plugin under Built-In group.
  
 
[[Image:UELiveLink_EnableOptiTrackPlugin.png|center|thumb|650px|OptiTrack -  Live Link plugin. Make sure the plugins are placed in the plugins folder either in the project or engine directory.]]
 
[[Image:UELiveLink_EnableOptiTrackPlugin.png|center|thumb|650px|OptiTrack -  Live Link plugin. Make sure the plugins are placed in the plugins folder either in the project or engine directory.]]
Line 29: Line 32:
 
====4. [UE] Open the LiveLink pane====
 
====4. [UE] Open the LiveLink pane====
 
<div class="padded">
 
<div class="padded">
The LiveLink pane can be accessed from ''Window &rarr; Live Link'' from the toolbar in UE.
+
Open the LiveLink pane from ''Window &rarr; Live Link'' in the toolbar.
 
<center><ul>
 
<center><ul>
 
<div class="thumblist">[[Image:UELiveLink_LiveLinkAcess.png|center|thumb|274px|LiveLine pane in UE.]]</div>
 
<div class="thumblist">[[Image:UELiveLink_LiveLinkAcess.png|center|thumb|274px|LiveLine pane in UE.]]</div>
Line 45: Line 48:
 
====6. [UE] Check the Connection.====
 
====6. [UE] Check the Connection.====
 
<div class="padded">
 
<div class="padded">
If your settings were correct, then the plugin will list out all of the detected rigid bodies, and they should have green dots next to them signifying that the asset has been created and is receiving data. If the dots are yellow, then either your take is paused, or there is some sort of connection error.
+
If the streaming settings are correct and the connection to Motive server is successful, the plugin will list out all of the detected rigid bodies, and they should have green dots next to them indicating that the corresponding rigid body has been created and is receiving data. If the dots are yellow, it means that the client has stopped receiving data. Check if Motive is still tracking or if there is a connection error.
  
 
[[Image:UELiveLink_CheckConnection.png|center|thumb|550px|Connected to the rigid body data stream.]]
 
[[Image:UELiveLink_CheckConnection.png|center|thumb|550px|Connected to the rigid body data stream.]]
Line 53: Line 56:
 
=Using the Plugin=
 
=Using the Plugin=
 
----
 
----
==Live Link Rigid Body Tracking==
+
==Static Meshes or Camera Actors==
 
<div class="padded">
 
<div class="padded">
After you are connected to Motive through the Live Link plugin and have at least one rigid body tracking, then you can start animating an object in Unreal Engine as well.  
+
====1. Add the camera object or static mesh object that you wish to move====
 +
<div class="padded">
 +
Add a camera actor from the Place Actors pane or a static mesh from the project into your scene. For the static meshes, make sure their Mobility setting is set to ''Movable'' under the Transform properties.
 +
 
 +
<center><ul>
 +
<div class="thumblist">[[Image:LiveLink_Camera_CameraObject.png|thumb|300px|Camera actor listed under the Place Actors pane.]]</div>
 +
<div class="thumblist">[[Image:LiveLink_Camera_CubeObject.png|thumb|300px|Actor mobility set to movable.]]</div>
 +
</ul></center>
 +
</div>
  
====1. Set up the object to track====
+
====2. Add a LiveLinkController Component====
 
<div class="padded">
 
<div class="padded">
The first step is to set up your object to track. To do this, right-click on a skeleton mesh object you would like to track in the content browser, choose &rarr; Create &rarr; Anim Blueprint''. ''The mesh must be a skeleton mesh''.
+
Select an actor you want to animate, click the “Add Component” button on the details pane, and add ''Live Link Controller''.
  
[[Image:LiveLink_CreatingAnimBlueprint.png|center|thumb|550px|Creating Anim blueprint.]]
+
[[Image:LiveLink_Camera_AddLiveLinkController.png|thumb|center|300px|Live Link Controller component needs be added.]]
 
</div>
 
</div>
  
====2. Open Animation Blueprint====
+
====3. Select the target rigid body====
 
<div class="padded">
 
<div class="padded">
This will create an Animation Blueprint of the selected mesh. In the content browser, double-click the created animation blueprint to edit it.  
+
Under the Live Link Controller component properties &rarr; Subject Representation, select the target rigid body.
  
[[Image:LiveLink_AnimBlueprint..png|center|thumb|300px|Animation blueprint create in the content browser.]]
+
[[Image:LiveLink_Camera_ConfigLiveLinkController.png|thumb|center|300px|The target rigid body selected under Live Link Controller component properties.]]
 
</div>
 
</div>
  
====3. Add Live Link Pose to the blueprint ====
+
====4. Check====
 
<div class="padded">
 
<div class="padded">
In the blueprint, access the ''AnimGraph'' tab. There should be an output pose node in there already. Then right-click in the empty space and search for "Live Link" to find the "Live Link Pose" and add it to the graph.
+
Once the target rigid body is selected, each object with the Live Link Controller component attached and configured will be animated in the scene.
 +
</div>
 +
</div>
  
[[Image:LiveLink_AnimGraph1.png|center|thumb|550px|Adding Live Link Pose node.]]
+
==Timecode Setup==
</div>
+
When the camera system is synchronized to another master sync device and a timecode signal is feeding into [[External_Device_Sync_Guide:_eSync|eSync]], then the received timecode can be used in UE project through the plugin.
  
====4. Connect Live Link pose to the Output Pose====
 
 
<div class="padded">
 
<div class="padded">
On the Live Link Pose node, you should see the Live Link Subject Name property. Use the dropdown and select the name of a rigid body to use.  Then, drag the pose output and connect it to the Output Pose.
+
====1. Set Timecode Provider under project settings====
 +
<div class="padded">
 +
From ''Edit &rarr; Project Settings'', search timecode and under Engine - General settings, you should find settings for the timecode. Here, set the the Timecode Provider to LiveLinkTimeCodeProvider.
  
[[Image:LiveLink_ConnectOuputPose.png|center|thumb|550px|Live Link Pose of a Bat rigid body connected to the output pose.]]
+
[[Image:LiveLink_Timecode1.png|thumb|center|650px|Select the timecode provider under project settings.]]
 
</div>
 
</div>
  
====5. Set the retarget asset====
+
====2. Set OptiTrack source in the Live Link pane as the Timecode Provider====
 
<div class="padded">
 
<div class="padded">
Select the “Live Link Pose” node and select the “OptiTrackLiveLinkRetarget” option in the “Retarget Asset” property in the Details pane.  
+
Open the Live Link pane, and select the OptiTrack subject that we created when first setting up the plugin connection. Then, under its properties, check the Timecode Provider box.
  
[[Image:LiveLink_RetargetAsset.png|center|thumb|550px|Selecting Retarget Asset.]]
+
[[Image:LiveLink_Timecode2.png|thumb|center|650px|Enable timecode provider setting in the plugin.]]
 
</div>
 
</div>
  
====6. Compile and save ====
+
====3. Check====
 
<div class="padded">
 
<div class="padded">
Compile and save the blueprint. Once the changes have been compiled, you should see the object animating according to the rigid body tracking in the mini 3D viewport.  
+
The timecode from Motive should now be seen in the Take Recorder pane. Take Recorder pane can be found under Window &rarr; Cinematic &rarr; Take Recorder in the toolbar.
  
[[Image:LiveLink_RB_Compiled.png|center|thumb|750px|Compiled blueprint with the mesh being animated.]]
+
[[Image:LiveLink_Timecode_7.png|thumb|center|550px|Timecode shown in the Take Recorder pane.]]
 
</div>
 
</div>
 
</div>
 
</div>
  
==Live Link Camera Plugin==
+
==Skeletons==
 
<div class="padded">
 
<div class="padded">
====1. Configure the plugin under project settings====
+
====1. Create a new Animation Blueprint====
<div class="padded">
+
<div class="padded">  
Open the project settings and in the LiveLink section, for the Default Role Settings, set the Role to LiveLinkAnimationRole, and set the Setting Class to OptiTrackSubjectSettings.
+
Right click the mesh you would like to use and select "Create > Anim Blueprint"
 +
 
 +
<center><ul>
 +
<div class="thumblist">[[Image:LLSkeleton_New_Anim_Blueprint.png|thumb|400px|New animation blueprint.]]</div>
 +
</ul></center>
  
[[Image:LiveLink_Camera_ProjectSettings.png|center|thumb|650px|Live Link project settings]]
 
 
</div>
 
</div>
  
====2. Connect====
+
====2. Name and Open the Animation Blueprint====
 
<div class="padded">
 
<div class="padded">
Connect to Motive via LiveLink, each rigid body asset should already have their own translator:
+
Name the animation blueprint something reasonable, then double click it to open the blueprint.
  
[[Image:LiveLink_Camera_Translator.png|center|thumb|550px|]]
+
[[Image:LLSkeleton_Rename.png|thumb|center|600px|Rename blueprint.]]
 
</div>
 
</div>
  
====3. Add a camera object====
+
====3. Hook up your Blueprint====
<div class="padded">  
+
<div class="padded">
Drag a camera or other actor object into your scene from the Place Actors pane.  
+
Create a "Live Link Pose" component and connect it to the "Output Pose". Assign the "Live Link Subject Name" to the skeleton that you would like to use.
 +
 
 +
[[Image:LLSkeleton_Blueprint.png|thumb|center|600px|Connect to pose, then choose name.]]
  
[[Image:LiveLink_Camera_CameraObject.png|center|thumb|250px]]
+
Change the "Retarget Asset" property in the Details pane of the blueprint editor to "OptiTrackLiveLinkRetarget"
</div>
 
  
====4. Add LiveLinkController to the camera====
+
[[Image:LLSkeleton_Retarget.png|thumb|center|400px|Retargeting option.]]
<div class="padded">
 
Select the object you want to animate(in this case, a cube mesh), click the “Add Component” button on the details pane, and add LiveLinkController.
 
  
[[Image:LiveLink_Camera_AddLiveLinkController.png|center|thumb|387px]]
 
 
</div>
 
</div>
  
====5. Configure the LiveLinkController component ====
+
====4. Getting the Skeleton to Animate====
 
<div class="padded">
 
<div class="padded">
Under the LiveLinkPonentController, choose the asset name for the Subject Representation property. You can expand the setting by clicking the triangle next to it and choose LiveLinkTransformRole for the Role setting.
 
  
[[Image:LiveLink_Camera_ConfigLiveLinkController.png|center|thumb|350px|]]
+
To animate the skeleton in real time click the Animation Blueprint from earlier. In the Details pane press "+ Add Component > Live Link Skeleton Animation". After you add that component the mesh should start animating.
</div>
 
  
====6. Check====
+
To animate the skeleton in a game, just press the play button. Adding the "Live Link Skeleton Animation" object is not necessary to animate in play mode.  
<div class="padded">
 
At this point the camera should be animating in the scene.  
 
</div>
 
</div>
 
  
==Live Link Timecode Setup==
+
[[Image:LLSkeleton_LiveTrackingComponent.png|thumb|center|600px|Component to animate skeleton.]]
When the camera system is synchronized to another master sync device and a timecode signal is feeding into [[External_Device_Sync_Guide:_eSync|eSync]], then the received timecode can be used in UE project through the plugin.
 
  
<div class="padded">
 
====1. Check the camera system frame rate.====
 
<div class="padded">
 
In Motive, double-check and make sure the camera frame rate is a multiple of the genlock/timecode input. If this is not the case, then you will get warning messages, since there will not be a regular one-to-one assignment of timecode frames/subframes and motion capture frames.
 
 
</div>
 
</div>
  
====2. Configure Evaluation Mode ====
+
====Debugging Note====
 
<div class="padded">
 
<div class="padded">
In the LiveLink pane, select the OptiTrack source, then set the Evaluation Mode to ''Timecode'' under the settings. Once that's done, the Timecode Frame Rate should get set automatically.
+
If the retargeting doesn't match the mesh correctly, then you can create a new OptiTrackLiveLinkRetarget blueprint from scratch and modify the bone mapping names.
 +
 
 +
[[Image:LLSkeleton_Debugging.png|thumb|center|650px|Options for retargeting to different names.]]
  
<center><ul>
 
<div class="thumblist">[[Image:LiveLink_Timecode_2.png|center|thumb|550px]]</div>
 
<div class="thumblist">[[Image:LiveLink_Timecode_1.png|center|thumb|550px|]]</div>
 
</ul></center>
 
 
</div>
 
</div>
  
====3. Create a Timecode blueprint====
 
<div class="padded">
 
Click '''Add New''' button on the Content Browser and select Blueprint Class. Then, search and select LiveLinkTimecodeProvider as the parent class to add. Name the created blueprint to something such as "OptitrackTimeCode".
 
 
<center><ul>
 
<div class="thumblist">[[Image:LiveLink_Timecode_4.png|center|thumb|550px]]</div>
 
<div class="thumblist">[[Image:LiveLink_Timecode_3.png|center|thumb|550px]]</div>
 
</ul></center>
 
 
</div>
 
</div>
  
====4. Set Subject Key====
+
=Standalone Game Mode=
<div class="padded">
+
----
Open the created blueprint and select one of the rigid body asset as the Subject Key, then compile and save it.
+
For testing the project in standalone game mode, or when developing an nDislay application, the Live Link plugin settings must be saved out and selected as the default preset to be loaded onto the project. If this is not done, the configured settings may not get applied. After configuring the LiveLink plugin settings, save out the preset from the Live Link pane first. Then, open the Project Settings and find Live Link section in the sidebar. Here, you can select the default Live Link preset to load onto the project, as shown in the screenshot below. Once the preset is properly saved and loaded, the corresponding plugin settings will be applied to the standalone game mode.
  
[[Image:LiveLink_Timecode_5.png|center|thumb|452px]]
+
If all the configuration is correct, the actors will get animated in the newly opened game window when playing the project in the standalone game mode.
</div>
 
  
====5. Set Timecode Provider under project settings====
+
<center><ul>
<div class="padded">
+
<li class="thumblist">[[Image:LiveLink_Standalone_Preset1.png|thumb|520px|Creating a preset from Live Link panel. Select '''Save As Preset''']]</li>
From ''Edit &rarr; Project Settings'', go to General Settings and for Timecode Provider enter the name of the blueprint created in step 4:
+
<li class="thumblist">[[Image:LiveLink_Standalone_Preset2.png|thumb|500px|Assigning the preset in the Project Settings.]]</li>
 +
<li class="thumblist">[[Image:LiveLink_Standalone4.png|thumb|Playing a scene in standalone game mode.]]</li>
 +
</ul></center>
  
[[Image:LiveLink_Timecode_6.png|center|550px|]]
+
=Troubleshooting=
</div>
+
----
 
 
====6. Check the timecode in the Take Recorder pane====
 
 
<div class="padded">
 
<div class="padded">
Check that the correct timecode is now seen in the Take Recorder pane.  
+
<div class="mw-collapsible mw-collapsed" style="background-color:#f9ffe5; border-style: dotted; border-width: 1px; padding: 1em">
 +
'''Q - Trying to add more than 64 frames in the same frame. Oldest frames will be discarded.'''
 +
<div class="mw-collapsible-content padded">
 +
A - This notification message may appear at the bottom of the Live Link pane if the frame rate in the data stream doesn't match the rendering frame rate inside UE. This is within notification within the Engine only, so it should not interfere with the project. If this notification must be removed, you can go to the Project Settings &rarr; Engine &rarr; General Settings &rarr; Framerate section, check ''Use Fixed Frame Rate'' option, and set the Fixed Frame Rate to be the same rate as the Motive frame rate.
  
[[Image:LiveLink_Timecode_7.png|center|550px|]]
+
[[Image:LiveLink_TS_Framerate.png|center]]
</div>
+
</div></div>
 
</div>
 
</div>

Latest revision as of 17:05, 24 August 2021

Main PageOptiTrack Unreal Engine 4 Plugin → Unreal Engine: OptiTrack Live Link Plugin

Overview

This page provides instructions on how to use the OptiTrack Unreal Engin Live Link plugin. The plugin communicates with Unreal's built-in Live Link system by providing a Live Link source for receiving tracking data streamed from Motive. This plugin can be used for controlling cameras and objects in virtual production applications. When needed, the OptiTrack Unreal Engine 4 Plugin can also be alongside this plugin.


Setup


1. [Motive] Setup rigid body streaming in Motive.

Get Motive streaming with at least one rigid body. Make sure the Streaming settings are configured correctly, and the rigid body is active under the Assets pane.

2. [UE] Install the OptiTrack plugins in Unreal Engine (UE).

You can install the OptiTrack Unreal Engine plugin by putting the plugin files into one of the following directories:

  • A global engine plugin can be placed in C:\Program Files\Epic Games\[Engine Version]\Engine\Plugins
  • A project-specific plugin can be placed in [Project Directory]\Plugins

3. [UE] Enable the plugins in UE project.

Go to Edit → Plugins and enable two of the required plugins. First one is the OptiTrack - Live Link plugin under Installed group, and the second one is the built-in Live Link plugin under Built-In group.

OptiTrack - Live Link plugin. Make sure the plugins are placed in the plugins folder either in the project or engine directory.
Built-In Live Link plugin.

4. [UE] Open the LiveLink pane

Open the LiveLink pane from Window → Live Link in the toolbar.

    LiveLine pane in UE.
    LiveLine pane in UE.

5. [UE] Configure and create a new OptiTrack source

In the LiveLink pane under Source options, go to the OptiTrack Source menu and configure the proper connection settings and click Create. Please make sure to use matching network settings configured from the Streaming pane in Motive.

Creating OptiTrack source with client streaming settings.

6. [UE] Check the Connection.

If the streaming settings are correct and the connection to Motive server is successful, the plugin will list out all of the detected rigid bodies, and they should have green dots next to them indicating that the corresponding rigid body has been created and is receiving data. If the dots are yellow, it means that the client has stopped receiving data. Check if Motive is still tracking or if there is a connection error.

Connected to the rigid body data stream.

Using the Plugin


Static Meshes or Camera Actors

1. Add the camera object or static mesh object that you wish to move

Add a camera actor from the Place Actors pane or a static mesh from the project into your scene. For the static meshes, make sure their Mobility setting is set to Movable under the Transform properties.

    Camera actor listed under the Place Actors pane.
    Actor mobility set to movable.

2. Add a LiveLinkController Component

Select an actor you want to animate, click the “Add Component” button on the details pane, and add Live Link Controller.

Live Link Controller component needs be added.

3. Select the target rigid body

Under the Live Link Controller component properties → Subject Representation, select the target rigid body.

The target rigid body selected under Live Link Controller component properties.

4. Check

Once the target rigid body is selected, each object with the Live Link Controller component attached and configured will be animated in the scene.

Timecode Setup

When the camera system is synchronized to another master sync device and a timecode signal is feeding into eSync, then the received timecode can be used in UE project through the plugin.

1. Set Timecode Provider under project settings

From Edit → Project Settings, search timecode and under Engine - General settings, you should find settings for the timecode. Here, set the the Timecode Provider to LiveLinkTimeCodeProvider.

Select the timecode provider under project settings.

2. Set OptiTrack source in the Live Link pane as the Timecode Provider

Open the Live Link pane, and select the OptiTrack subject that we created when first setting up the plugin connection. Then, under its properties, check the Timecode Provider box.

Enable timecode provider setting in the plugin.

3. Check

The timecode from Motive should now be seen in the Take Recorder pane. Take Recorder pane can be found under Window → Cinematic → Take Recorder in the toolbar.

Timecode shown in the Take Recorder pane.

Skeletons

1. Create a new Animation Blueprint

Right click the mesh you would like to use and select "Create > Anim Blueprint"

    New animation blueprint.

2. Name and Open the Animation Blueprint

Name the animation blueprint something reasonable, then double click it to open the blueprint.

Rename blueprint.

3. Hook up your Blueprint

Create a "Live Link Pose" component and connect it to the "Output Pose". Assign the "Live Link Subject Name" to the skeleton that you would like to use.

Connect to pose, then choose name.

Change the "Retarget Asset" property in the Details pane of the blueprint editor to "OptiTrackLiveLinkRetarget"

Retargeting option.

4. Getting the Skeleton to Animate

To animate the skeleton in real time click the Animation Blueprint from earlier. In the Details pane press "+ Add Component > Live Link Skeleton Animation". After you add that component the mesh should start animating.

To animate the skeleton in a game, just press the play button. Adding the "Live Link Skeleton Animation" object is not necessary to animate in play mode.

Component to animate skeleton.

Debugging Note

If the retargeting doesn't match the mesh correctly, then you can create a new OptiTrackLiveLinkRetarget blueprint from scratch and modify the bone mapping names.

Options for retargeting to different names.

Standalone Game Mode


For testing the project in standalone game mode, or when developing an nDislay application, the Live Link plugin settings must be saved out and selected as the default preset to be loaded onto the project. If this is not done, the configured settings may not get applied. After configuring the LiveLink plugin settings, save out the preset from the Live Link pane first. Then, open the Project Settings and find Live Link section in the sidebar. Here, you can select the default Live Link preset to load onto the project, as shown in the screenshot below. Once the preset is properly saved and loaded, the corresponding plugin settings will be applied to the standalone game mode.

If all the configuration is correct, the actors will get animated in the newly opened game window when playing the project in the standalone game mode.

  • Creating a preset from Live Link panel. Select Save As Preset
  • Assigning the preset in the Project Settings.
  • Playing a scene in standalone game mode.

Troubleshooting


Q - Trying to add more than 64 frames in the same frame. Oldest frames will be discarded.

A - This notification message may appear at the bottom of the Live Link pane if the frame rate in the data stream doesn't match the rendering frame rate inside UE. This is within notification within the Engine only, so it should not interfere with the project. If this notification must be removed, you can go to the Project Settings → Engine → General Settings → Framerate section, check Use Fixed Frame Rate option, and set the Fixed Frame Rate to be the same rate as the Motive frame rate.

LiveLink TS Framerate.png