Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Copy/Re-use animation from one instance on another one with the same skeleton #7622

Open
hannojg opened this issue Mar 1, 2024 · 8 comments · Fixed by margelo/react-native-filament#21
Labels
enhancement New feature or request gltf Specific to glTF support

Comments

@hannojg
Copy link
Contributor

hannojg commented Mar 1, 2024

Is your feature request related to a problem? Please describe.

In my use-case I have a main character model. This model contains all animations for the character.
I now have many other models that are clothes or accessories that have the same skeleton as the character, but don't contain any animations.
When the character equips for example a hat and we run the idle animation of the character, we copy over the idle animation to the hat model. Playing both simultaneously gives the impression of the hat fitting on the character's hat, as it's following all of its animations.

The current app code is written in threejs but we want to migrate to filament, and we're wondering if we can use the same approach with filament.
Here is an example code snippet of how to do this in threejs:

// this._character being the main model with all animations and
// this._accessoryObject being the accessory model we want to copy the animations to
this._character.copyAnimationsTo(
	this._accessoryObject.animations,
	this._accessoryObject.mixer,
);

copyAnimationsTo(animationsMap: { [key: string]: THREE.AnimationAction }, mixer: THREE.AnimationMixer) {
	const animations = this._character.animations
	animations.forEach(animation => {
		const animationAction = mixer.clipAction(animation)
		animationsMap[animation.name] = animationAction
	})
}

Describe the solution you'd like

I haven't thought of any exact solution yet, as I feel it might introduce a new API (if the feature doesn't already exist?) and I don't have any specific API in mind.

Maybe something like:

animator->addInstance(otherAsset->getInstance())

Describe alternatives you've considered

I see that the Animator has (internal) code i.e. addInstance, however thats internal and only usable with an FFilamentInstance.

OS and backend

I think it's not.

@hannojg
Copy link
Contributor Author

hannojg commented Mar 1, 2024

I got it working by making a new public constructor for the Animator:

// Animator.h
public:
Animator(FilamentAsset *asset, FilamentInstance *instance) : Animator(downcast(asset), downcast(instance))
{
}

and then instantiating an animator on my own like this:

Animator* animator = new gltfio::Animator(mainCharacterAsset, hatAsset->getInstance());

And then using this animator instead of hatAsset->getInstance()->getAnimator() to drive the animation.

I am sure the constructor was private for a reason and shouldn't be exposed 😅 But it proves it's possible with filament, which is awesome! Are you open to discuss an API to enable this feature?

@hannojg
Copy link
Contributor Author

hannojg commented Mar 1, 2024

As a side note: I tried to use animator->addInstance with a FilamentInstance (I changed the code for that).
After doing so I was under the impression that the instance would be then controlled by the same animator. However the animator only animated the main instance (and not the one I added with addInstance).

In our use case it would be actually beneficial to use the same animator for all the (different) instances, to save resources. However, I am not sure if thats feasible?

@pixelflinger pixelflinger added enhancement New feature or request gltf Specific to glTF support labels Mar 5, 2024
@mrousavy
Copy link

mrousavy commented Mar 8, 2024

I feel like that use-case should be handled fundamentally different though, no? As in; use one model for that specific animation set, then work with that model, e.g. by attaching skins to it (like a hat on the penguin), instead of creating a new model with the hat and copying over the animations.
But I might be wrong in how filament works.

@hannojg
Copy link
Contributor Author

hannojg commented Mar 13, 2024

Note: The copied animator wasn't supporting applyCrossFade correctly, I addressed that in this PR:

@ninjz
Copy link

ninjz commented Mar 19, 2024

I haven't yet tried @hannojg's solution but if it works to be able to apply the animation from one model to another with the same bone structure it would be very beneficial to add the public constructor.

My use case I've talked about in this thread: SceneView/sceneview-android#326 I've been pulling my hair out for a while now over this issue. I've been using SceneView in my project which depends on Filament and though I will probably make these changes in my own fork it would be nice to have these changes applied upstream to the main projects.

Is there any particular reason the public constructor shouldn't be added?

@hannojg
Copy link
Contributor Author

hannojg commented May 15, 2024

I think I found a better solution to this than copying the whole animator.

The problem with copying the animator is that all animation data will be duplicated. In our case we had a lot and complex animations. This easily starts taking a lot of memory, so that approach is kind of suboptimal.

Our use case is character clothing. The clothings share the same skeleton as the main character.
So a better solution is to apply all transformation of the bones of the character to the bones of the clothing.

In practice I implemented this in three steps:

  1. Loop over all entities, their names and their transforms from the main character
  2. Loop over all entities of the clothing, and apply the same transformation to entities matching the same name
  3. Lastly, update the bone matrices of the clothing using the animator instance/state from the main character

This way we are not duplicating any animation data.

To get point 3) to work I needed to add a simple method to the Animator class. Will open a PR for that soon, really hope we can support this use case 🙏

// Edit: with this PR we are able to get the use case of clothing working:

@ninjz
Copy link

ninjz commented May 20, 2024

@hannojg Thanks for your work on this. This is a really important functionality for my use case.

I'm trying your solution on my Android application but I'm having troubles playing the animation from the animation file to the model.

This is what I'm doing to apply the updateBoneMatricesForInstance call:

inner class FrameCallback : Choreographer.FrameCallback {
    private val startTime = System.nanoTime()
    override fun doFrame(frameTimeNanos: Long) {
        choreographer.postFrameCallback(this)
        modelViewer.render(frameTimeNanos)

        animationAnimator?.apply {
            if (animationCount > 0) {
                val elapsedTimeSeconds = (frameTimeNanos - startTime).toDouble() / 1_000_000_000
                applyAnimation(0, elapsedTimeSeconds.toFloat())
            }
            updateBoneMatricesForInstance(modelViewer.asset!!.instance)
        }
    }
}

And this is what I have for loading the model and the animation:

private fun loadAnimation() {
   val engine = Engine.create()
   val materialProvider = UbershaderProvider(engine)
   val assetLoader = AssetLoader(engine, materialProvider, EntityManager.get())
   val resourceLoader = ResourceLoader(engine, true)

   val buffer = assets.open("animations/jogging.glb").use { input ->
       val bytes = ByteArray(input.available())
       input.read(bytes)
       ByteBuffer.allocateDirect(bytes.size).apply {
           order(ByteOrder.nativeOrder())
           put(bytes)
           rewind()
       }
   }

   val asset = assetLoader.createAsset(buffer);
   asset?.let { asset ->
       resourceLoader.asyncBeginLoad(asset)
       animationAnimator = asset.instance.animator
       asset.releaseSourceData()
   }
}

private fun loadModel() {
   val buffer = assets.open("models/female_masterScene.glb").use { input ->
       val bytes = ByteArray(input.available())
       input.read(bytes)
       ByteBuffer.allocateDirect(bytes.size).apply {
           order(ByteOrder.nativeOrder())
           put(bytes)
           rewind()
       }
   }

   modelViewer.loadModelGlb(buffer)
   modelViewer.transformToUnitCube()
}

What am I missing here?

@hannojg
Copy link
Contributor Author

hannojg commented May 29, 2024

Yeah so for the solution you have to sync all entities, as I outlined here:

  1. Loop over all entities, their names and their transforms from the main character
  2. Loop over all entities of the clothing, and apply the same transformation to entities matching the same name
  3. Lastly, update the bone matrices of the clothing using the animator instance/state from the main charac

The PR I put up only concerns the third (3) point. An implementation of the first two could look something like this (I think sharing this is interesting for the full perspective on this issue and maybe the filament maintainers can better understand the use case [because maybe there is a better solution!]):

(Pseudo code):

void AnimatorWrapper::applyAnimationToInstance(FilamentInstance* instanceToSync) {
  // _instance is the FilamentInstance to the master asset containing the animation we want to sync with
  FilamentInstance* masterInstance = _instance;

  const FilamentAsset* asset = instanceToSync->getAsset();
  Engine* engine = asset->getEngine();
  TransformManager& transformManager = engine->getTransformManager();
  Animator* masterAnimator = masterInstance->getAnimator();

  // 👀 STEP 1
  // Get name map for master instanceToSync
  size_t masterEntitiesCount = masterInstance->getEntityCount();
  const Entity* masterEntities = masterInstance->getEntities();
  std::map<std::string, Entity> masterEntityMap;
  for (size_t entityIndex = 0; entityIndex < masterEntitiesCount; entityIndex++) {
    const Entity masterEntity = masterEntities[entityIndex];
    NameComponentManager::Instance masterNameInstance = GlobalNameComponentManager::getInstance()->getInstance(masterEntity);
    if (!masterNameInstance.isValid()) {
      continue;
    }
    auto masterInstanceName = GlobalNameComponentManager::getInstance()->getName(masterNameInstance);
    masterEntityMap[masterInstanceName] = masterEntity;
  }

  // Get name map for instanceToSync
  size_t instanceEntitiesCount = instanceToSync->getEntityCount();
  const Entity* instanceEntities = instanceToSync->getEntities();
  std::map<std::string, Entity> instanceEntityMap;
  for (size_t entityIndex = 0; entityIndex < instanceEntitiesCount; entityIndex++) {
    const Entity instanceEntity = instanceEntities[entityIndex];
    NameComponentManager::Instance instanceNameInstance = GlobalNameComponentManager::getInstance()->getInstance(instanceEntity);
    if (!instanceNameInstance.isValid()) {
      continue;
    }
    auto instanceName = GlobalNameComponentManager::getInstance()->getName(instanceNameInstance);
    instanceEntityMap[instanceName] = instanceEntity;
  }

  // 👀 STEP 2
  // Sync the same named entities:
  for (auto const& [name, masterEntity] : masterEntityMap) {
    auto instanceEntity = instanceEntityMap[name];
    if (instanceEntity.isNull()) {
      continue;
    }

    // Sync the transform
    TransformManager::Instance masterTransformInstance = transformManager.getInstance(masterEntity);
    TransformManager::Instance instanceTransformInstance = transformManager.getInstance(instanceEntity);

    if (!masterTransformInstance.isValid() || !instanceTransformInstance.isValid()) {
      Logger::log(TAG, "Transform instanceToSync is for entity named %s is invalid", name.c_str());
      continue;
    }

    math::mat4f masterTransform = transformManager.getTransform(masterTransformInstance);
    transformManager.setTransform(instanceTransformInstance, masterTransform);
  }

  // 👀 STEP 3
  // Syncing the bones / joints
  masterAnimator->updateBoneMatricesForInstance(instanceToSync);
}

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
enhancement New feature or request gltf Specific to glTF support
Projects
None yet
Development

Successfully merging a pull request may close this issue.

4 participants