To test this feature, visit your live site.

Feature Request: Puppet Tool & Mask Tool

Hi RainyRizzle,

When I look at AnyPortrait, it shows great similarities with Live2D. But if it also starts to show similarities with After Effects (in useful tools), it will be such a powerful animation tool...

I have been doing 2D character animations via After Effects for a long time. And there is a critical tool for me, its name is Puppet Tool.

Puppet tool has: Auto mesh generation, Expansion, Density

Thanks to this tool, it becomes very easy to give deformation on the image.

Moreover, we can create vertices automatically and adjust their number and frequency.

I will show you what it does in a simple way with a video example.

As you know, we have to create dozens of layers for meshes for minutes or hours. We used to create it completely by hand, then you brought the auto generate feature and it really saved us time. However, even if it is automatic, we have to fix it manually and unfortunately the UX still needs to improve, it takes time to change things.

So why wouldn't it be better? This tool creates everything automatically and we can control it with parameters.

Personally, I often use this tool when I want to add flexibility and provide a soft animation while doing After Effects character animations. (I use it; chest animations, breath, eye blink, slight movement on joints etc...) In short, almost everywhere. It is such a useful and powerful tool. Please take a look how do I make Eye Blink animation in After Effects:

(Note: Mask Tool is required for to hiding the mesh when eye closed)

Full Video: https://www.youtube.com/watch?v=OZDoJj1ILec (7:19)

Puppet Tool used for eyelashes, Mask Tool used for eye iris.

Why Mask Tool is required?

Because in the current AnyPortrait structure, we pull down the upper points to close the eye. This is actually true, but since there is no mask tool, it feels like you are scaling down, and this is not a good thing for me. In my personal opinion, the real solution for eye closing animation should be with the mask tool.

See the Eye Blink in AnyPortrait (it feels like eye iris scaling down):

I hope I could explain better why this tool is necessary.

By the way, yes I know, After Effects is a huge program developed by Adobe with hundreds of thousands of employees, while AnyPortrait has a small team developed by one or a few people. I can imagine that it will not be easy and maybe even more difficult due to Unity Engine limitations.

But in any case, it is inevitable that it is a really important tool.

I mentioned to you in the past that I needed a mask tool and we provided a temporary solution with shaders. I could not preview the mask in the editor and I had to run the game to see it and I had to start the game and test the animation over and over again to make that mask.

You cannot imagine how good it would be if there was a mask tool in AnyPortrait and we could use it in the animation timeline and as a morph.

This was a suggestion, I hope it helps to improve AnyPortrait better.

With my kind regards :)

8 comments

Comments (8)

RainyRizzle

Hi!

Thank you for suggesting a great feature.

We would like to respond to the two ideas you suggested.

We apologize in advance for the slightly long response.

The first is about the proposal related to the introduction of Puppet Tool.

We had previously received a proposal for that feature and reviewed it.

It is not only because of the difficulty of implementing that feature, but also because of two aspects that we decided that it would not be easy to implement it as is.

The first is "Performance".

Users use AnyPortrait for various purposes, but this tool was basically developed for games.

So we are quite concerned about introducing features that affect performance.

After Effects is not a real-time rendering program, so it is free from this limitation, but it will not be easy to introduce it in a program for games.

Of course, if we develop really well, this issue can be overcome, but we do not think positively.

We will explain this further in the "Pin Tool" section below.

The second is the "Structural problem".

AnyPortrait is a tool that adapts the 3D character creation method for games to 2D.

So it has a traditional procedural work method of "Image > Mesh > Rigging > Animation".

Since this is related to system design, it is quite difficult to create a function that goes beyond this structure.

(This is probably the main reason why there are restrictions on the implementation of functions in production programs.)

"Puppet Tool" is basically a function corresponding to the "Rigging" stage, but in After Effects, it can cause various side effects because it involves the "Mesh" stage before that.

If we had considered Puppet Tool from the beginning and designed AnyPortrait more flexibly, it would have been easier to implement, but it is not easy to implement the function in an already completed system.

Even so, we also wanted to utilize the advantages of Puppet Tool.

So our team implemented the ideas that make up Puppet Tool one by one.

One of them is the function to automatically generate meshes in batches, and the same goes for the "Pin Tool" that we implemented.

For this function, "Pin Tool", please refer to the following manuals.

https://rainyrizzle.github.io/en/AdvancedManual/AD_AddPin.html

https://rainyrizzle.github.io/en/AdvancedManual/AD_PinMorphWork.html

Since Puppet Tool has a working principle closer to Morph than Rigging, we added "Pin Tool" to Morph modifier.

At first glance, it seems like we can implement Puppet Tool like Pin Tool, but in fact, Pin Tool is a function that suffered from the "Performance" problem described above.

Pin Tool requires quite a lot of computation.

This is because it performs "Runtime Rigging", "Transform", and "Morph" at once.

Since it was not easy to operate this function in-game, we overcame this problem by "converting the corresponding data to Morph data and saving it when Baking is executed."

We would appreciate it if you could understand that we also tried our best to overcome the limitations of Puppet Tool.

Of course, this is not enough to fulfill the key idea of Puppet Tool.

You seem to think of this feature as a convenience feature of Morph and suggested it to us, but other users suggested it as a "more flexible Rigging Tool".

So we are looking for the answer in "Bone IK" to solve this.

Recently, we have made a lot of improvements to the IK algorithm.

This is the basic step for "Variable Length IK" and "IK-based Animation" that we are currently working on.

It is unlikely that we will be able to release these features in the near future, but please keep an eye out for them.

Even if the result will probably be closer to Spine than After Effects, we will try to show you more interesting results than now.

Next is the answer for your second question about Mask.

At first, we thought the Mask tool was easy.

We simply thought that we could implement the feature with creating a "mesh that doesn't render and only acts as a mask."

When we searched for related cases, we found that creators were using it in more complex and diverse ways, such as "combining masks" , "inserting metadata into separate channels or RenderTextures to create outlines" or etc.

So we are reviewing it to provide expandability, but this will cause problems that conflict with the existing rendering system design.

In particular, it is not an easy problem because we have to make major changes to the "Material Library" that manages materials and the rendering optimization system.

However, this problem is an internal matter for us, so please think of it as not an unsolvable problem, but just a bit of a difficult predicament for us.

Thank you again for the great ideas.

We probably won't be able to show you these ideas anytime soon, but we'll do our best to show you someday.

Thank you.

DryreL

Replying to

Hi RainyRizzle,

No problem, let me explain it again.

It would be better if I give an example with a real usage scenario why I need Mask Shape. It would be better if you could even tell me how to solve this in the current version of AnyPortrait.

I found an image on the web, will make some quick adjustments to show you.

Use Scenario:

A character has a food or weapon or anything in their hand.

Let's think there's a milkshake on it's hand.

I want to see this character drinking a milkshake in an animation I make.

The straw of the milkshake needs to be masked when it gets close to the character's mouth so that it looks like they are drinking from it.

I created a custom mask shape (in green area).

(Image taken from Lexica)

In the next frames, the character is drinking the milkshake. So the straw needs to be in the character's mouth.

In this example I used a rectangle but a custom shape is definitely required, this is because the character's lip will be curved so we will shape the mask according to the shape of the lip.

Now let's move our mask.

When I pull the mask down from the top, the red X will now be out and the top will be hidden.

Now only mask shape (green area) is visible. Great!

Final version:

This was something I quickly set up and visualized in Photoshop.

Milkshake is just an example, these examples can be varied.

But I think you understand me better why it is necessary. It would be great to be able to preview it in the editor.

I hope my explanation was more explanatory with the images.

Thank you!

Edited

RainyRizzle

Replying to

Thank you for your detailed explanation.

We understand your concerns.

We will try to improve the mask to improve usability and expandability.

Thank you.

DryreL

Replying to

I'm glad you understood with the visual explanation!

I was surprised that AnyPortrait, which I first started using in 2020, did not have such an important feature. I was expecting it to be have this feature with updates. So 4 years is not a short time :)

But I'm happy to see that you're taking it into consideration now. I use the Mask Shape tool so much in my own animations that not being able to use it in AnyPortrait made me feel like I was missing one of my hands (haha).

Now I'm happier, thank you :)