[Bugs] Some problems of kinematic models. (face direction and kinematic chain modeling)

[IsshikiHugh]

Thanks for your great work! When studying your wonderful motion representation, I found two small problem which might influence your effects. And they are all in common/skeleton.py.

1. Face Direction

This problem is already mentioned here: #112 #107. The direction of the across vector might be wrong.

HumanML3D/common/skeleton.py

Lines 58 to 62 in 6b72fcd

	l_hip, r_hip, sdr_r, sdr_l = face_joint_idx
	across1 = joints[:, r_hip] - joints[:, l_hip]
	across2 = joints[:, sdr_r] - joints[:, sdr_l]
	across = across1 + across2
	across = across / np.sqrt((across**2).sum(axis=-1))[:, np.newaxis]

Which is different from your code in motion_representation.ipynb.

HumanML3D/motion_representation.ipynb

Lines 85 to 89 in 6b72fcd

	" r_hip, l_hip, sdr_r, sdr_l = face_joint_indx\n",
	" across1 = root_pos_init[r_hip] - root_pos_init[l_hip]\n",
	" across2 = root_pos_init[sdr_r] - root_pos_init[sdr_l]\n",
	" across = across1 + across2\n",
	" across = across / np.sqrt((across ** 2).sum(axis=-1))[..., np.newaxis]\n",

This may lead to some errors, but thanks to the redundant 263-dimensional information, this can be recovered to a certain extent

2. Kinematic Chain

This problem may not be as serious as the above.

HumanML3D/common/skeleton.py

Lines 81 to 99 in 6b72fcd

	root_quat[0] = np.array([[1.0, 0.0, 0.0, 0.0]])
	quat_params[:, 0] = root_quat
	# quat_params[0, 0] = np.array([[1.0, 0.0, 0.0, 0.0]])
	for chain in self._kinematic_tree:
	R = root_quat
	for j in range(len(chain) - 1):
	# (batch, 3)
	u = self._raw_offset_np[chain[j+1]][np.newaxis,...].repeat(len(joints), axis=0)
	# print(u.shape)
	# (batch, 3)
	v = joints[:, chain[j+1]] - joints[:, chain[j]]
	v = v / np.sqrt((v**2).sum(axis=-1))[:, np.newaxis]
	# print(u.shape, v.shape)
	rot_u_v = qbetween_np(u, v)
	R_loc = qmul_np(qinv_np(R), rot_u_v)
	quat_params[:,chain[j + 1], :] = R_loc
	R = qmul_np(R, R_loc)

Here, each joints' position(except roots') will be converted to the rotation about it's parent along kinematic chain. However, not every chain start from root, which means line 85 is not very suitable especially for the arms' chains.

I thought it will be more suitable if it's like:

R = quat_params[:, chain[0]]

Again, thanks for your great work!

[EricGuo5513]

Hi, thanks for pointing this out. Now it would be too late to change the code, since many people have done experiments on the obtained dataset. However, I will highlight your issue, so people will see your comments.

[IsshikiHugh]

Hi, thanks for pointing this out. Now it would be too late to change the code, since many people have done experiments on the obtained dataset. However, I will highlight your issue, so people will see your comments.

Thanks!

[Silverster98]

Thanks for your great work! When studying your wonderful motion representation, I found two small problem which might influence your effects. And they are all in common/skeleton.py.

1. Face Direction

This problem is already mentioned here: #112 #107. The direction of the across vector might be wrong.

HumanML3D/common/skeleton.py

Lines 58 to 62 in 6b72fcd

	l_hip, r_hip, sdr_r, sdr_l = face_joint_idx
	across1 = joints[:, r_hip] - joints[:, l_hip]
	across2 = joints[:, sdr_r] - joints[:, sdr_l]
	across = across1 + across2
	across = across / np.sqrt((across**2).sum(axis=-1))[:, np.newaxis]

Which is different from your code in motion_representation.ipynb.

HumanML3D/motion_representation.ipynb

Lines 85 to 89 in 6b72fcd

	" r_hip, l_hip, sdr_r, sdr_l = face_joint_indx\n",
	" across1 = root_pos_init[r_hip] - root_pos_init[l_hip]\n",
	" across2 = root_pos_init[sdr_r] - root_pos_init[sdr_l]\n",
	" across = across1 + across2\n",
	" across = across / np.sqrt((across ** 2).sum(axis=-1))[..., np.newaxis]\n",

This may lead to some errors, but thanks to the redundant 263-dimensional information, this can be recovered to a certain extent

2. Kinematic Chain

This problem may not be as serious as the above.

HumanML3D/common/skeleton.py

Lines 81 to 99 in 6b72fcd

	root_quat[0] = np.array([[1.0, 0.0, 0.0, 0.0]])
	quat_params[:, 0] = root_quat
	# quat_params[0, 0] = np.array([[1.0, 0.0, 0.0, 0.0]])
	for chain in self._kinematic_tree:
	R = root_quat
	for j in range(len(chain) - 1):
	# (batch, 3)
	u = self._raw_offset_np[chain[j+1]][np.newaxis,...].repeat(len(joints), axis=0)
	# print(u.shape)
	# (batch, 3)
	v = joints[:, chain[j+1]] - joints[:, chain[j]]
	v = v / np.sqrt((v**2).sum(axis=-1))[:, np.newaxis]
	# print(u.shape, v.shape)
	rot_u_v = qbetween_np(u, v)
	R_loc = qmul_np(qinv_np(R), rot_u_v)
	quat_params[:,chain[j + 1], :] = R_loc
	R = qmul_np(R, R_loc)

Here, each joints' position(except roots') will be converted to the rotation about it's parent along kinematic chain. However, not every chain start from root, which means line 85 is not very suitable especially for the arms' chains.

I thought it will be more suitable if it's like:

R = quat_params[:, chain[0]]

Again, thanks for your great work!

Hi,

I noticed that some processed motions are really weird when I change R = root_quat to R = quat_params[:, chain[0]] in Line 85 in HumanML3D/common/skeleton.py. I present the motion animation with ID 000004.

000004.mp4

It seems that the left arm and right arm swap after the first frame. But I can not figure out the reason.

[QingYuan-L]

@Silverster98 Here also needs to be modified synchronously，line 140:
https://github.com/EricGuo5513/HumanML3D/blob/6b72fcda6c4fb90be69efd784942bf08be2d51e1/common/skeleton.py#L126C5-L140C38

Before

R = quat_params[:, 0] # Using global root rotation

After

R = quat_params[:, chain[0]] # Using the rotation of the current chain's starting joint

[Silverster98]

https://github.com/EricGuo5513/HumanML3D/blob/6b72fcda6c4fb90be69efd784942bf08be2d51e1/common/skeleton.py#L126C5-L140C38

Thanks, I will try.

[kai-lan]

@Silverster98 Here also needs to be modified synchronously，line 140: https://github.com/EricGuo5513/HumanML3D/blob/6b72fcda6c4fb90be69efd784942bf08be2d51e1/common/skeleton.py#L126C5-L140C38

Before

R = quat_params[:, 0] # Using global root rotation

After

R = quat_params[:, chain[0]] # Using the rotation of the current chain's starting joint

After thinking for a while, I realized it needs more modification than that.

For left and right arms, the chain's starting joint is 9 : spine3, so the rotation there should be accumulated all the way from root joint: (0: hip -> 3: spine1 -> 6: spine2 -> 9: spine3).

A quick hack is as follows

def forward_kinematics(self, quat_params, root_pos, skel_joints=None, do_root_R=True):
        # quat_params (batch_size, joints_num, 4)
        # joints (batch_size, joints_num, 3)
        # root_pos (batch_size, 3)
        if skel_joints is not None:
            offsets = self.get_offsets_joints_batch(skel_joints)
        if len(self._offset.shape) == 2:
            offsets = self._offset.expand(quat_params.shape[0], -1, -1)
        joints = torch.zeros(quat_params.shape[:-1] + (3,)).to(self.device)
        joints[:, 0] = root_pos
        for chain in self._kinematic_tree:
            if do_root_R:
                if chain[0] == 9:
                    R = global_R
                else:
                    R = quat_params[:, 0]
            else:
                R = torch.tensor([[1.0, 0.0, 0.0, 0.0]]).expand(len(quat_params), -1).detach().to(self.device)
            for i in range(1, len(chain)):
                R = qmul(R, quat_params[:, chain[i]])
                if chain[i] == 9:
                    global_R = R.copy()
                offset_vec = offsets[:, chain[i]]
                joints[:, chain[i]] = qrot(R, offset_vec) + joints[:, chain[i-1]]
        return joints