Checking user name of refresh token

[VILLAN3LL3]

Maybe I got something wrong but I didn't find the point where the user name of the loaded refresh token is checked. The only thing which is checked is the client id.

You don't need an access token to create a new access token via refresh token.
So if someone else manages to get your refresh token id, they will always be able to issue a new access token in your name.

Shouldn't the refresh token challenge at least be protected by a (potentially expired, but at least validated) access token so that we can compare the username of the access token to the username of the refresh token object?

Is this a security issue or am I missing something?