This is always a bit tricky to get right but hopefully I can help!
Before I try and explain anything I highly recommend reading this article which has some awesome advice on naming your transformations to make understanding the multiplication order easier - https://www.sebastiansylvan.com/post/matrix_naming_convention/. In Lumberyard we use column-major notation so you need to use
localFromWorld depending on the direction but it still works out quite nicely (I use this style in all the viewport and White Box code).
To transform a point in world space to the local space of the Entity you need a
worldFromLocal transformation (as we want to go from world space to local space). If you have a point in the local space of an entity and multiply it by the entity’s transform, you get the position in world space. So you can think of the entity transform being
worldFromLocal, and you need to go the other way. To do this simply invert the
worldFromLocal transform which then gives you
localFromWorld, and you can multiply this by your direction/position in world space to get it in terms of the Entity.
If you multiply a Vector3 by a Transform, the default behaviour is to translate the point. For the direction you only want the orientation to change (no translation). To do this the Transform has a function as part of its API which I believe is called
Multiply3x3, this will transform the direction into the right space.
One other gotcha to watch out for is things can go wrong if the entity you’re working with has non-uniform scale - to get things to work initially just make sure the entity has unit scale. This will simplify things but stuff might break if you change this later (we have to do a bunch to handle this in the Manipulator code which gets really annoying… non-uniform scale leads to all sorts of annoying bugs I can tell you.)
I’ve no doubt you know more about Script Canvas than me so hopefully, you can take what I’ve said and map it to Script Canvas nodes. The other thing I’d recommend is trying to add some debug drawing to help figure out what’s going on if you can. Or start with really simple directions like down the Z-axis and then output the values to check things are being transformed as you’d expect.
Here’s some pseudocode for what you need to do…
Transform worldFromLocal = entity.getTransform();
Transform localFromWorld = worldFromLocal.getInverse();
Vector3 localRayPosition = localFromWorld * worldRayPosition;
Vector3 localRayDirection = localFromWorld.Multiply3x3(worldRayDirection);
I hope that helps a little. Another talk I’d highly recommend watching which might help with understanding some of this is an awesome talk by a speaker called Squirrel Eiserloh - https://youtu.be/o1n02xKP138 It talks a lot about these types of transformations and has a really nice explanation.