How can i modify my code to perform raycast?

I write c++ and lua to perform raycast.
First i write c++ to return direction.

AZ::Vector3 GetRaycastHit::getDirection(float x, float y) {
		// invert the mouse y for the raycast

		float invMouseY = static_cast<float>(gEnv->pRenderer->GetHeight()) - y;
		Vec3 mStart, mEnd;
		gEnv->pSystem->GetViewCamera().Unproject(Vec3(x, invMouseY, 0.0f), mStart);
		gEnv->pSystem->GetViewCamera().Unproject(Vec3(x, invMouseY, 1.0f), mEnd);

		Vec3 direction = mEnd - mStart;

		return AZ::Vector3(direction.x, direction.y, direction.z);


And expose to lua:

					->Method("getdir", &test::GetRaycastHit::getDirection)
					//->Method("raycast", &test::GetRaycastHit::perform2Raycast)
				//	->Method("getstring", &test::GetRaycastHit::getString)
				//	->Method("ray333", &test::GetRaycastHit::perform3Raycast)

Then i perform raycast in lua when i click mouse:

self.Inputs.OnPressed =function ()
		local a=UiCursorBus.Broadcast.GetUiCursorPosition()
		Debug.Log("cursor: x=" .. a.x .. "    y=" .. a.y)
		local ray=GetRaycastHit()

		local dir=ray:getdir(a.x,a.y)
		Debug.Log("dir=" .. dir.x .. "  " .. dir.y .. "  " .. dir.y)
		local ray2 = RayCastConfiguration()
	    ray2.origin = self.cameraPosition
	    ray2.direction = dir

	    ray2.maxDistance = 1000.0
	    local  result = PhysicsSystemRequestBus.Broadcast.RayCast(ray2)
		Debug.Log("hit entity name=" .. GameEntityContextRequestBus.Broadcast.GetEntityName(self.hitEntity))
	    	Debug.Log("#result < 0")

If i set the rotation of camera x=270, y=0, z=0,the direction is almost same,and dont have debugdraw.

I i set the rotation of camera x=270, y=0.1,z=0.1,the direction is different and show debugdraw,but i cant click the entity accurately.

Sorry for my bad name convention.
Could someone help me modify my code?

hi! to the find the best practice about this you are better search similar code in engine sources)
for this you can use TotalCommander do find all uses of word “Unproject” in modern Ly’s components

I finded this for you, you can try use same aprouch that they do. when they trying to select UIWorldCanvases.

bool UiCanvasManager::HandleInputEventForInWorldCanvases(const AzFramework::InputChannel::Snapshot& inputSnapshot, const AZ::Vector2& viewportPos)
    // First we need to construct a ray from the either the center of the screen or the mouse position.
    // This requires knowledge of the camera
    // for initial testing we will just use a ray in the center of the viewport
    CCamera& cam = GetISystem()->GetViewCamera();

    // construct a ray from the camera position in the view direction of the camera
    const float rayLength = 5000.0f;
    Vec3 rayOrigin(cam.GetPosition());
    Vec3 rayDirection = cam.GetViewdir() * rayLength;

    // If the mouse cursor is visible we will assume that the ray should be in the direction of the
    // mouse pointer. This is a temporary solution. A better solution is to be able to configure the
    // LyShine system to say how ray input should be handled.
    bool isCursorVisible = false;
    UiCursorBus::BroadcastResult(isCursorVisible, &UiCursorInterface::IsUiCursorVisible);
    if (isCursorVisible)
        // for some reason Unproject seems to work when given the viewport pos with (0,0) at the
        // bottom left as opposed to the top left - even though that function specifically sets top left
        // to (0,0).
        const float viewportYInverted = cam.GetViewSurfaceZ() - viewportPos.GetY();

        // Unproject to get the screen position in world space, use arbitrary Z that is within the depth range
        Vec3 flippedViewportPos(viewportPos.GetX(), viewportYInverted, 0.5f);
        Vec3 unprojectedPos;
        cam.Unproject(flippedViewportPos, unprojectedPos);

        // We want a vector relative to the camera origin
        Vec3 rayVec = unprojectedPos - rayOrigin;

        // we want to ensure that the ray is a certain length so normalize it and scale it
        rayDirection = rayVec * rayLength;
1 Like