c++ - C# WndProc method not receiving expected Msg type

Question

I've been trying to communicate with another software using its documented COPYDATA API. User Xanotos has been incredibly helpful in this question I asked which has the sending method sorted and working fine. I am having issues that the receiving WndProc method does not seem to catch the expected response. Here is the link to COPYDATA API documentation for reference.

The current method is below. Testing has shown the WndProc does receive messages, but not the ones I expect, namely a struct depending on which message was sent.

Declarations:

 [DllImport("User32.dll", SetLastError = true, EntryPoint = "FindWindow")]
    public static extern IntPtr FindWindow(String lpClassName, String lpWindowName);

    [DllImport("User32.dll", SetLastError = true, EntryPoint = "SendMessage")]
    public static extern IntPtr SendMessage(IntPtr hWnd, int Msg, int wParam, ref COPYDATASTRUCT lParam);

    [StructLayout(LayoutKind.Sequential)]
    public struct COPYDATASTRUCT
    {
        public IntPtr dwData;    // Any value the sender chooses.  Perhaps its main window handle?
        public int cbData;       // The count of bytes in the message.
        public IntPtr lpData;    // The address of the message.
    }
    public struct ExternalGetPositionType
    {
        public double X;
        public double Y;
        public double Z;
        public double W;
    }
    const int WM_COPYDATA = 0x004A;
    const int EXTERNAL_CD_COMMAND_RUN_ASYNC = 0x8001;
    const int EXTERNAL_CD_GET_POSITION_PCS = 0x8011;
    const int EXTERNAL_CD_GET_POSITION_MCS = 0x8012;

Sending Methods (these all work fine and return true):

public static IntPtr RunAsync(IntPtr hwnd, string str)
    {
        // We have to add a  terminator, so len + 1 / len + 2 for Unicode
        int len = Encoding.Default.GetByteCount(str);
        var buff = new byte[len + 1]; // len + 2 for Unicode
        Encoding.Default.GetBytes(str, 0, str.Length, buff, 0);

        IntPtr ret;

        GCHandle h = default(GCHandle);

        try
        {
            h = GCHandle.Alloc(buff, GCHandleType.Pinned);

            var cds = new COPYDATASTRUCT();
            cds.dwData = (IntPtr)EXTERNAL_CD_COMMAND_RUN_ASYNC;
            cds.lpData = h.AddrOfPinnedObject();
            cds.cbData = buff.Length;

            ret = SendMessage(hwnd, WM_COPYDATA, 0, ref cds);
        }
        finally
        {
            if (h.IsAllocated)
            {
                h.Free();
            }
        }

        return ret;
    }

    public static IntPtr GetPosition(IntPtr hwnd, bool pcs, ExternalGetPositionType position)
    {
        // We cheat here... It is much easier to pin an array than to copy around a struct
        var positions = new[]
        {
    position
};

        IntPtr ret;

        GCHandle h = default(GCHandle);

        try
        {
            h = GCHandle.Alloc(positions, GCHandleType.Pinned);

            var cds = new COPYDATASTRUCT();
            cds.dwData = pcs ? (IntPtr)EXTERNAL_CD_GET_POSITION_PCS : (IntPtr)EXTERNAL_CD_GET_POSITION_MCS;
            cds.lpData = h.AddrOfPinnedObject();
            cds.cbData = Marshal.SizeOf<ExternalGetPositionType>();

            ret = SendMessage(hwnd, WM_COPYDATA, 0, ref cds);
        }
        finally
        {
            if (h.IsAllocated)
            {
                h.Free();
            }
        }

        return ret;
    }

WndProc Method - This is where things don't work as expected. The if(m.Msg == WM_COPYDATA) never returns true when GetPosition is called.

protected override void WndProc(ref Message m)
    {
        if (m.Msg == WM_COPYDATA) //this does not execute
        {
            COPYDATASTRUCT cds = Marshal.PtrToStructure<COPYDATASTRUCT>(m.LParam);
            label5.Text = "message received";
            if (cds.dwData == (IntPtr)EXTERNAL_CD_COMMAND_RUN_ASYNC)
            {
                label5.Text = "NORMAL";
                string str = Marshal.PtrToStringAnsi(cds.lpData);

                Debug.WriteLine($"EXTERNAL_CD_COMMAND_RUN_ASYNC: {str}");
                toolStripStatusLabel1.Text = $"COMMAND";
                m.Result = (IntPtr)100; // If you want to return a value
            }
            else if (cds.dwData == (IntPtr)EXTERNAL_CD_GET_POSITION_PCS) //this does not execute
            {
                label5.Text = "MSC";
                if (cds.cbData >= Marshal.SizeOf<ExternalGetPositionType>())
                {
                    var position = Marshal.PtrToStructure<ExternalGetPositionType>(cds.lpData);

                    Debug.WriteLine($"EXTERNAL_CD_GET_POSITION_PCS: X = {position.X}, Y = {position.Y}, Z = {position.Z}, W = {position.W}");
                    toolStripStatusLabel1.Text = $"External MCS = {position.X}";
                    label4.Text = position.X.ToString();

                    m.Result = (IntPtr)200;
                }
                else
                {
                    m.Result = (IntPtr)0;
                }
            }
            else if (cds.dwData == (IntPtr)EXTERNAL_CD_GET_POSITION_MCS) //this does not execute
            {
                label5.Text = "MSC"; //this does not execute
                if (cds.cbData >= Marshal.SizeOf<ExternalGetPositionType>())
                {
                    var position = Marshal.PtrToStructure<ExternalGetPositionType>(cds.lpData);

                    Debug.WriteLine($"EXTERNAL_CD_GET_POSITION_MCS: X = {position.X}, Y = {position.Y}, Z = {position.Z}, W = {position.W}");
                    toolStripStatusLabel1.Text = $"External MCS = {position.X}";
                    label4.Text = position.X.ToString(); 

                    m.Result = (IntPtr)300;
                }
                else
                {
                    m.Result = (IntPtr)0;
                }
            }

            return;
        }
        MessageBox.Show(m.Msg.ToString()); //this DOES execute
        
        base.WndProc(ref m);
    }

Both programs are 32 bit.

Any ideas why I am not receiving the expected Msg?

Answer 1

The documentation you provided is unclear, but it seems you need to set wParam as the receiving window handle to be able to receive data in response.

At the bottom of page 3 it has an example:

::SendMessage(m_pWnd->GetSafeHwnd(), WM_COPYDATA,(WPARAM)this->GetSafeHwnd( ,(LPARAM)&MyCDS);

Whereas your code has this:

ret = SendMessage(hwnd, WM_COPYDATA, 0, ref cds);

Instead you need

ret = SendMessage(hwnd, WM_COPYDATA, this.Handle, ref cds);

You must also redefine wParam on SendMessage as IntPtr. It is in any case wrong, and probably only worked because you were on 32-bit.