Modularised subclassing

This article is based on Visual Basic 6. Find other Visual Basic 6 articles.

In an extension of this method, it is possible to 'modularise' subclassing and have your subclassing handled inside a form or class module.

To do this you will need an extra class module, called ISubclass:

 Public Function WndProc(ByVal hwnd As Long, ByVal uMsg As Long, ByVal wParam As Long, ByVal lParam As Long, bHandled As Boolean, RetVal As Boolean) As Long
 
 End Function

This class looks useless, and, on its own, is. However, what you have created with this class is an interface that can be 'implemented' (a form of inheritance supported by VB6) into other class modules, and forms. You could omit this and opt for a simpler, but less flexible, solution, but if you are familiar with inheritance then the benefits of using this particular method will be apparent. If you are not familiar with implementation inheritance then there are plenty of tutorials out there - http://www.google.co.uk/search?hl=en&safe=off&q=vb6+implementation+inheritance&spell=1

Now for the module coding: The module will work much in the same way as the previous example, except for the fact that you are not passed a pointer or a reference to your module function, which is the Window Procedure for whatever window you are subclassing. So how do you get access to your object? You use ObjPtr and SetProp as follows:

First things first, we will have to write a function to subclass a window (note: API declarations are not shown here).

 Public Sub SubclassWnd(ByVal hwnd As Long, oHandler As ISubclass) 
     Dim pOldWndProc As Long
     
     pOldWndProc = SetWindowLong(hwnd, GWL_WNDPROC, AddressOf fnWndProc)
     SetProp hwnd, "pOldWndProc", pOldWndProc
     SetProp hwnd, "pHandler", ObjPtr(oHandler)
 End Sub

What's going on here? The sub is passed a reference to an instance of ISubclass (which is really a reference to your class/form that is type-compatible with ISubclass because it implements it). Once the window is subclassed in the normal way, we use SetProp to give the window a property telling us what the old window proc address is, and one which is a pointer to our ISubclass instance. As we are passed the hwnd of the window in the Window Procedure, we can then use the properties of the window to retrieve the ISubclass object and call methods on it.

Before we proceed with the Window Procedure, let's just make a quick sub to un-subclass the window:

 Public Sub UnSubclassWnd(ByVal hwnd As Long)
     Dim pOldWndProc As Long
     
     pOldWndProc = GetProp(hwnd, "pOldWndProc")
     SetWindowLong hwnd, GWL_WNDPROC, pOldWndProc
     RemoveProp hwnd, "pOldWndProc"
     RemoveProp hwnd, "pHandler"
 End Sub

As you can see, the use of GetProp is demonstrated here in order to get the old Window Proc address and set it back to that. RemoveProp is also used to clean up a bit.

On with the Window Proc:

 Private Function fnWndProc(ByVal hwnd As Long, ByVal uMsg As Long, ByVal wParam As Long, ByVal lParam As Long) As Long
     Dim pOldWndProc As Long
     Dim pHandler As Long
     Dim oHandler As ISubclass
     Dim bHandled As Boolean
     Dim bRet As Boolean
     Dim RetVal As Long
     
     pOldWndProc = GetProp(hwnd, "pOldWndProc")
     pHandler = GetProp(hwnd, "pHandler")
     
     If pHandler <> 0 Then
         CopyMemory oHandler, pHandler, 4&
         RetVal = oHandler.WndProc(hwnd, uMsg, wParam, lParam, bHandled, bRet)
         ZeroMemory oHandler, 4&
     End If
     
     If Not bHandled Then
         If pOldWndProc <> 0 Then
             If Not bRet Then
                 fnWndProc = CallWindowProc(pOldWndProc, hwnd, uMsg, wParam, lParam)
             Else
                 Call CallWindowProc(pOldWndProc, hwnd, uMsg, wParam, lParam)
                 fnWndProc = RetVal
             End If
         Else
             If Not bRet Then
                 fnWndProc = DefWindowProc(hwnd, uMsg, wParam, lParam)
             Else
                 Call DefWindowProc(hwnd, uMsg, wParam, lParam)
                 fnWndProc = RetVal
             End If
         End If
     Else
         If Not bRet Then
             fnWndProc = 0
         Else
             fnWndProc = RetVal
         End If
     End If
 End Function

While this may look a little daunting at first, the only complex thing that is going on here really is the usage of GetProp and CopyMemory to get an ISubclass instance. This works exactly the same as getting the old Window Proc address as we did in the unsubclassing procedure, and exactly the same as getting an object reference from a pointer in the previous example.

All of the rest of the function is simply to handle return values and the 'bHandled' variable that is passed to ISubclass.WndProc. If ISubclass.WndProc returns with bHandled true then CallWindowProc and DefWindowProc are not called, otherwise one or the other is called depending on whether or not GetProp returned to us a proper address for the old Window Proc. In all cases, if bRet is true when ISubclass.WndProc returns then the return value of ISubclass.WndProc is returned with fnWndProc. Otherwise the return of CallWindowProc/DefWindowProc or 0 is returned.