Over the last decade I've done a lot of work on the python wrappers. The wrapper-generator code in "vtkPython.c" grew from 760 lines of code mid-way through my PhD, to 5300 lines of code last month. And a lot of that code was within a single, massive function. That was the function that generated a python "wrapper" function for each method in the C++ class being wrapped. Now, my guideline is that a comfortable file length is 1000 lines, and anything over 1500 is too long.

So I split off three files: one for Python-to-C++ data conversion functions, one for reusable code-generation functions, and one for reusable text/help processing. And the main file, still 3700 lines long, was reorganized into many smallish subroutines with clear inputs and outputs. But the most important thing is that the generated code, i.e. the C++ code that this code writes out, is much cleaner and more compact than before. No goto statements or complex error handling logic, just a simple flow from beginning to end. And it is 25% faster than before.

This was the last time that I'll have to hold the whole file in my head at once, so to speak. Now it is possible to understand the program one subroutine at a time, and I won't have to do a major study of the code every time that I want to add a couple new features. Which is good, because I have many features left to add, but I'll have to do them in bits and snatches when I find the time.

Last week I switched vtkWrapPython.c over to the new vtkParse API, which means that vtkWrapPython now has access to much more information about the VTK classes than it used to. After this was done, it was amazing how straightforward it was to add new features to the python wrappers:

• Multidimensional args like Multiply3x3(double A[3][3], double x[3], double y[3]) are wrapped.
• Reference args like GetDistance(double p1[3], double p2[3], double &d) are wrapped.
• The types size_t and ssize_t are wrapped.
• All typedef'd types are wrapped, as long as they resolve to a supported type.
• Methods with optional args with default values are properly wrapped.

All of these changes except for reference args were only possible because of the move to the new vtkParse API. The old API simply did not make enough information available.

The reference arg support required a different bit of trickery. Python's ints and floats are immutable (as are Python strings) so it is impossible to change their value through a reference arg the way that C does. So the obvious solution was to make a new python type that is a mutable numeric type. My new mutable() type behaves just like a number, except that its value can be changed. You construct a mutable object like this, mutable(2), and it will behave just like a "2". I also plan to make it so that mutable('string') will generate a mutable string object.

Pointers are still not wrapped, except in cases where the wrappers know the size of the memory area that they point to. Proper wrapping of pointers is dependent on developing a hinting system that will allow the wrappers to deduce the size. For the vtkDataArray::SetTuple(double *) method, I have hard-coded a hint that the size is given by GetNumberOfComponents(). This works fine for this one class, but there are tons of pointer methods in VTK and they cannot all be hard-coded into the wrappers. Once a good hinting scheme is in place, most of these methods can be automatically wrapped. That is a project that will have to wait for another day.

Python and VTK each have their own garbage collection systems, with reference counts as the primary means of tracking references. The two GC systems are tied together by a very simple rule: the python wrappers hold only one reference to the VTK object. Because of this simple rule, there are only two places in the Python wrapper code where we have to worry about VTK reference counts: 1) the point where VTK objects first enter Python and have a PyVTKObject constructed for them, and 2) the point where the PyVTKObject is destroyed by Python's GC system.

The above system would work perfectly except that, several years ago, I added a Python "dict" to PyVTKObject so that python users could add custom attributes to their VTK objects. I even made it possible to add new methods by subclassing VTK classes within Python. Hence, VTK objects can by extended with pure-python attributes and methods. The problem was that these attributes would only last for as long as the PyVTKObject existed. So if a PyVTKObject was e.g. stored in a vtkCollection and then retrieved, it would come back as a stripped object with all of its pythonic attributes removed because only the vtkObject and not the PyVTKObject would be stored. I would work around this by always using pythonic containers to store python-customized VTK objects.

I have finally fixed the python wrappers so that this unexpected stripping of pythonic attributes no longer occurs. Whenever a PyVTKObject is destroyed, its attributes are saved in a 'ghost' object. The ghost is simple, it is just a C struct that holds the pythonic attributes and a weak pointer to the VTK object. Every time a new ghost object is to be added to the list, the weak pointers are checked, and any ghosts for VTK objects that no longer exist are erased. This keeps the "graveyard" from growing continuously.

So now whenever a VTK object pointer enters Python, the graveyard (the list of ghosts) is checked to see if the object used to have special pythonic attributes. If so, then a PyVTKObject is constructed with these attributes. So to all appearances, the user will be getting the old object back again, but really, they are getting a resurrected ghost. I really don't think that I did a good job of explaining all of that. But since I'm just filling some quiet time during my vacation, it will just have to stand as it is.

Today I replaced the internals of VTK's weak pointer implementation. The weak pointers added an observer to their object so they could catch the DeleteEvent signal that every VTK object emits when it destructs. This made weak pointers very resource-intensive, and it also meant that a weak pointer could be left with an invalid pointer if someone called RemoveAllObservers() on the object. My new implementation adds an optional list of weak pointers to vtkObjectBase. This is very lightweight, and it means that observers and weak pointers will no longer get in each other's way.

Since this new implementation works with vtkObjectBase instead of vtkObject, it will also allow me to do a useful thing with the VTK Python wrappers. It used to be that when a python reference to VTK object went out of scope, the pythonic part of the VTK object would be destroyed, even if C++ references to the object still existed. So if the VTK object was later returned to Python, the pythonic part would have to be recreated and important parts of it like its python "dict" would be lost. With weak pointers, I can make an engram of the python object before it destructs, and then resurrect it when the VTK object returns to Python, or I can mark the engram for deletion when the VTK object destructs.

This was a fun little project. I wrote a C++ preprocessor for WrapVTK and will probably add it to VTK itself sometime this week. This preprocessor is specifically meant to be used with a wrapping system, it includes all header files and stores all macros but it does not expand them, instead it allows the wrappers to wrap them. Except for the lack of macro expansion, it is a full preprocessor. It can evaluate expressions and handle #if directives. It can provide an integer value for any macro that will evaluate to an integer after expansion.

The addition of a preprocessor to the VTK wrappers allows a bunch of "hocus pocus" to be removed from the wrapper code. No longer will inscrutable regular expressions be needed to block out odd code chunks that can't be wrapped. No longer will the wrapper-generators need to rely on #ifdefs to decide what types can be wrapped. Instead, the parser will be able to correctly follow the #ifs and #elses in the wrapped header files themselves. Even better, difficult header files like vtkType.h and vtkConfigure.h can be wrapped and all the constants that they define can be made available in Python. It's all good.