OS/2 Documentation Generation

Open Watcom provides development tools for several host environments. Using Open Watcom one can develop programs on DOS, Windows (even the old Windows v3.1), OS/2, and Linux. To do this effectively, however, one needs to be able to read the Open Watcom provided documentation on each of these platforms as well. That documentation contains information about the Open Watcom tools and run time libraries. It is an essential reference for the Open Watcom user.

Internally the documentation is in a format known as WGML, a text based markup language developed originally at Waterloo University. Through a series of steps the WGML source is transformed to a platform specific format that can be read by a platform specific documentation viewer. This process allows us to use the same documentation source for all platforms which is obviously highly desirable.

The final step of the documentation tool chain is a "help compiler" that outputs the necessary platform specific format. For Windows we are using the free help compiler created by Microsoft for this purpose (it is a separate download, we do not distribute this tool with the Open Watcom source). Similarly on OS/2 we have been using the IBM help compiler ipfc to produce the OS/2 version of the documentation. Unfortunately ipfc has been something of a thorn in our side. Recent versions of the tool have a bug that causes it to crash when processing certain Open Watcom documents. However, older versions of the tool are unable to process other of the Open Watcom documents. Thus building the OS/2 documentation required an elaborate dance.

1. Build as many documents as possible with the new version of ipfc until the tool crashes.

2. Manually build the troublesome document with the old version of ipfc.

3. Build the remaining documents with the new version of ipfc.

What's more during the most recent release cycle I noticed other problems with the OS/2 documentation build that seemed to be related to incompatibilities between our latest documents and IBM's ipfc. Finally, because ipfc is only available on OS/2 it is necessary to use OS/2 to build the OS/2 help files. This is somewhat awkward for me since I can't build the entire system on the Windows machines that I typically use.

However, early in the 1.9 development cycle Lawrence Haynes surprised the rest of us by contributing a clean room reimplementation of ipfc, called wipfc to the Open Watcom project. Wow! Now we have our own OS/2 help compiler and are in control of its source code. We can compile wpifc for all our platforms so now we can build OS/2 help on Windows, Linux, and even DOS (as well as on OS/2, of course). Furthermore we can address the issues that have plagued ipfc in the past to further simplify the build process and, hopefully, provide better documents in the long run. In particular, wipfc can compile all the Open Watcom documents without crashing. That's a step forward.

The wipfc tool is not yet 100% correct, however. Although it can compile the entire documentation set, the documents produced have some (mostly minor) problems. It is my hope that these final issues can be worked out before the v1.9 release. A more interesting question, perhaps, is this: should we include wipfc in future Open Watcom distributions? The OS/2 community might find it useful to have access to an open source version of the tool that is actively maintained as part of an open source project. For that matter one might wonder if we should distribute the entire WGML tool chain to enable third parties to use WGML for their own documentation. That is, perhaps, a good topic for a later post.

By the way, as far as I know we do not have a documentation compiler or viewer for Linux (unless the DOS tools happens to work when compiled as a Linux applications... probably not right now). This is one aspect of Open Watcom's Linux support that needs to be improved before the status of the Linux host becomes non-"experimental."

I should also mention that the Open Watcom build system can create PDF versions of the documentation using the third party Ghostscript tool. This is done on the build servers; fresh PDF documents can be downloaded from them and PDF documents for recently released versions of Open Watcom can be found on the main Open Watcom web site.

Open Watcom 1.8 Released!

After an extended release candidate cycle, the next version of Open Watcom is finally ready! The process of producing v1.8 took much longer than I expected and probably longer than necessary. What have I learned from this process? Several things...

• We should release more often so that the amount of change in each release is less. This will make digesting those changes a little easier and hopefully reduce the complications that arise during the release process. It also has the very significant benefit of getting fixes and new features into the hands of the user community more quickly.

• The release process needs to be streamlined. Right now there are many labor intensive steps that are being done manually. Although some steps require a significant amount of human judgment and may be difficult to automate, others could certainly be handled by (or helped by) appropriate scripting. If the release process was more automated it would be easier to make releases more often.

• The release branch should get regular exercise. In the past I've mostly ignored the release branch in our Perforce depot until it was time to start making release candidates. I think it may be helpful to instead periodically branch the trunk over to the release branch and verify that there are no unforeseen problems (for example with merging changes or with building/testing the potential release). I imagine that issues will be easier to fix if they are discovered early and well before an actual release needs to be made.

• There should be better coordination between my building of the release and contributor activities. I need to communicate better with the contributor community to make sure that planned changes are either completely in place or have not yet started before officially branching the release for the first time. It probably makes sense to have a feature freeze for a few weeks before the release and shift development activity to testing and cleanup during that time. Because of the small number of Open Watcom contributors we have not bothered to do this in the past, but I think such an approach help in the long run.

In the interests of making faster releases, I've decided to try and get version 1.9 out the door in six months or so, regardless of how much or how little is done during that time. This suggests a feature freeze some time in July, release candidates some time in August, and a final release by (say) September 1. Of course this schedule is arbitrary and subject to change, but I think sticking to something approximately like it would be healthy.

Open Watcom 1.8 Release Candidate 4

Open Watcom 1.8 Release Candidate 4 is ready for evaluation by the community. One of the new features in this upcoming release is the offering of installers for DOS and Linux systems (as well, of course, as the usual installers for Windows and OS/2 systems). The DOS and Linux installers have been classified as experimental until now. Consequently there have been quite a few quirks in them that needed fixing. This release candidate addresses several of those quirks. In addition, a few minor fixes to some header files have also been included.

Unless something significant accidentally got broken between RC3 and RC4, it is likely that RC4 will end up being the final release. We should know in a few days! The new release candidate can be downloaded from the usual places.

Open Watcom 1.8 Release Candidate 3

Open Watcom 1.8 Release Candidate 3 is ready for evaluation by the community. There are still a few issues identified in the earlier release candidates that have not yet been (intentionally) addressed in this latest one. However, I wanted to get this release candidate into the hands of the community sooner rather than later so that the fixes that have been applied so far can be evaluated.

Although there is a good chance that there will be an RC4, I believe the system is converging to a final release. Most of the problems identified at this point are fairly obscure and shouldn't affect most users. The new release candidate can be downloaded from the usual places.

Open Watcom 1.8 Release Candidate 2

Open Watcom 1.8 Release Candidate 2 is ready for evaluation by the community. This release candidate fixes almost all of the issues identified in RC1. Several issues related to the new Win32 API headers and libraries have been fixed. The new release candidate can be downloaded from the usual places.

Open Watcom + LLVM?

The Open Watcom project has respectable code generators for 16 bit and 32 bit x86 architecture processors. The project also has some experimental code generators for certain other architectures (for example, Alpha and MIPS) but they have never been sufficiently completed to warrant distribution.

One could argue that the x86 is in decline and that the future of that architecture is bleak. For example, many new desktop and server machines are using x86_64 architecture so supporting 64 bit code generation for that architecture would seem like a priority. Of course there are many other architectures worth supporting as well... both large and small. Also who knows what the future of processor design will bring. Considering the limited resources of the Open Watcom project, it seems unlikely that Open Watcom could ever hope to support more than a tiny subset of potentially interesting targets.

One possible solution would be to support some kind of virtual target and then use the tools provided by that virtual target to do final code generation onto whatever real targets the virtual target supports. In other words, what is needed is something like LLVM.

I am no LLVM expert, but as I understand it LLVM defines an intermediate language with a level of abstraction that is similar to assembly language. This intermediate language can be expressed in textual form (just like assembly language) or in a binary "bitcode" form. The LLVM toolkit then provides tools to link bitcode modules together and to do code generation to one of several supported target architectures. There are several nice things about this from Open Watcom's point of view.

• LLVM is a very active project and it is likely that new targets will be added to it regularly as time goes on.

• LLVM can do some advanced "link time" optimizations on the bitcode files.

• LLVM provides various back end tools that operate on bitcode files, such as a library manager, a profiler, an assembler, and a debugger (I believe).

To tap into LLVM, what needs to happen is for Open Watcom to grow a code generator that takes the internal intermediate language used by the Open Watcom compilers and translates it into LLVM's IR ("intermediate representation"). At that point the tool chain would switch over the LLVM's tools for linking, and final code generation, etc. This would allow the Open Watcom compilers (and front end tools like WMAKE, the IDE, etc) to take advantage of the rich collection of targets and advanced features provided by LLVM.

Although the basic idea sounds simple, making this a reality would require a significant amount of work. For one thing, to do it nicely it would be desirable to compile the LLVM back end tools with Open Watcom. Alas, the LLVM compilation system is very Unix-centric; getting it to build with Open Watcom tools would no doubt be a challenge. Also the LLVM documentation warns that LLVM's source code "stresses" C++ compilers. I suspect compiling LLVM might require applying further fixes to Open Watcom C++ (which would be a good thing for its own sake). Of course getting the Open Watcom C and C++ runtime libraries to compile for LLVM might also be an involved process.

Ultimately I'd love to see an LLVM target, perhaps with subtargets for each supported processor architecture, offered to users of the Open Watcom installer. Selecting the LLVM target would install LLVM back end tools and generally enable LLVM based development in an otherwise Open Watcom environment.

To explore these ideas further, and just to get to know LLVM a little better, I downloaded LLVM v2.4 and successfully compiled the back end tools on my OpenSUSE 10.2 machine. My thought was to study the system and learn my way around its assembly language. At that point I might be ready to think about building an OW->LLVM code generator. I could rely on the gcc-compiled LLVM back end tools at first. That would allow me to experiment with an OW/LLVM combination without having to immediately fuss with getting LLVM to compile with Open Watcom.

While I'm on the subject of virtual targets, I am lead to wonder if it makes sense to chart out a similar plan for targeting the intermediate language used by the CLR (aka .NET). Hmm...

Open Watcom 1.8 Release Candidate 1

After a rather long delay, Open Watcom 1.8 Release Candidate 1 is finally ready for public evaluation. It can be downloaded from the usual locations. I am pleased to say that the issues I discussed in my previous entry, as well as some other issues that were discovered since then, have all been fixed in this release candidate.

The path to RC1 was not as smooth as I might have liked. It took a lot longer to produce than I expected. One problem was that I am prone to being a perfectionist and I hesitate to release something unless everything about it is just right. Of course perfection is not a realistic goal especially considering the size of the Open Watcom project and the resources the contributors have to spend on it. Setting aside those unrealistic ideals help me finally move things along.

However, I think the real issue was much more mundane. I simply attempted to branch 1.8 prematurely considering the magnitude of the changes in this release. I was just too anxious to get RC1 out the door. A longish list of problems (meaning regressions relative to 1.7) appeared shortly after I made my initial branch, necessitating a longish list of fixes. After a while it became clear that just branching the entire project again was the easiest and most reliable way to ensure that all the necessary fixes eventually got into 1.8. That seems like a sure sign that the initial branching was too early.

At this point 1.8 appears reasonably stable. While I'm sure new problems will be found with the release candidate (after all, that's what release candidates are for), I do at least feel that RC1 is a true release candidate in every sense of the word. It's not just a development snapshot. It's not just a beta. It's the real deal.

I encourage everyone in the Open Watcom user community to download 1.8RC1 and try it out. As I've said before, this is a significant release with big changes. It will need all the exercise it can get!