Raising the Visibility of GLOBAL, part 3

Last time we pulled the call to the SUBROUTINE of interest (SUBROUTINE ABC) behind a facade, SUBROUTINE ABC_FACADE. This time, we make the dependencies of SUBROUTINE ABC on the variables in GLOBAL explicit.

I'm going to rely heavily on the compiler, because it is better at tracking picky details than I am.

1. We remove the USE GLOBAL statement in SUBROUTINE ABC.
2. We insert an IMPLICIT NONE statement
3. We insert a USE TYPES statement, where TYPES is the MODULE that contains:

• the INTEGER parameters that specify the SELECTED_KIND for INTEGER and REAL variables
• The definitions of all of our derived types

At this point we compile. If we previously cleaned up all of the warnings, and explicitly declared all of the variables use in SUBROUTINE ABC, the only errors and warnings we should get come from variables that are defined in GLOBAL.

For every variable that shows up, we create a new argument to SUBROUTINE ABC, and specify its type. As they say, lather, rinse, repeat until the compile is clean.

Don't be surprised if you have an extremely appalling list of parameters (I've generated calls with over 35 arguments using this approach). The nasty dependence was there all along, you've just made it manifest.

Now for the really important step:

Examine GLOBAL and look up the type of every variable. If any of them are ALLOCATABLE and/or POINTERS, you must introduce an INTERFACE (with the signature and types of the variables in SUBROUTINE ABC). This interface has to be included in SUBROUTINE ABC_FACADE. Note that this is another advantage of introducing SUBROUTINE ABC_FACADE;we only have to have the INTERFACE (inline or via an INCLUDE) in SUBROUTINE ABC. All of the client code is blissfully unaware that anything has changed.

In the compilers I've used, failing to do this will not yield an error or even a warning. Everything will be fine until we get to the routine that calls ABC_FACADE. When the caller is called, a nasty and completely uninformative run-time error is generated. For example, using Silverfrost, the .NET run-time gives an "illegal program" error from the JIT compiler. The most heinous part of this is that some compilers won't blink an eye; everything will run just fine. But I've seen another version of the same compiler choke and die. Even if you don't have a problem, fix it. Or be prepared to have previously fine code suddenly go down in flames when you upgrade your compiler or send your "perfect code" to someone else (usually a boss, or someone at a funding agency).

Tidying Up

Next, for each of these parameters, take the time to specify INTENT as part of the declaration. Since we're already mucking around in these variables, it is a perfect time to go ahead and give the compiler the extra information. I've found that the INTENT specification is invaluable. Say I originally do this refactoring, specify INTENT, and get the compile clean. A week later I'm back, trying to restructure this routine, and I assign to a variable that is an argument (say re-using as a temporary variable within the routine). Bang! Nasty bug has appeared out of nowhere, and it's going to be a freaking bear to find. With INTENT(IN) specified, the compiler chokes immediately, showing me the error of my ways. It's not a fool-proof system (as a fool, I can be quite ingenious), but I'll take any help I can get.

So, we've simultaneously:

• Made all of the dependencies of SUBROUTINE ABC on GLOBAL explicit (our goal)
• Insulated the client code from the effects--it calls ABC_FACADE as if nothing has happened. A first, clear example of this is the case where we have to introduce an INTERFACE for

Now we are free to restructure SUBROUTINE ABC, secure in the knowledge that we aren't inadvertently creating side effects (or, more likely, bugs), by messing up a GLOBAL variable that we haven't recognized.

Raising the Visibility of GLOBAL, part 2: Extracting a Facade

Last time we pulled COMMON blocks into single module of their own, hereafter called GLOBAL.

If the code has any Fortran 90 in it, we are likely to find derived types as well. At the very least, we are likely to want to have our specifications of precision (the SELECTED_KIND) available. This is an excellent opportunity to make use of the DRY (Don't Repeat Yourself) principle--every source of information should have one, canonical place that it resides.

For both of these aspects, the next step is to pull the types into a separate file (typically called something intuitive like XXX_TYPES.for). This can be a MODULE, or a simple INCLUDE file. I prefer to use MODULEs; it reflects my intent better. I generally reserve INCLUDE files for interfaces. Separating the types (parameters, derived types, etc.) from the COMMON data allows me to make broad use of the types, without requiring me to drag around the baggage of the global data. GLOBAL then USEs TYPEs, ensuring that I haven't broken anything, and requiring no changes to any other code.

All of this has been preparing for the main step of pulling the routine we care about behind a Facade. To rehash: what I currently have is a SUBROUTINE ABC with a call with few (or no) parameters. BUT, there is a ton of back-channel communication going on via reads and writes of variables in the COMMON structures. I want to make the communication explicit. To do so I follow three steps:

1. Create a new subroutine with the name SUBROUTINE ABC_FACADE. In this SUBROUTINE, we USE the GLOBAL block. Beyond variable declarations, the only thing in the ABC_FACADE routine is call to ABC.
2. Change all calls to ABC to calls to ABC_FACADE. I let the compiler do the heavy lifting here. I change the name of ABC to ABCx, compile, and locate all the link errors. At each location, change ABC -> ABC_FACADE. No other modifications.
3. Change the name of ABCx back to ABC. Compile and run a quick test. No functionality should have been affected. At worst, we have created a slight degradation in performance, due to the intermediate SUBROUTINE call.

So what have we gained? The ability to make changes to the interface of SUBROUTINE ABC without affecting any client code, which we will indeed do, next time.

Baseline Steps

Today I want to talk about modifications we make before we begin explicitly refactoring.

1: Pull each routine into its own file.

Like all modifications, this has pluses and minuses. The major minuses are:

a) There are more routines to keep track of -- using version control, this shouldn't be an issue

b) Make files-- If your program had been a monolithic beast before you started working with it, you may have been able to avoid the special joy that is MAKE. As they say, it builds character....

The advantage here is locality. Having each routine in its own file lets us find something on line 127, instead of searching for line 12097. The more we work with the code, the more powerful it becomes to have a small, clear area in which to focus our attention.

2: Turn the compiler warnings to maximum, and clean up everything.

This is another pain point. When I first applied this to my most recent project I had well over 200 things to fix. Mostly these fall into one of 2 classes:

A) Variables that are declared but not used. These are completely benign. Delete each one and move on. No fuss, no muss.

B) Variables that are not properly initialized. This is the first class bugs that we encounter. Everything should be explicitly initialized. Yes, I know that it is possible to use compiler switches to have uninitialized variables set to 0. That works, until you switch compilers or change optimization settings, or .... And then you get gibberish results, usually the night before a conference presentation, so that you aren't sure whether any of the results are trustworthy.

The basic principle is that we want the code to convey _ALL_ of the information about what the routine does. Relying on compiler settings is another way letting important information slide out of the code.

But the real reason for getting to the compile clean is that when we change the code, a new warning will stand out in clear relief, rather than getting lost in a blizzard of old, non-information warnings. Having the ability to draw our attention to an unintended change, as close in time to when that change was made, is invaluable.

3: Insert IMPLICIT NONE at the beginning of every routine.

Again, the goal is to make the code communicate as much as possible. The implicit variable types made sense in the late 1950's. They don't today. Using IMPLICIT NONE forces us to clearly how we intend to use variables.

The biggest advantage of IMPLICIT NONE is that it it removes a class of errors. Under implicit typing, if we misspell a variable name, the compiler silently creates a new variable. IMPLICIT NONE lets the compiler help, just like a spell checked does for a document.

4: Use KIND to specify the precision you require.

The specification of precision is another way of making your intent clear. The whole point of KIND is to facilitate updating and porting code. Take advantage of it.

5: Add INTENT settings for every argument.

INTENT is another way of letting the code communicate. Every argument can be specified as read only, write only, or read/write. This is another point may seem like overkill, but as we start to restructure the code, the added information to the compiler will again help it draw our attention to the exact point at which we've change the code's behavior.

All of this may seem like busy work, and who needs that? But I've found that taking the time to lay a solid groundwork pays handsomely, because it give quick feedback when a change has an unintended consequence.

Setting the Stage

Today I want to three preparatory steps that are vital to actively working on code. My belief in these comes from many painful experiences.

Step 1: Install a version control system and use it.

THIS IS VITAL. About every two weeks I have a situation in version control this saves me major heartache. As the saying goes, "Learn From My Fail."

Presumably the code you are working on is of value or you wouldn't be wasting your time fixing it. In the process of re-working the code, we are going to apply a lot of small changes. Every so often, a change will have an effect that we didn't foresee. The feeling of terror when the compile is fine, but the program suddenly starts giving screwy results is one I don't wish on anyone. In the best case, you still have the editor open and can undo your way back to a correctly functioning version. If not, you are faced with two unpleasant choices; toss all of your work to date to go back to the original program, or try to debug your way from the current version to one that works.

Version control software gives you a third choice: check out the most recent version that did work. This may mean discarding an afternoon's work, but I've found that it is almost always better than trying to debug my way out of a hole.

There are several systems that are: 1) free, 2) easy to set up and use. Probably the big 4 these days are CVS, Subversion, git, and Mercurial. In a Linux/Unix environment, you probably have a couple of these options pre-installed. CVS and Subversion have the simplest day-to-day use for someone working alone; git and Mercurial shine in collaboration within a team. Which you use is less important than using something. All have nice online tutorials, and all have books dedicated to them (if you are a book person, cruise over to look at the books on the Pragmatic Programmer site for a couple of very readable introductions).

Step 2: Get a small problem that exercises most of the code.

Regression testing is a very weak form of validation. But it is absolutely vital to have some verification that we haven't broken the code. Unit tests are certainly preferable, but I often have to deal with a SUBROUTINE that is a "big ball of mud", several thousand lines with a cyclomatic complexity over 100. McCabe's original work suggests values of 5 or less, with the occasional routine allowed to be in the 6-10 range (provided that a special dispensation was obtained from a suitable religious official :-).

Writing a suite of unit tests for such a monster is beyond my tolerance for pain, and would effectively wind up being a regression test anyway. Taking a known problem with known good solution is my compromise between best practice of having good unit tests, and the unacceptable risk of working with no verification that I haven't broken anything.

You are going to be running problem this lots (probably a few times an hour). Anything that takes more than a couple of minutes is going to get annoying really quickly. Take time to pare the problem down to something that runs in under a minute. It is well worth the effort.

Step 3: Automate the verification of the test.

In the programs I work on, even a small example can generate several hundred pages of output. My enthusiasm for trying to eyeball compare two versions of this file is non-existent.

Automation is a must. The goal is something that says a terse "OK" if the file matches, and gives you come info if there is a problem. For simple files, the operating system's file comparison command is sufficient:

fc original_output new_output

This is the simplest thing and it works. If the files don't agree, though, the amount of output that scrolls by can make it bewildering to figure out the differences. You can pipe the output into another file:

fc original_output new_output > results

and then open results in an editor.

If you are using this lowest tech approach, I've found that the best next step is to fire up an editor that supports tabbed documents, and load both copies. Flipping back and forth will allow you to zoom in on the difference(s) you care about.

This approach fails miserably if you have time/date stamps in the output. The file comparison routine will flag every one of these as a "difference", although they are differences you couldn't care less about. We'll need a somewhat sharper tool to deal with this issue, and I'll discuss a dead-simple one I use in a future post.

Summing Up:

So, the preparatory steps are:

1) Install version control and import the original version of the code. This serves as a powerful "undo" for your changes.

2) Find a small problem that runs in under a minute. This will increase your confidence that you haven't broken anything.

3) Automate the comparison of old v. new output. This allow you to get quick, simple feedback on whether you have broken something. The sooner you know, the easier it is to fix.

Raising Visibility in GLOBAL, part 1

One of the key "features" of Fortran is the global, COMMON block. The major plus of the COMMON blocks is that they allow us to write SUBROUTINEs and FUNCTIONs with relatively short lists of arguments. All of the detailed information passes back-channel in the COMMON block.

This is also the major down-side of using COMMON blocks. It is devilishly hard to figure out where variables get set to certain values. In "Working Effectively with Legacy Code", Michael Feathers describes a similar problem as "Sensing--We break dependencies to sense when we can't access values or code computes." Here, Feathers is referring to things we actually can't see, like whether a socket was correctly opened.

To make this work, we need to apply three preparatory refactorings. These are small, but depending upon the

The problem of COMMON blocks is related, the problem of Visibility. We can't see the flows of information, making it impossible to reason about the code. Today I want to describe three techniques that I use to deal with problems of visibility.

1) Pull COMMON blocks into a separate MODULE. Once this is done, delete the COMMON statement(s) from each of the effected SUBROUTINEs/FUNCTIONs, one at a time, and insert a USING statement to bring in the MODULE. Next, using the compiler, verify that the COMMON is actually needed. If you did the earlier refactorings of:

• Introducing IMPLICIT NONE into each routine
• Cleaning up all warnings

Now this bears fruit. Removing a COMMON block will yield a set of compiler errors that tell you exactly which variables resided in that COMMON.

At this point, you will probably find out that the COMMON statements you had were, indeed, needed. Not to worry; the information we gather here will be put to good use in later steps.

The other difficulty may be name collisions across the COMMON blocks. Fixing this may seem like more busy work. I would argue that, given the nature of COMMON blocks, naming different variables the same thing is likely to cause (or reflect) confusion in what concepts the separate variables are meant to represent. Flushing that out is part of being able to work meaningfully with the code.

Another approach can be more useful, depending on the exact circumstances. Pull each COMMON into its own MODULE, and include it only where it is needed. This is most useful when we have a few COMMON blocks that are essentially independent of each other. I list this as a secondary strategy not because it is less useful. it is more useful. In my experience, finding code with relatively independent sets of variables is uncommon (pardon the pun). The reasons are two-fold:

• Older Fortran compiler design was generally based on the idea of memory overlays. In the tight memory configurations of the time, routines would be pulled out and replaced in the same location by the next routine. This swap could include COMMON blocks that were not used by the new routine. By putting all variables in a single COMMON block, we guarantee that there memory will not be swapped and possibly corrupted. Modern OS design is generally stack-based (indeed the language standard is moving in that direction), making this less of an issue.
• The second, less common reason is that separate blocks generally flows out of a mindset of strongly decomposing the variables into pieces associated with certain routines. This line of thinking is much more object-oriented than was typical of early Fortran.

2) Pull your user-defined types into a separate file, named something like XXX_TYPES.inc. This is a perfect place to stash your SELECTED_KIND definitions that you are using for KIND specifications. Including this file will be less intrusive than specifying the SELECTED_KIND in each and every file. More importantly, it will guarantee that the types specified are coherent and agree across routines.

Overall, today seems like a big step backward. We've taken individual COMMON blocks and essentially created global data out of them.

The Beginning

For better or for worse, I've come to spend most of my time modernizing legacy Fortran programs.

This site is dedicated to capturing the ideas and techniques that I've discovered in the process. I will catalog the re-factorings that I routinely use, and ways to bring things under test so that I can transition them to more modern languages, especially Java and/or C#.