Domain-specific languages, software factories, and craft

[Note:  if you're not a developer, you probably won't be interested in this one.  Feel free to read on, of course.]

    I've been doing some reading and thinking about domain-specific languages (DSL) and software factories (SF) lately.  There's more to learn, since I'm definitely new with the material, but I do want to take a shot and get some of my current state of mind on paper magnetic storage.

    There traditionally have been two perspectives on writing code.  One says that it's all art, it's all fundamentally a creative exercise, and so the way to improve code is to improve craft; in other words, become a better developer.

    Another perspective says that programming needs to become more like construction or physical engineering: we need to get repeatable processes, with robust tools that serve us in those repeatable processes, and with repeatable results.  I mean, we can draw up blueprints for buildings of arbitrary size, and estimate the cost and effort of building them down to a fairly small variance.  You can't say that about software development... not even close.  Furthermore, relying on craft just doesn't scale; the world needs more developers, and it needs really good ones, and that's a lot to ask for.

    And on the side of the second perspective come SF's and DSL's.  I mean, the F in SF stands for "factories"... that sounds pretty repeatable and process-driven to me.

    I come down on the side of the first perspective... code is art, to me, and craft and skill separate good developers from bad developers, just as they separate good and bad painters and good and bad cabinetry.

    I've been trying to work out why I feel that way, and what my hesitation is about SF's.

    Before I start that, let me simply observe that it seems to me that a DSL is in the eye of the beholder.  The first question I have to ask is: which domain are you referring to?  And secondly:  is that domain vertical (i.e. code for manufacturing tasks) or horizontal (i.e. writing efficient database code that can be used in any industry)?  Some would say that SQL is a domain-specific language... I'm not sure I agree, but it depends on your definition of "domain," doesn't it?  If we place languages on a continuum from general purpose <--> domain-specific, well, some clearly will be general purpose (C++), some clearly will be domain-specific (CodeSmith's template language) and some fall in the middle.  Or, where they fall on that continuum depends on whether your domain is broad or narrow.  Or, where they fall depends on whether your domain is vertical or horizontal.  So, the bottom line is: I'm not sure what the definition of "DSL" is, until I know which "D" you're referring to.

    So, back to software factories.  My fundamental problem with the analogy to physical engineering and building processes has to do with the fundamental differences between programming and building.

    Buildings:

• start with complete specifications in excruciating detail from multiple views; i.e. external view, electrical, plumbing, foundation, floor plans, etc.
• are supposed to be designed to stand the test of time with little to no modification
• are built with commodity parts (except for artisanal decorative touches)

    Programs:

• never start with complete specifications, even though they're supposed to
• should be explicitly designed to be easily modifiable
• have the devil in the details, and generally have few "commodity" parts

    When I build a house, I can hand the plans for my bathroom to a competent contractor and expect him to be able to assemble a team and collect the commodity parts that make up the bathroom (sink, cabinetry, bathtub, shower rod, PVC pipes, valves, toilet, tile, humidity-resistant paint, wiring, lighting, electrical outlets, etc.) and assemble them to specification.  Why do I have this expectation?

    Two reasons come to mind.  First, every place that the pieces meet (interfaces) are built to an industry standard.  The valve will match up with the pipe it attaches to.  The toilet will match up with the pipes it's connected to.  The outlets will match up to the wiring.  Second, there is no customization required to use the lowest-complexity pieces in the system.  All the contractor has to do is buy the pieces, and assemble them.  Unless there's something fancy going on, the valves or light switches don't require any modification by the installer.  The bathtub doesn't need to be changed or specially configured.  It just is what it is.  Hell, the very fact that I can use the word "installer" implies that the work is not one of modification but of assembly and placement.  (And I in no way mean to imply that this kind of work isn't creative and interesting... just that the end pieces are what they are.)

    The plan should be fully done before beginning... if it's not, the process comes to significant rework and cost overruns, and so the construction industry has gotten in the habit of having full plans before they start.  I also expect that once the bathroom is done, that I won't have to touch the construction of it for the life of the house, unless I choose to.  It had all better be put in place and just work.

    Let's compare all of that to programming.

    I never start with a complete spec, unless I'm NASA.  We take a shot at it, we give it a go, but the specifications for writing software never capture all of the final details in sufficient granularity.  I wish it were otherwise, but anything non-trivial has too much detail to be fully captured until you get in there and start working.  And that doesn't count on-the-fly changes from the business.

    Second, programs have to be designed to be as flexible as possible in allowing future modifications.  In fact, the more flexible the program, the better it will stand the test of time in terms of changing to meet changes within the business.  The interface complexities introduced in this kind of flexibility simply take years to understand how to do with skill.  An inexperienced developer simply can't do this kind of architecture well.

    Third, the least-complex pieces of the system, the individual functions and methods, frequently must be hand-coded.  There's no "function store" you can go to in order to purchase your end-pieces.  Also, once you get down to that level of detail in code, you sometimes find very difficult problems based on working with existing systems, which requires some clever work to get through.

    And so the analogy breaks down for me, and I can't see myself getting passionate about SF's and the DSL's that allow them to function.

    With all of that said, I'm all in favor of generating as much code in a solution as possible, and for having frameworks in solutions that simplify complex interactions.  I think every developer should learn the DSL within CodeSmith to help them create templates from which as much code as feasible within their solutions can be generated.  Generated code is bug-free code, and in that I agree with backers of SF.

    Perhaps the chasm between the two camps will be bridged with functional programming... what little I understand of the way functional languages allow for composability through monoids and monads might lead to a useful answer, and for that reason, I'm very much interested in getting familiar with F# and Haskell.

    But, for now, I just don't see how we get to engineering-level repeatability with programming... I just don't.  And that's why I'm so passionate about refactoring, and agile development, and code generation.  These are the tools that demonstrate a level of skill with development that will be required to continue to build increasingly complex systems.

    I reserve the right to change my mind about any/all of this as I learn more... :-)