Metadata driven development, the Holy Grail of software development.
December 19, 2011 Leave a comment
In life, there are thinkers, and there are doers.
I recently had the privilege of having a discussion about “the holy grail of software development” with a colleague of mine, who I consider to be one of the greatest thinkers I have ever met. Unfortunately, after the conversation, I was left with a weird taste, which I later explained by the fact that even the greatest among the thinkers, can only think I terms of what they already know. Of course, the more bits and pieces of information they know, the more combinations they can make, and the more puzzles they solve. But still, it is extremely hard, if not nearly impossible, for anyone to think beyond the limits of one’s current knowledge.
Similarly, when people see LightSwitch for the first time, they tend to label it with labels that they already own. They compare it to what they already used before, or read about. They try to fit it in known boxes, and unfortunately, for most software developers, that’s either the box labeled “software made by expert software engineers”, or the one leaning more towards “rapid application development”. LightSwitch however, does not fit in the first box, but surprisingly, does not fit in the RAD box as well. LightSwitch fits in a third box, called “metadata driven development”, which unfortunately is a box unknown to most…
In this article, I’d like to take the time to explain my vision on metadata driven development, and even if you don’t consider it to be the “Holy Grail” of software development (which I personally do), simply learning about a this ideology will offer anyone the opportunity to make more combinations, and solve more puzzles…
Traditional software development vs metadata driven development (MDDD).
Think about your current projects. Suppose you have to expand the software product you are currently working on to support your customer’s need to be able to manage “people”. From your customer’s needs, each person has a last name, first name, phone number and an email address to be managed. When you are asked to develop this, some of you will take the WET approach (“we enjoy typing”) and actually type out this classes from scratch. Others, who use RAD tools, will design the entities in a more graphical environment, which in turn will generate these classes for you. Maybe you will have combined the best of both worlds, and start by partially coding some classes and have some tools (like the powershell commands in the mvc scaffolding nuget packages) generate the rest. Whatever road you took, the end result of most of us will almost always be a combination of classes in the trend of “PeopleProcessService”, “PeopleRepository”, “PeopleManagementView”, … Almost always, you will rely on some frameworks (sometimes developed in-house), offering some common abstractions, such as a “ManagementScreen”, or even simply an “IScreen”, for your “PeopleManagementView”.
Excluding testing and deployment, there’s three (well: two and a half) important components to consider in MDDD, steps on the road to take, phases of development, before the software can run and “people can be managed”. Each of these components contains one single version of the truth, following the DRY principle perfectly.
- The heart of an MDD application, contains the single version of the truth which is your business logic. To help you visualize this, imagine an XML file which contains metadata, describing everything from entities, their corresponding business logic and validation rules, users and how different user levels offer different possibilities or actions to perform, to a rough layout of your screens and a description of the user processes, wizards, steps to take, … It doesn’t stop there, a true MDD application in his metadata, will include limited documentation, input for different test cases, or even partially describe the desired architecture of the application, stating whether CQRS should be used for some entities or not, whether some entities should be eagerly cached, lazily cached, or loaded on demand, … It goes without saying that this metadata needs very good tooling to be managed, preferably with different levels. Project managers and support staff should be able to tweak or configure the business logic (such as: the minimum and maximum limits on one account transaction, …) in tools they are comfortable using, software engineers (doers) will design the entities, screens, … and architects (or the thinkers among the software engineers in a smaller company) will tweak the architectural parts.
- Next component, is the compile time process. During this process, a very limited amount of C# specific classes (or of course, the programming language of your choice), will be generated to serve the single purpose of supporting the human flaws in the former and the latter phase. Without proper tooling, it’s hard to read and understand metadata, and even harder to effectively debug it. To enable the framework team (more about this later) to do their job properly, the metadata needs to be translated into something they can understand, in our example, a “person” class. Another flaw is that no metadata language will ever be able to express every business rule, and sometimes some additional functional programming will still have to be done, which again creates the need for POCO (plain old C# objects) for the functional developers to work against.
- The final phase to consider, is the runtime phase. Here, a carefully designed framework loads up both the metadata and the generated classes, and brings them together in an application. This phase contains the single version of the truth which are your technological specific choices, combined into the user experience you want to offer the end users.
- The first phase is technology independent. It’s written in its own metadata language.
- The second phase is only dependent on the development language of choice. (Very likely to be C# if you’re a reader of my blog.)
- The last phase is completely technology dependent.
MDDD has a lot of similarities with whatever road you took as a traditional software engineer, to develop the “people” functionality, but the end result will be different.
The benefits of metadata driven development.
- More fun for software engineers. There’s a good chance, you as a software engineer are not working on a anti missile satellite shield operating system, or working out a new physics engine for Battlefield 4. More likely, you are working on a payroll application, maintaining a huge CRM, or developing a real estate management suite. Writing functional software is boring, because most software engineers are thinkers, and not doers. They have little interest in the actual business they write software for, but prefer thinking and brainstorming about user experience, tweaking performance, discussing architectural practices and studying design patterns. MDDD allows for a greater part of the team to work on these things they find exciting, and not on functional programming. It probably requires quite some organisational changes in your company, but i suspect this will be worth it in the end. (I now find it interesting to see how one of Michael Washington’s firsts posts about LightSwitch also envisions that LightSwitch could bring a serious organisational change…)
- Faster time to market. Granted, MDDD requires more of an initial investment in the teams that create and maintain the metadata language and the teams that create the technology specific frameworks to handle the metadata at runtime, but the effective amount of functional programming done, is ridiculously small, where as this is exactly what our customers pay for. Same functionality, developed in less time equals happier customers and more profit.
- Faster time to market, again. In traditional software development, “if you can express it, you can develop it”. This means, if the salesperson has an agreement with the customer, a functional analyst can write it in a document, an architect can design it, and a software engineer can code it. Another reason why MDDD speeds up development (and thus lowers costs), is that both functional developers, salesmen, project managers, … work on the same metadata, just on different levels of detail. ”If you can express it, you developed it”.
- Lower cost of change. For both functional as technological choices, there is only one version of every piece of truth. Compare these requests to change something, to how long it would take you in your current project:
- The maximum length of a person’s last name should be 150, instead of 120 characters long. This should be changed in the database, should be checked on the server-side, and visible to the end-user by limiting the amount of characters in each textbox where the last name is shown. In MDDD, this means changing one element in your metadata.
- Your UX responsible decided that each button ( / command) should have a tooltip, which you didn’t implement. Your current software suite has 5000+ buttons. In MDDD,this means changing the one place where the decision is made to represent commands as a button, and creating a button that has a tooltip instead.
- Because of the sheer amount of users, the servers of some of your applications are lagging and your team decides all server calls should be done asynchronously from now on. In MDDD, this means making one change in the one place where is decided on how a command is routed to a specific server call.
- Your QA team wants to create automated (coded) UI tests, and asks you to add a dependency property to each of your links, buttons and rows. In MDDD, there are three places in the framework that will require a small change: where links, buttons and rows respectively are mapped to the metadata.
- Management would like fully working WP7 aswell as a Windows 8 version, and an HTML5 website written in ASP.NET MVC3, out of the 30+ WPF applications you have previously created, to remain competitive in an rapidly moving forward market. In MDDD, this is still a huge change, because your company will need to set up 3 new frameworks (or framework teams). However, after this investment, not only can users choose to run the WPF desktop, WP7, Win7 version or browse the web app, but they can do so for all 30+ previously created applications as well.
How LightSwitch fits in.
- Is MDDD your idea, or are you just describing LightSwitch? MDDD is not my idea, I just combined answers I already had with answers LightSwitch offers to some common development issues, with some unanswered questions. The colleague mentioned in the intro, told me later that there was a name for this way of software development: MDDD, which has apparently been around for quite some years aswell, but to my knowledge, never been as closely approached as the LightSwitch team has accomplished.
- What’s wrong with the “PeopleManagementView”, ie. describing how the screen should look in a language that was intended to do just that, Silverlight XAML for example? Describing how a screen should look and feel, should be done in the metadata language, so it can survive the technological choice of that moment. If it’s not in the metadata, your metadata is not reusable and a substantially large amount of work will have to be redone with every change across your application, be it some kind of UX thing or the entire technology used to render your UI. If it is in your metadata, but that metadata is used to generate a “PeopleManagementView” (the RAD way), then I consider this to be a duplication. Two versions of the truth to maintain, once again reducing the flexibility of MDDD. Using the metadata at runtime is exactly what defines MDDD, and makes it so powerful.
- How does MDDD compare to the SOLID design principles, architectural patterns such as SQRS, … ? I fail to see how MDDD opposes these good practises and well thought patterns. MDDD is more of an ideology, a way of making software. Both the limited amount of generated classes that serve as a translation of the metadata, as the framework that works with the metadata at runtime, as the tooling used to maintain the metadata, should embrace these design principles, patterns and practices.