I practiced model-driven development before the industry renamed a slice of it low-code. At that time the field had many schools, Domain-Specific Modeling (DSM) and Model-Driven Architecture (MDA) being the most common; on the technology side there was even more variety. Having done PhD research at the intersection of modeling and simulation, MDA, and DSM, and worked with the major technologies, I saw one commonality in the design of a modeling method and a modeling language. That commonality, known loosely under many names and here referred to as metamodeling (MM), captures a part of domain knowledge useful for modeling: it constrains incorrect constructs and expresses the wanted variability in generated artifacts. Its disadvantage was the investment required up front: a metamodel and the infrastructure around it had to exist before any value could be gained. Once low-code platforms like Mendix and OutSystems became mainstream, they absorbed metamodeling by providing a good-enough method and modeling language out of the box; in my case its application remained limited to practicing domain-driven design in Mendix.
Now, with the coming of AI and much attention shifting towards using domain knowledge (context engineering, feeding agents, RAG), the application of metamodeling becomes relevant again. A natural question arises, worth answering with data rather than opinion: what is the measured effect of structured domain knowledge on generated output? This would also be an opportunity to brush up the metamodeling skill. The result? I went into the experiment expecting the metamodel to win cleanly over the no-metamodel condition. It won categorically, but not cleanly: the output was large, took much longer to review, and contained errors where I did not expect them.
Headline numbers
The setup, in brief: the same general-purpose LLM decomposed twenty business epics into user stories under two conditions — the business description alone (no MM), or with the full architecture metamodel added (with MM). Three runs were executed per condition, every run a fresh conversation, every output scored against a control decomposition fixed before any run was made. The numbers the experiment produced:
| Control (volume reference) | No MM (A) | With MM (B) | |
|---|---|---|---|
| Domain-knowledge compliance checks fired / failed | 355 / — | 105 / 42–51 | 440–515 / 82–94 |
| Deliverables matched, of 55 | 100% | 2–4% (53–54 missing, 19–20 extra) | 55–62% (21–25 missing, 38–47 extra) |
| Flags caught, of 6 | 100% | 0% | 0–33% |
Table 1. Headline numbers per run, raw as scored, before any review and adjustment; a range is the spread over three runs.
Note that all runs are checked for compliance with the same domain knowledge, whether or not the run saw the metamodel. The control column is the expert’s decomposition, shown for scale. “Extra” means produced, with no counterpart in the expert’s decomposition. Keep in mind that failed counts are absolute; their volume is the focus of the table, and specific error rates are out of scope for this article.
The three no-metamodel runs (condition A, runs Ar1–Ar3) performed as expected, directionally (they had no metamodel, after all): they fired 105 compliance checks against the control’s 355 and matched almost none of the expert’s deliverables. The metamodel runs (condition B, runs Br1–Br3) were the surprise: they fired more compliance checks than the control itself, 440–515, and failed 82–94 of them; they matched barely more than half of the expert’s deliverables while producing 38–47 extras; and of the six flags expected they caught at most two, and in one run none.
It felt as if the metamodel had not done its work. Whether that feeling survives the analysis is what the articles that follow are about.
Knowledge domain and metamodel
I chose not to invent a new domain but to reuse a familiar, recognizable one. To reduce the chance that the resulting metamodel as a whole exists in the model’s training data, I combined the domain with policies specific to a fictive organization (introduced below). The LLM setup, described later, reduces the chance further. The resulting metamodel captures architectural decomposition: splitting business epics into user stories whose boundaries respect system ownership, architectural constraints, and a number of policies. It is this mix that makes the domain, and the metamodel built on it, stand out from generic data.
Figure 1. The architecture metamodel on one page, abridged: every section, sample items, counts stated. Derived from the text the runs received.
The metamodel itself is a structured document in plain text; the runs received the text, not the picture.
The business case and the materials
The fictive business context, as given to the model:
Stichting Thuisfonds is a Dutch charitable foundation that provides one-time financial gifts (grants) to citizens facing unexpected essential home repair costs. Examples include a leaky roof, broken heating, or structural damage that the citizen cannot afford to fix independently.
Users are Dutch citizens who apply for a grant through the Portal App, and Thuisfonds staff (assessors and regional managers) who review and process applications internally.
The process is simple: a citizen submits an application with supporting documents; an assessor reviews it; if more information is needed, the citizen is asked, resubmits, and the assessor resumes the review; if approved, a grant is created, payment is initiated, and the citizen receives a decision letter. Rejection was left out of scope, literally, to keep the experiment feasible. For the experiment I wrote 20 business epics covering this process end to end, in business language, metamodel-free. They range from simple (one system, one deliverable) to multi-system and event-driven. Six are deliberately problematic — for example, a naive performance shortcut or a manager’s report in the wrong place. They were planted so that flagging could be measured at all: five violate a metamodel rule and one duplicates another epic. Those six are the flags in the headline numbers. By the metamodel’s own instructions a decomposition should flag them; for the no-metamodel runs (A) this row is a zero baseline, needed for contrast, and nothing was expected of them there.
The experiment
Procedure
Two conditions, A and B, three runs each. Every run received the same materials in the same order: the task instructions (split epics into stories; identify gaps, overlaps and contradictions, and sequencing dependencies), the business context above, and the 20 epics. The B runs received one more input: the full metamodel text, placed between the business context and the epics. Nothing else differed, in prompts or model settings. The two conditions are deliberate extremes of each other, 0% and 100% of the architectural knowledge, chosen to make the effect visible; real practice sits somewhere between. Within a condition, every run was a fresh, isolated conversation with the same model and the same input; no run knew of another.
Model setup
Three settings, identical for both conditions, were chosen so that the metamodel would be the primary source of architectural knowledge in B and a real blind spot in A. These are design choices, made on purpose; the limitations section says what they leave untested. First, web search off: A could otherwise look up generic architecture knowledge and narrow the gap. Second, Instant mode instead of extended thinking: extended reasoning could partially reconstruct architectural knowledge from training data in A. Last but not least, a capable general-purpose model rather than a specialist, GPT-4.5: a model with strong prior knowledge of layered architectures could reconstruct part of the metamodel unaided.
The control
The foundation of the experiment is that the ground truth was built in advance: a control decomposition of the same 20 epics, created by me as domain expert and reviewed by an AI loaded with the metamodel (AI questioned and pushed back, and I decided), and frozen before any run was made. The metamodel and the control are siblings, not parent and child: both derive from the same domain knowledge. The metamodel is a selection from it, filtered for purpose; the control was cross-checked against it. Every run was scored against the control, in the metamodel’s vocabulary. This is what makes the results measurements rather than impressions, small as the experiment is. Note that the reviewing AI was Claude, Opus at the time and later Fable: a different model family than the one under test, and not itself under test.
What was scored
Every run’s output was asked two questions, and each has its own measure. First: did the run produce what the control contains? That is agreement with the control: which of its 55 deliverables a run is missing, which of the run’s stories have no counterpart there (the extras), and whether the six flags were raised. Second: is what the run produced permitted by the metamodel? That is compliance: for each story, does it name a system that may own it, a deliverable type it may be, an operation, an entity or event, and a realization the metamodel recognizes. Compliance checks fire only on what was produced, independently of whether it agrees with the control, so the two answers never imply each other. In other words, an output can pass every compliance check and still be invented. In the headline table, the first row is compliance; the second and third are agreement.
Figure 2. Scoreboard, raw as scored: six runs against the control, summed over the 20 epics.
The headline numbers are aggregates of this table. Rows O, T, A, E, F are the compliance checks (the headline table’s first row): O ownership (does the story name a system that may own it); T deliverable type; A operation; E entity or event; F functional realization. Each cell reads passed/failed; ”—” means nothing of that kind was produced. Row D is deliverables against the control (the second row): −missing +extra of the control’s 55 (the legend calls them gap and creep); the bar under each cell is the share of the control’s deliverables the run matched. Row INV is the flags (the third row): −missed of the six expected. The control column gives the base counts, for scale. Every number in the table is traceable to a recorded step under version control: mechanical transformations and judgment calls are kept separate, and the judgment calls were reviewed and attacked before they counted.
Adjusted scoring
The headline numbers show the raw scores. A logged review then credited admissible deviations, since many roads lead to Rome: I judged the runs blind to their condition (A or B), an AI holding the control data challenged each decision, and every decision was recorded and applied to all runs alike. Credited were, among others, renames that preserve meaning, an undetermined system slot (A’s ownership and type failures in the figure) read as “any”, and a violation named in commentary but not encoded as a flag. The last was admitted because the metamodel asks that violations be flagged and nowhere forbids decomposing them; the stories built anyway remain counted as unwarranted extras. Adjusted, the numbers move: A’s compliance checks failed 42–51 → 10–12, deliverables matched 2–4% → 16–18%, flags 0% → 17%; B’s compliance checks failed 82–94 → 52–76, deliverables matched 55–62% → 75–78%, flags 0–33% → 83–100%.
Limitations
The domain is non-exotic by design. Each constituent (layered systems, REST APIs, events) is a generic data point present in general training data; the specificity comes from combining them. The no-metamodel condition (A) therefore had some architectural knowledge available from training data. An organization’s own domain, its processes, rules and vocabulary, would give the model less to draw on and, presumably, widen the gap. This is an expectation; the experiment did not test it.
There is one metamodel, written once and taken as given. By construction it is the whole of the architectural knowledge given in context, 100%, and the no-metamodel condition (A) has none of it in context. Nothing in between was tested: not a partial metamodel, not the same knowledge in prose. How a better or worse metamodel would have moved the results was out of scope.
Only one model was tested, and not the newest. GPT-4.5 was chosen to keep a gap between the domain knowledge and what the model already carries in its weights; a newer model would raise the no-metamodel baseline (A) and shrink the gap. Whether it would also shrink the metamodel condition’s (B) failures is a question a single experiment cannot answer. My expectation is that it would not; the test is the same experiment on a newer model, and it has not been run.
The scoring is not fully mechanical. Two steps required judgment, simple in kind but large in volume: reading each run’s prose into score-able form, and decisions on the adjustments. Neither could be trusted to an AI alone, which hallucinated too often at them, nor to a human alone, for whom the volume was too tiring to stay consistent. So the human judged and the AI challenged, with every decision recorded. Everything else in the scoring was mechanical: a fixed procedure, run by AI, tested on known cases before it touched the runs; the kind of work whose result can be verified rather than trusted.
Conclusion
All of the above – the experiment, the scoring, its procedures and test cases, and every mechanical step – was done with text files, a chat window, and version control to keep every change verifiable. No custom AI infrastructure was used.
The experiment itself is n = 1: one domain, one metamodel, one model, three runs per condition, which makes it initial evidence. The metamodel condition (B) won by a wide margin, and the surprise is in what it still got wrong. The results raise questions about a) the failures themselves: the gaps, the well-formed extras, the invented vocabulary, the violations recognized but built anyway; and b) the metamodel’s part in those failures. Those are the subject of the follow-up analysis.
Finally, this experiment suggests that putting AI to work on your own processes does not require new tooling to start. You need your domain knowledge, written down, and the discipline to check the output against it.
AI: judgment before engagement. Building that judgment is the work I do.