The journeylism of @yreynhout

On CQRS, DDD(D), ES, …

Reply to “DDD, CQRS & ES: Lessons Learned”

Last Monday I attended a meetup of the Belgian Domain Driven Design group. One of the presentations was titled DDD, CQRS & ES: Lessons Learned, which – what did you expect? – caught my interest. Gitte, the presenter, explained the problems she and her team came across while applying the aforementioned set of acronyms. None of these stumbling blocks came as a surprise to me. It’s the same kind of issues that almost everyone faces when they first set out on this journey. The solutions are often trivial and diverse, but sometimes they’re not and may require a bit of explanation. I didn’t get a chance to discuss some of the alternatives with Gitte directly. You know how it goes, you end up talking to a lot of people but not everyone, and perhaps, in retrospect, it’s better to let things sink in, instead of blurting out what can only come across as condescending “I know better” type of remarks, which, honestly, is not my intent. So without further ado, let’s go over some of those problems:

Unique incremental numbers

Description

The basic problem here is that they wanted to assign each dossier a unique number with the added requirement that it had to be incremental. She went on to explain that the read model (for user interface purposes) wasn’t a good place, since that was just a side-effect of replaying events and would easily lose track of the highest number. Also the eventual consistent nature of that read model made it unsuitable for such purposes. In the end they settled on a singleton-eque, event-sourced aggregate that was made responsible for handing out unique numbers.

As an aside, if Greg Young earned a dollar for the number of times the “uniqueness” problem was discussed on the DDD/CARS mailing list, he’d be able to buy a new laptop (I’m being modest).

Reply

The problem with “uniqueness” requirements is that, well, very often there’s a deeper underlying reason why people want them. In this case, I can’t shake the feeling that it was a mixture of (a) having a human readable identifier that is easily communicated over the phone or in a letter, either among employees of the corporation or between employees and the customers/suppliers of the corporation and (b) knowing the number of dossier handled a year by just looking at the number. Obviously I’m going out on a limb in this particular case, but in general digging deeper in the domain gives a better understanding of why these requirements are in fact requirements. The aforementioned requirements could be solved in an entirely different manner than making an event-sourced number generator. Numbers could be assigned at a later point in time ordering all dossier births sequentially, or we could have a number reservation service (used quite liberal here) that hands out exclusive numbers and keeps a time-based lock on said number for us, or a high-lo reservation mechanism of numbers. But such efforts should be weighed against “real” requirements. What’s the business impact of handing out numbers in a different order, or having temporary gaps in the set of numbers, or what if the numbers do not correspond to the chronological order of events? Who is losing sleep over this? Now, the most common response to this is “But, but, but this was dead easy when we used our relational model. The database generated incremental numbers for us”. I’m sorry, but, but, but, you seem to confuse the requirement with the implementation at that point. Yes, you can leverage that implementation to satisfy the requirement, nothing wrong with doing that in your CRUD or anemic based model, but how aware are you of that underlying requirement? Did you even stop and think about it? Because that’s what these acronyms just made you do …

Akin to this problem is set based validation for which you can read this piece of prior art.

Refactoring

Description

For some reason they’ve found refactoring to be cumbersome, i.e. pulling behavior and corresponding events out of a particular aggregate and moving it into one or more new aggregates. Implementation wise they were using a derivative of Mark Nijhof’s Fohjin.

Reply

I don’t think the intent was to refactor here. Refactoring, i.e. restructuring code on the inside without changing external behavior, is much easier using an event-sourced model, especially since tests can be written using messages instead of the nasty coupling to implementation I usually get to see. The reason you break out to new aggregates is due to new insights, either around consistency boundaries or concepts that were implicit before or concepts that were misunderstood (Hm … how did this even pass analysis, design, coding, Q&A? The reality we live in, I guess). That’s not refactoring to me, but I don’t want to be splitting hairs either. Regardless, the problem remains. Yet, state based models have to deal with this issue as well. You bend your code to a new structure and you transform the data into a shape that fits the new structure. Hopefully, you’ve been collecting enough historical data to be able to provide sane input to that data transformation. State based models tend to “forget” historical facts because nobody deemed them to be important. Historical structural models are an even more painful experience I’ve blocked from my memory, so let’s not go there. Granted, tooling is more pervasive for structural models (or should I say the data that underpins them). But it’s not exactly a walk in the park either. It’s always going to be work, either way. It’s just different work.

One thing I noticed was that they were using a base class for their events (https://github.com/MarkNijhof/Fohjin/blob/master/Fohjin.DDD.Example/Fohjin.DDD.Events/DomainEvent.cs) with a property called “AggregateId”. That’s one source of trouble when you want to move events to another aggregate. Don’t do that. Just call it what it is, e.g. a CustomerId or an OrderId or a DossierId. The aggregate an event was produced by should cater for the identification of the corresponding stream, not the event itself (though e.g. an attribute/annotation based approach could work as well). Don’t get me wrong, you can copy all data you like into the event, including the aggregate id, the event id, the version. But I doubt it’s wise to make it the basis of your infrastructure. At least, such has been my experience. I could make a similar argument for the version and event id.

Another remark that struck me as odd was the fact that event migration is non-trivial. Odd, since I’ve always perceived event streams as lightweight, easy to transform, merge or split. This should be dead easy. If it’s not, you might want to start digging to see where the friction is coming from.

Da Big Fat Aggregate Cat

Description

Some of their aggregates were getting too big, resulting in too many lines of code, especially since decision and state changing behavior had been split into two different methods.

Reply

Their future solution to this issue is looking into splitting up the decision making and state tracking in two different classes. The good news is, it’s been done before, no need to reinvent that wheel. The most common reasons for these obese aggregates is that they have too fine-grained behaviors, their behaviors take parameter lists that require scrolling off-screen, the consistency boundary is modeled after a noun, or we might be forcing an event-sourced approach onto what would be better served with a mixture of a document and events. It’s hard to tell without looking at the specifics, but it’s often an indicator that more investigation is needed instead of brushing it off as a code navigation issue. Vaughn Vernon’s book and papers, as well as this excellent post by Gojko Adzic should be a gentle reminder of what care and consideration should go into designing your aggregates.

Tooling

Description

Tooling for building these types of systems isn’t as pervasive as it is for structural/state-based models. Be prepared to build a lot of it yourself.

Reply

From an operational point of view there might be some truth to this statement and there’s definitely room for improvement. But maybe that’s what I love about it. It’s more about cherry-picking existing tooling that fills the gaps that need filling. For the most part it’s about understanding the mechanics and the forces at play. There’s no wrong or right, there’s just various trade-offs and specific requirements/needs. Trying to come up with something that satisfies a lot of the generic requirements would lead us down the path of the lowest common denominator or a Swiss army knife you configure using lots of xml, json, yaml or even code. Let’s not go there … ever again. I know, sounds like hand-waving, but give me a specific problem, and I’ll try to give you a specific answer.

To put things into perspective, I’ve built a stream viewer/smuggler/transformer/rewriter on top of NEventStore, all in under one day’s worth of work. Not production quality, but good enough nonetheless. Alternatively, I could have chosen to leverage a document store, put all the events in it using a catch-up subscription and reaped the benefits of full text searching those events, all in under a few hours, all because I know how to marry a paradigm and a set of technologies … and this to me is the essential part.

In the end …

I’ve just reiterated some of the problems they’ve encountered. It’s good people discuss these things out in the open, because that’s how you shorten the feedback loop, even if the solutions seem trivial or highly depend on context. I hope it’s clear that my replies should not be interpreted as glorifying the acronyms nor as bashing of the presenter.

Advertisements

2 responses to “Reply to “DDD, CQRS & ES: Lessons Learned”

  1. Tom Ceuppens (@KyorCode) November 13, 2014 at 09:16

    Could you elaborate a bit more on the ‘AggregateId’ problem? How would you associate the events with the correct aggregate?

  2. yreynhout November 13, 2014 at 13:57

    I can. There are many ways of going about it, really. Mostly depends on how it affects calling code and what “feels” comfortable. In AggregateSource for example, there’s a type called Fact that is a named tuple over the aggregate’s identifier and the event. Another way of doing it might be to sprinkle an attribute/annotation on the property of the event that identifies the aggregate. Yet another way might be to expose the aggregate id on the root entity of the aggregate and – upon getting the changes – compose two pieces of information. In all these cases, I’m favoring composition instead of marking my events with an interface or having them inherit some base class. You may wonder “How does that help with refactoring?”. Well, I’m now free to move (could be a copy mind you) events to different streams WITHOUT having to touch the events (no fiddling with an “AggregateId” property required). In essence, the aggregate an event belongs to is not tied to its payload. The payload can still contain the aggregate identifier because it pertains to the event, but it’s not there for infrastructural but rather factual reasons. Obviously there’s a limit to what one can do here without affecting the past, but that’s no different in a structural model (e.g. if you didn’t capture it). I can’t be anymore specific than this without resorting to speech, code or diagrams …

    Mind you that a stream name and aggregate identifier might be different, e.g. a stream name could capture the bounded context name, the tenant (assuming multi-tenancy) the aggregate belongs too, or even the name of the aggregate (here be dragons though), whereas an aggregate identifier is more local and might be just a unique identifier within the set of aggregates of a certain type. A stream name probably doesn’t go into the payload of the event, while an aggregate identifier in the form of e.g. a DossierId might be very useful information to embed.

    People that are trying to rip apart their aggregates might want to invest more time into the analysis and design phase next time round. That’s not to say they haven’t or that one would be shielded from ever having to re-partition streams. But often, that’s where the trouble starts. What most seem to forget is that non-functional requirements such as size (number of things on the inside), expected life-time and states, volume (number of events), whether the associated concepts are more document or behavior oriented or a mixture, all have an impact on how one designs aggregates. That doesn’t imply that we should start out with non-functional requirements, but it does mean we can’t turn a blind eye on them.

    HTH,
    Yves.

Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out / Change )

Twitter picture

You are commenting using your Twitter account. Log Out / Change )

Facebook photo

You are commenting using your Facebook account. Log Out / Change )

Google+ photo

You are commenting using your Google+ account. Log Out / Change )

Connecting to %s

%d bloggers like this: