If I had to choose the killer feature of Event Sourcing, I’d select projections. Why? I’ll explain that in detail in this article.
\nEvents are facts; they represent something that has happened in the past. Projections are a different interpretation of the set of facts. Sounds enigmatic? Let’s have a look at this picture.
\nEvents are facts and, thus, an excellent source for data exploration. Projections, together with analytics, and data drilling, can bring valuable insights. For instance, by checking products bought in the e-commerce system, we can find the gaps like abandoned carts, build a recommendation engine, etc. Read more in Never Lose Data Again - Event Sourcing to the Rescue!.
\nAs mentioned, a single fact can have multiple interpretations; events can be a source of numerous projections. For instance, information about the product added to the shopping cart can trigger the contents view update, change the product availability, and feed the information out for analytics.
\nOf course, nothing stops you from batch processing by taking all the events, merging them and storing the result. For instance, if you’re doing set-based aggregations like the total of sold products, money accrual, and average grades in school, it could be more efficient if done en masse.
\nAlthough most tooling shows that they need eventual consistency, that’s incorrect. Their state doesn’t have to be updated with a delay. It’s a technical “detail” of the exact implementation. If you’re storing events and read models in the same relational database (like e.g. Marten in Postgres), then you can wrap all in a transaction. Then either events are appended and projections updated, or no change is made.
\nOf course, we should not be afraid of eventual consistency. How soon is now? There’s no now! Everything in the real world happens with some delay. Also, if we want to make a reliable and fault-tolerant processing pipeline doing stuff as a background process may be a go-to way. Read more in Outbox, Inbox patterns and delivery guarantees explained.
\nProjections are a concept that’s part of Event-Driven Architecture and essential building block of CQRS. They’re not tied to Event Sourcing. Yet, event stores can help you in making them efficient. Most of the time, they allow you to subscribe to the appended events notifications, for instance, Marten’s Async Daemon or EventStoreDB subscriptions. They are an outbox pattern provided as a commodity by the database. That gives both the durability for the events and strong guarantees around processing them.
\nThe significant benefit of the projections is that they’re predictable. That means that the same logic will generate the same result for the same events.
\nAs events are our source of truth, then we can think of read models as secondary data. If we treat events as the source of truth and base projection processing only on data we get from them, we can also rebuild our read models.
\nWe could break it into two phases:
\nHow to decide if the gap is “small enough”? We might never be fully caught up if events are appended continuously. Because of that, we need to define some threshold, e.g. in our context, live means that we have a maximum of ten events to process from the latest registered event. Of course, this can be done case by case and differ for various projections. Read also more in Let’s talk about positions in event stores.
\nWe can use different projection handling techniques depending on the phase we’re in, e.g., batch processing when we’re catching up and left-fold when we’re live.
\nProjections rebuilds are usually made asynchronously, while the left-fold approach can work well in sync and async ways.
\nThe simplest rebuild we can do is truncate the current state of the read model and then reapply all events through projection logic. We can do that if we can afford the downtime. There will be a time when there’s no data, or we’re far from being up to date.
\nIf we cannot afford downtime, we can do a blue-green rebuild. In that, we’re setting up a read model in the other storage (e.g. different database, schema, table, etc.). Then we’re reapplying events to this secondary read model. Once we’re caught up, we can switch queries to target the new read model and archive the old one.
\nThrough my window, I see the result of good plans but poor execution. Opposite my flat, there is a partially completed construction place. Buildings were supposed to be eye-catching Mediterranean style apartments. Delivery date? Two years ago. Actual? More and more unknown.
\nSome time ago, I heard that using Event Sourcing makes creating Event-Driven Architecture easier. The arguments were correct, that if we’re already publishing events to trigger business workflows, then at some point, we may want to also store events to not lose information. Agreed. However, I also heard that keeping the state as events will simplify things. We’ll have a source of truth with a record of the system behaviour. This will allow, e.g. to confront the results of the operations with the recorded state. I’d agree with that, with one distinction. It’s easier as long as you already know Event Sourcing.
\nMany people in the DDD community claim that the essential is to properly break down the system into autonomous parts called bounded contexts. Once we have it, the rest is secondary and will sort itself out. For sure.
\nMany seasoned programmers speak similarly about new technologies. They claim that they can translate past experience into new technologies. That’s true that by analogy, they can catch the big picture quicker. But isn’t it a bold assumption to say that Win.Forms specialist will learn Angular quickly?
\nThe end result may differ a lot from the initial ideas. I saw the plan of those buildings next to me. Now I can see the effects of the execution. Or actually, the lack.
\nI believe that we should carefully acknowledge not only the point of view of our authorities but also their seating point. If we want to find out how to form a wall, do we ask an architect or a foreman? An architect may know the theory, but the practice is what we’re looking for. On the other hand, if you want to know where to put the wall, you prefer the architect to do measurements. At least if you don’t want to have the roof falling to your head.
\nAfter I had torn a ligament in my knee, I went to two qualified orthopedists. One said I should have surgery and do a reconstruction. The second stated that there is no need for that; rehabilitation should be enough. Guess which one had a specialization in surgery and which in rehabilitation?
\nPeople usually give us advice from the point where they’re currently standing. They are entitled to a biased view. An architect who rarely does programming will tend to downplay the value of implementation and tactical patterns. Midlevel developers will focus on technicalities instead of the global system impact. The team manager or consultant will emphasize the importance of soft skills (or esoteric techniques known only to them).
\nThe truth is that we need all of them. The excellent plan will fall on the bad execution. The best execution for the wrong case will be just a waste of time. We should carefully evaluate the advice considering what we need and what an expert can give us.
\nTherefore, when we’re reading an article, watching a talk, let’s also pay attention to the place where the person is standing. The perspective from there may be much different from where we are right now. That can be good, as it may push us in the right direction. But it may also be misleading, as we accidentally take biases of this person without understanding the tradeoffs. Personally, I prefer to follow not only people from pedestal but also those that are closer to my position. A bit further in the journey, but not too far. That helps me to calibrate my view as those people are more relative to my daily struggles.
\nPolish historical leader Józef Piłsudzki reportedly used to say: “Right is like an ass, everyone has its own”.
\nCheers!
\nOskar
"},"site":{"siteMetadata":{"facebook":{"appId":""}}}},"pageContext":{"slug":"/projections_and_read_models_in_event_driven_architecture/","lang":"en","langKey":"en","prev":{"id":"cbd757de-9e1c-5670-bc4c-2766593b259f","fields":{"slug":"/how_to_validate_business_logic/","prefix":"2023-01-15","source":"posts","langKey":"en"},"frontmatter":{"title":"How to validate business logic","category":"Event Sourcing"}},"next":{"id":"de486c79-7268-55c7-ac18-c703428647a8","fields":{"slug":"/stacking_the_bricks/","prefix":"2023-01-29","source":"posts","langKey":"en"},"frontmatter":{"title":"Stacking the bricks in the software development process","category":"Coding Life"}},"source":"posts"}}, "staticQueryHashes": ["2742854296"]}