Building occasionally connected applications using event sourcing
•Als PPTX, PDF herunterladen•
7 gefällt mir•13,197 views
I've recently got the opportunity to work on a large enterprise-class system that needs to be deployed on multiple occasionally connected oil platforms and boats. Already the system's architecture was based on the Command Query Separation principles, this gave us a completely new challenge. After several months of looking at alternatives, we decided to go the Event Sourcing direction. In this in-depth session, I'd like you to learn the many alternatives we observed, the pros and cons, and the technical details of our final solution in which we use EventStore 3.0 and elements of NCQRS and Lokad.CQRS to synchronize systems over unreliable connections in a very efficient way.
Empfohlen
Weitere ähnliche Inhalte
Was ist angesagt?
Was ist angesagt? (20)
Andere mochten auch
Andere mochten auch (20)
Ähnlich wie Building occasionally connected applications using event sourcing
Ähnlich wie Building occasionally connected applications using event sourcing (20)
Mehr von Dennis Doomen
Mehr von Dennis Doomen (20)
Kürzlich hochgeladen
Kürzlich hochgeladen (20)
Building occasionally connected applications using event sourcing
- 1. Occassionally Connected Applications With Event Sourcing Dennis Doomen © 2014 Aviva Solutions 21 november 2014 dennis.doomen@avivasolutions.nl
- 2. About Me • Principal Consultant • 17 years in IT • C++ origins but C# since 2001 • Specialties • .NET • Architecture • Scrum/Kanban/XP • ALM • Speaker • Public initiatives • C# Coding Guidelines • Fluent Assertions • Internet • www.dennisdoomen.net • DZone MVB • @ddoomen © 2014 Aviva Solutions Dennis Doomen 21 november 2014
- 3. Contents • What we tried to solve • Our options • What we choose for • Design Challenges • Scalability Challenges • Architectural Challenges • Concurrency Challenges • Future? © 2014 Aviva Solutions 21 november 2014
- 4. What we tried to solve © 2014 Aviva Solutions 21 november 2014
- 5. Central Farm Seaborn Servers High- Latency LAN Decentralized Servers attachments) Mobile Clients Intermittent Satellite Connection 3G (w/o WLAN (attachments) Only synchronize what is relevant! >100 XCopy RDBMS © 2014 Aviva Solutions 21 november 2014
- 6. What we had Commands Web Application Command Service Projections Queries Query Processor Command Handlers Domain Model Repositories Domain Services SQL / Oracle Service Agents Backoffice Systems Query Handlers Repositories Patterns & Principles • Domain Driven Design • CQRS • Repository Abstraction • TDD/BDD • SOLID Framework & Libraries • NHibernate • Autofac • ASP.NET WebForms / MVC • SQLite © 2014 Aviva Solutions 21 november 2014
- 7. Our options 1. SQL Server Replication 2. Command Synchronization 3. Aggregate Serialization 4. RavenDB 5. Event Sourcing © 2014 Aviva Solutions 21 november 2014
- 8. 1. SQL Server Replication Automatic reconnects Proven technology Automatic schema updates Development can be outsourced Synchronization over HTTP Only SQL Server (Express) Filtering is limited Affects client infrastructure No real-world examples Unsure about limitations Requires SQL expertize. Difficult to diagnose issues during development © 2014 Aviva Solutions 21 november 2014
- 9. 2. Command Synchronization Supports any database No special deployments requirements Potentially unresolvable out-of-sync issue Requires bypassing business rules Command != actual change Limited filtering possibilities Custom serialization Separate queue of commands © 2014 Aviva Solutions 21 november 2014
- 10. 3. Aggregate Serialization Supports any database No special deployments requirements Doesn’t interfere with business rules No out-of-sync issues Requires specialized code per aggregate Needs an enforced order of synchronization. Payload is not optimized Requires a lot of querying and change tracking © 2014 Aviva Solutions 21 november 2014
- 11. 4. RavenDB Reduces complexity Smart reconnects Synchronization built-in Archiving built-in Built-in caching Very scalable Improves productivity Improves query performance Lucene based Acceptance by enterprise clients New querying (map/reduce) concepts Limited track record © 2014 Aviva Solutions 21 november 2014
- 12. Event Sourcing Web Application Command Service Execute query ChangeUserEmailCommand Query Processor ChangeUser EmailHandler Invoke method User Aggregate Submit changes Get<TAggregateRoot>(key, version) Domain UOW UserProjector Query Handler LINQ, HQL, SQL Projections UOW Read DB Load(events) DAO Commits Write DB Submit Apply Get changes Dispatcher Commit Handle(UserEmailChangedEvent) © 2014 Aviva Solutions 21 november 2014
- 13. 5. Event Sourcing Transparent to clients Works with any SQL/No-SQL DB Can improve performance Advance conflict resolution Natural unit of synchronization Can be very scalable Lots of new concepts Big impact on domain model Synchronization must be custom-built Lots of unforeseen challenges © 2014 Aviva Solutions 21 november 2014
- 14. What we chose for © 2014 Aviva Solutions 21 november 2014
- 15. The Revised Architecture Commands Command Service Domain Model Events Dispatcher Domain Repository Command Handlers Domain Services Event Store Service Agents Web Application Backoffice Systems Queries Projections Query Processor Query Handler Projectors Projection Repositories Query Store Events © 2014 Aviva Solutions 21 november 2014
- 16. Advantages • Optimized projections • Projections = cache -> rebuild anytime • Reporting node • Intrinsic auditing • Supports temporal projections © 2014 Aviva Solutions 21 november 2014
- 17. Synchronization Overview Just changes ownership of an aggregate and Primary Node Laptop / Secondary Node Check-out / Check-in forces synchronization Each aggregate is owned by a node to prevent functional conflicts SOAP/WS-* E.g. UserPasswordChanged, DocumentSigned, Contains a subset of the master Changes Changes Images, etc 6298 6298 6297 3 2 1 User Interface Changes Queries Attachments, Blobs Query Data Users Groups Documents Objects Folders 6301 6300 6299 3 2 1 User Interface Queries Changes Attachments, Images, etc Blobs Can be rebuild from the changes table at any point in time Query Data Users Groups Documents Objects Folders TitleChanged, RoleRevoked WIFI/LAN/3G WIFI/LAN Optimized for querying SOAP/WS-* HTTPS-REST © 2014 Aviva Solutions 21 november 2014
- 18. Design Challenges © 2014 Aviva Solutions 21 november 2014
- 19. NEventStore vs Lokad.CQRS vs NCQRS Apply(EmailChangedEvent) User Aggregate new User() EventStoreDataMapper Caller ChangeEmail() Open Stream(guid) Write Into Stream Per Aggregate NEventStore 3 Commits User State GetAggregateRoot(key) Domain Unit Of Work Load(key) Load(events) OnEventApplied() SubmitChanges © 2014 Aviva Solutions 21 november 2014
- 20. Immediate vs Eventual Consistency Submit Changes EventStoreDataMapper NEventStore 3 Queries Projector NHibernate Dispatched Event Projections Commits Transactional Boundary Transactional Boundary © 2014 Aviva Solutions 21 november 2014
- 21. Event Versioning Event V1 Event V2 Some Event Combined Event Other Event Combined Event Separate Event Separate Event © 2014 Aviva Solutions 21 november 2014
- 22. Other Challenges • No direct AR dependencies • Granularity of events • Domain Events as facts, not triggers © 2014 Aviva Solutions 21 november 2014
- 23. Scalability Issues © 2014 Aviva Solutions 21 november 2014
- 24. Scalability Limitations • Commits table grows and gets slower • No partitioning of global data • Single point of failure • Upgrades take too much time © 2014 Aviva Solutions 21 november 2014
- 25. Out-of-place migration Version 1.0 Version 3.0 Changes 6299 6298 6297 3 2 1 Blobs Query Data Users Groups Documents Objects Folders Changes 6301 6300 6299 3 2 1 Blobs Query Data Users Groups Documents Objects Folders Schema 1.0 Pass 2= down time Pass 1= no down time Schema 3.0 Setup Upconverts commits, blobs, etc © 2014 Aviva Solutions 21 november 2014
- 26. Multi-Tenant Scaling systems Specific reporting Integration Node Reporting Server Connects to back-office Node (owns tenants 1 & 2) Node (owns tenants 3 & 4) Node (owns tenants 5 & 6) Node (failover for tenants 5 & 6) representations of the data Node (owns tenants 9 & 10) Node (owns ref. data & node config) Node (uses tenants 1 & 2, owns tenant 11) Decentralized node Node (uses tenants 1 & 11) Node (uses tenants 3 & 5) Node (uses tenants 5 & 10) Owns global data and ownership configuration © 2014 Aviva Solutions 21 november 2014
- 27. Architectural Issues © 2014 Aviva Solutions 21 november 2014
- 28. Limiting Constraints • Recovery in web farm • Selective rebuilding of projections • No asynchronous dispatching • Relational database is bottleneck during load balancing • Network overhead web->database © 2014 Aviva Solutions 21 november 2014
- 29. Load Balancer First Attempt Web Site Web Site Worker Thread RDBMS Projections Worker Thread RDBMS Commits Application Tier Database Tier © 2014 Aviva Solutions 21 november 2014
- 30. Load Balancer Second Attempt Web Site Web Site Worker Thread RDBMS Projections RDBMS Commits Worker Thread RDBMS Projections Application Tier Database Tier © 2014 Aviva Solutions 21 november 2014
- 31. Final Attempt Load Balancer Web Site Web Site Application Tier Worker Thread RavenDB Projections RDBMS Commits Worker Thread RavenDB Projections Database Tier © 2014 Aviva Solutions 21 november 2014
- 32. RavenDB Advantages • No migrations needed • No more column length problems • Faster • Less SQL Server load • More scalable • Dynamic Indexing • Keyboard/faceted search w/ Lucene • Concious decision on asynchronicity Disadvantages • New technology, new concepts • No LINQ • Async indexes • Requires rewrite of all projectors and queries © 2014 Aviva Solutions 21 november 2014
- 33. Front-End Server Web Site WCF Services ? Worker Thread ? ? RavenDB Projections Commits Application Tier Database Tier Job Scheduler ? © 2014 Aviva Solutions 21 november 2014
- 34. Front-End Server /queries/{query} Worker Job Scheduler Thread RavenDB Projections Web Site Commits Application Tier Database Tier WCF Services © 2014 Aviva Solutions 21 november 2014
- 35. OWIN Startup Main Repository Component Repository Web Site Module Module Queries Queries Query Handlers QueryHost RavenDB Query RavenSession Checkpoint Store HTTP-based QueryProcessor Interfaces Query Handlers HTTP (network or memory) Projectors Projectors RavenDB Commit EventStore QueryHost Raven QueryHost Client QueryHost /queries/{query} Querying Controller Thread Pool Owin WebAPI 2 NEventStore Katana Autofac Query Host Settings Durable Commit Dispatcher © 2014 Aviva Solutions 21 november 2014
- 36. Concurrency Issues © 2014 Aviva Solutions 21 november 2014
- 37. Event Merging Primary Node Laptop / Secondary Node StreamId: User-Dedo, Rev: 6 RoleRevokedEvent StreamId: User-Dedo PasswordChangedEvent StreamId: User-Dedo, Rev: 5 PasswordChangedEvent StreamId: User-Dedo, Rev: 4 GrantedRoleEvent StreamId: User-Dedo, Rev: 3 PhoneNumberAddedEvent GrantedRoleEvent UserCreatedEvent StreamId: User-Dedo, Rev: 7 PasswordChangedEvent StreamId: User-Dedo, Rev: 6 GrantedRoleEvent StreamId: User-Dedo, Rev: 5 RoleRevokedEvent StreamId: User-Dedo, Rev: 4 PasswordChangedEvent StreamId: User-Dedo, Rev: 3 PhoneNumberAddedEvent GrantedRoleEvent UserCreatedEvent Last Sync Point Time © 2014 Aviva Solutions 21 november 2014
- 38. Future? © 2014 Aviva Solutions 21 november 2014
- 39. Opportunities • In-memory projections • Gossip-based detection • ATOM feeds © 2014 Aviva Solutions 21 november 2014
- 40. Reading Material • NEventStore on GitHub • Domain Events by Udi Dahan • Event Versioning by Rinat Abdullin • Effective Aggregate Design by Vaughn Vernon • NHibernate vs Entity Framework in DDD by me… • DomainDrivenDesign.org • RAFT, a distributed consensus protocol • Cedar by Damian Hickey © 2014 Aviva Solutions 21 november 2014
- 41. Email dennis.doomen@avivasolutions.nl Twitter @ddoomen Blog www.dennisdoomen.net © 2014 Aviva Solutions 21 november 2014
Hinweis der Redaktion
- Disadvantages No concurrency; conflicting service requests are simply rejected Staleness ignored; Scalability limited; No domain verbs; domain model does contain concepts from the domain model, just not the business actions Granularity DTOs; What operations to expose in your WCF service? What relations to include and when? What domain entities to flatten. Unneccesary DTO conversions; Conversions from/to domain model entities, traversing of relations, eager fatching Conflicting demands; query demands denormalized schema, commands require normalized integer schema
- Disadvantages No concurrency; conflicting service requests are simply rejected Staleness ignored; Scalability limited; No domain verbs; domain model does contain concepts from the domain model, just not the business actions Granularity DTOs; What operations to expose in your WCF service? What relations to include and when? What domain entities to flatten. Unneccesary DTO conversions; Conversions from/to domain model entities, traversing of relations, eager fatching Conflicting demands; query demands denormalized schema, commands require normalized integer schema
- Disadvantages No concurrency; conflicting service requests are simply rejected Staleness ignored; Scalability limited; No domain verbs; domain model does contain concepts from the domain model, just not the business actions Granularity DTOs; What operations to expose in your WCF service? What relations to include and when? What domain entities to flatten. Unneccesary DTO conversions; Conversions from/to domain model entities, traversing of relations, eager fatching Conflicting demands; query demands denormalized schema, commands require normalized integer schema
- Disadvantages No concurrency; conflicting service requests are simply rejected Staleness ignored; Scalability limited; No domain verbs; domain model does contain concepts from the domain model, just not the business actions Granularity DTOs; What operations to expose in your WCF service? What relations to include and when? What domain entities to flatten. Unneccesary DTO conversions; Conversions from/to domain model entities, traversing of relations, eager fatching Conflicting demands; query demands denormalized schema, commands require normalized integer schema
- Disadvantages No concurrency; conflicting service requests are simply rejected Staleness ignored; Scalability limited; No domain verbs; domain model does contain concepts from the domain model, just not the business actions Granularity DTOs; What operations to expose in your WCF service? What relations to include and when? What domain entities to flatten. Unneccesary DTO conversions; Conversions from/to domain model entities, traversing of relations, eager fatching Conflicting demands; query demands denormalized schema, commands require normalized integer schema
- Disadvantages No concurrency; conflicting service requests are simply rejected Staleness ignored; Scalability limited; No domain verbs; domain model does contain concepts from the domain model, just not the business actions Granularity DTOs; What operations to expose in your WCF service? What relations to include and when? What domain entities to flatten. Unneccesary DTO conversions; Conversions from/to domain model entities, traversing of relations, eager fatching Conflicting demands; query demands denormalized schema, commands require normalized integer schema
- Disadvantages No concurrency; conflicting service requests are simply rejected Staleness ignored; Scalability limited; No domain verbs; domain model does contain concepts from the domain model, just not the business actions Granularity DTOs; What operations to expose in your WCF service? What relations to include and when? What domain entities to flatten. Unneccesary DTO conversions; Conversions from/to domain model entities, traversing of relations, eager fatching Conflicting demands; query demands denormalized schema, commands require normalized integer schema