第12章開(kāi)始這段總結(jié)了整本書(shū)的脈絡(luò)览祖，以及我們面對(duì)不同問(wèn)題的時(shí)候應(yīng)該知道有不同的選擇，
而當(dāng)我們面對(duì)一個(gè)復(fù)雜系統(tǒng)的時(shí)候能颁，通常要整合很多不同DB 來(lái)用于不同的需求
cache 用來(lái)優(yōu)化 read heavy次慢， column index storage 用來(lái)用于 search feature等等
Combining specialized tools by deriving data
For example, it is common to need to integrate an OLTP database with a full-text search index in order to handle queries for arbitrary keywords.
Besides the database and the search index, perhaps you need to keep copies of the data in analytics systems (data warehouses, or batch and stream processing systems);
看來(lái)之前對(duì)stream processing system 有誤解，我以為是實(shí)時(shí)的那種系統(tǒng)粱檀，看來(lái)stream processing system 也主要用于用戶分析洲敢？
這段話也很深刻…… 有時(shí)候需要站在整個(gè) organization 層面去思考問(wèn)題， 因?yàn)閿?shù)據(jù)在不同人那里的需求是完全不一樣的茄蚯。
所以你在設(shè)計(jì)一個(gè)application的時(shí)候压彭， 一定要考慮全面，比如這個(gè)數(shù)據(jù)還可以用在什么地方渗常，然后通過(guò)不同的工具來(lái)給公司其他部門(mén)提供便利性
Surprisingly often I see software engineers make statements like, “In my experience, 99% of people only need X” or “...don’t need X” (for various values of X). I think that such statements say more about the experience of the speaker than about the actual usefulness of a technology. The range of different things you might want to do with data is dizzyingly wide. What one person considers to be an obscure and pointless feature may well be a central requirement for someone else. The need for data inte‐ gration often only becomes apparent if you zoom out and consider the dataflows across an entire organization.
所以 total order broadcast 等于 consensus
In formal terms, deciding on a total order of events is known as total order broadcast, which is equivalent to consensus (see “Consensus algorithms and total order broad‐ cast” on page 366). Most consensus algorithms are designed for situations in which the throughput of a single node is sufficient to process the entire stream of events, and these algorithms do not provide a mechanism for multiple nodes to share the work of ordering the events. It is still an open research problem to design consensus algorithms that can scale beyond the throughput of a single node and that work well in a geographically distributed setting.
目前還是一個(gè)沒(méi)有被解決的問(wèn)題壮不，consensus algorithm 目前來(lái)講都還是基于 single node 的
如果必須avoid single node bottleneck, conflict resolution 可能要在application 層面發(fā)生
Batch and Stream Processing
batch and stream processor 主要就是讓 right data ends up in the right place
他們有很多共同點(diǎn)， 最大不同就是 stream processor input is unbounded
而它們的實(shí)現(xiàn)方式現(xiàn)在越來(lái)越趨同了凳谦，
Spark performs stream processing on top of a batch processing engine by breaking the stream into microbatches, whereas Apache Flink performs batch processing on top of a stream processing engine [5]. In principle, one type of processing can be emulated on top of the other, although the performance characteristics vary: for example, microbatching may perform poorly on hopping or sliding windows [6].
http://data-artisans.com/batch-is-a-special-case-of-streaming/
[5] Kostas Tzoumas: “Batch Is a Special Case of Streaming,” data-artisans.com, Sep‐ tember 15, 2015.
對(duì)于functional programming不了解的我也是比較有幫助的忆畅，看來(lái)functional programming 主要是deterministic 并且不會(huì)影響input？
Batch processing has a quite strong functional flavor (even if the code is not written in a functional programming language): it encourages deterministic, pure functions whose output depends only on the input and which have no side effects other than the explicit outputs, treating inputs as immutable and outputs as append-only. Stream processing is similar, but it extends operators to allow managed, fault-tolerant state (see “Rebuilding state after a failure” on page 478).
derived data 能讓我們進(jìn)行 gradual migration
Derived views allow gradual evolution. If you want to restructure a dataset, you do not need to perform the migration as a sudden switch. Instead, you can maintain the old schema and the new schema side by side as two independently derived views onto the same underlying data. You can then start shifting a small number of users to the new view in order to test its performance and find any bugs, while most users con‐ tinue to be routed to the old view. Gradually, you can increase the proportion of users accessing the new view, and eventually you can drop the old view [10].
The beauty of such a gradual migration is that every stage of the process is easily reversible if something goes wrong: you always have a working system to go back to. By reducing the risk of irreversible damage, you can be more confident about going ahead, and thus move faster to improve your system [11].
https://stripe.com/blog/online-migrations
[10] Jacqueline Xu: “Online Migrations at Scale,” stripe.com, February 2, 2017.
http://conferences.oreilly.com/software-architecture/sa2015/public/schedule/detail/40388
[11] Molly Bartlett Dishman and Martin Fowler: “Agile Architecture,” at O’Reilly
Software Architecture Conference, March 2015.
Lambda architecture
Lambda architecture 就是讓兩個(gè)系統(tǒng)并行尸执，one is batch, one is stream
兩個(gè)系統(tǒng)都從always growing dataset 里面讀
If batch processing is used to reprocess historical data, and stream processing is used to process recent updates, then how do you combine the two? The lambda architec‐ ture [12] is a proposal in this area that has gained a lot of attention.
The core idea of the lambda architecture is that incoming data should be recorded by appending immutable events to an always-growing dataset, similarly to event sourc‐ ing (see “Event Sourcing” on page 457). From these events, read-optimized views are derived. The lambda architecture proposes running two different systems in parallel: a batch processing system such as Hadoop MapReduce, and a separate stream- processing system such as Storm.
In the lambda approach, the stream processor consumes the events and quickly pro‐ duces an approximate update to the view; the batch processor later consumes the same set of events and produces a corrected version of the derived view.
所以stream 給一個(gè) approximate view家凯， 過(guò)段時(shí)間 batch 給精確的 view
竟然已經(jīng)有整合的方案了？如失？
More recent work has enabled the benefits of the lambda architecture to be enjoyed without its downsides, by allowing both batch computations (reprocessing historical data) and stream computations (processing events as they arrive) to be implemented in the same system [15].
[15] Raul Castro Fernandez, Peter Pietzuch, Jay Kreps, et al.: “Liquid: Unifying Near‐ line and Offline Big Data Integration,” at 7th Biennial Conference on Innovative Data Systems Research (CIDR), January 2015.
- Tools for windowing by event time, not by processing time, since processing time is meaningless when reprocessing historical events (see “Reasoning About Time” on page 468). For example, Apache Beam provides an API for expressing such computations, which can then be run using Apache Flink or Google Cloud Dataflow.
感覺(jué)Flink 很強(qiáng)啊……
這個(gè)觀點(diǎn)太有意思了绊诲！
At a most abstract level, databases, Hadoop, and operating systems all perform the same functions: they store some data, and they allow you to process and query that data [16]. A database stores data in records of some data model (rows in tables, docu‐ ments, vertices in a graph, etc.) while an operating system’s filesystem stores data in files—but at their core, both are “information management” systems [17].
http://www.cs.virginia.edu/~zaher/classes/CS656/p365-ritchie.pdf
[16] Dennis M. Ritchie and Ken Thompson: “The UNIX Time-Sharing System,” Communications of the ACM, volume 17, number 7, pages 365–375, July 1974. doi: 10.1145/361011.361061
http://people.eecs.berkeley.edu/~brewer/cs262/systemr.html
[17] Eric A. Brewer and Joseph M. Hellerstein: “CS262a: Advanced Topics in Com‐ puter Systems,” lecture notes, University of California, Berkeley, cs.berkeley.edu, August 2011.
這一段話把書(shū)中很多概念都清晰的定義出來(lái)了，我覺(jué)得你可以把這段話放在一個(gè)地方供你搜索
DDIA 概念定義
Composing Data Storage Technologies
Over the course of this book we have discussed various features provided by data‐ bases and how they work, including:
? Secondary indexes, which allow you to efficiently search for records based on the value of a field (see “Other Indexing Structures” on page 85)
- Materialized views, which are a kind of precomputed cache of query results (see “Aggregation: Data Cubes and Materialized Views” on page 101)
- Replication logs, which keep copies of the data on other nodes up to date (see “Implementation of Replication Logs” on page 158)
- Full-text search indexes, which allow keyword search in text (see “Full-text search and fuzzy indexes” on page 88) and which are built into some relational databases [1]
我覺(jué)得這段話實(shí)際上很好的概括了分布式系統(tǒng)的philosophy, 就是整個(gè)系統(tǒng)是一個(gè)由不同部件組成的褪贵， 每個(gè)部件是一個(gè)獨(dú)立的個(gè)體掂之， 而系統(tǒng)把不同部件整合到一起
跟 unix 的 philosophy 是一致的抗俄，one component must do one thing, and does it well
Viewed like this, batch and stream processors are like elaborate implementations of triggers, stored procedures, and materialized view maintenance routines. The derived data systems they maintain are like different index types. For example, a relational database may support B-tree indexes, hash indexes, spatial indexes (see “Multi- column indexes” on page 87), and other types of indexes. In the emerging architec‐ ture of derived data systems, instead of implementing those facilities as features of a single integrated database product, they are provided by various different pieces of software, running on different machines, administered by different teams.
尤其是這句話
In the emerging architec‐ ture of derived data systems, instead of implementing those facilities as features of a single integrated database product, they are provided by various different pieces of software, running on different machines, administered by different teams.
DB本身想要提供所有功能就有點(diǎn)不現(xiàn)實(shí)
ha！ 我的想法跟Martin 又重合了 世舰， 證明我還是理解了他的意思嘛动雹！
我感覺(jué)下面這段話 說(shuō)明unifying writes 更加合適， 或者說(shuō)現(xiàn)在主流都是在向 unifying writes 靠攏
Where will these developments take us in the future? If we start from the premise that there is no single data model or storage format that is suitable for all access pat‐ terns, I speculate that there are two avenues by which different storage and process‐ ing tools can nevertheless be composed into a cohesive system:
Federated databases: unifying reads
It is possible to provide a unified query interface to a wide variety of underlying storage engines and processing methods—an approach known as a federated database or polystore [18, 19]. For example, PostgreSQL’s foreign data wrapper feature fits this pattern [20]. Applications that need a specialized data model or query interface can still access the underlying storage engines directly, while users who want to combine data from disparate places can do so easily through the federated interface.
A federated query interface follows the relational tradition of a single integrated system with a high-level query language and elegant semantics, but a compli‐ cated implementation.
Unbundled databases: unifying writes
While federation addresses read-only querying across several different systems, it does not have a good answer to synchronizing writes across those systems. We said that within a single database, creating a consistent index is a built-in feature. When we compose several storage systems, we similarly need to ensure that all data changes end up in all the right places, even in the face of faults. Making it easier to reliably plug together storage systems (e.g., through change data capture and event logs) is like unbundling a database’s index-maintenance features in a way that can synchronize writes across disparate technologies [7, 21].
The unbundled approach follows the Unix tradition of small tools that do one thing well [22], that communicate through a uniform low-level API (pipes), and that can be composed using a higher-level language (the shell) [16].
不對(duì)跟压，Martin focus 在 unifying writes 主要是因?yàn)樗?engineering 更復(fù)雜胰蝠。
Making unbundling work
Federation and unbundling are two sides of the same coin: composing a reliable, scalable, and maintainable system out of diverse components. Federated read-only querying requires mapping one data model into another, which takes some thought but is ultimately quite a manageable problem. I think that keeping the writes to several storage systems in sync is the harder engineering problem, and so I will focus on it.
看來(lái)你需要了解一下 unifying read 是如何實(shí)現(xiàn)的， 因?yàn)樗麄儾恍枰?sync on write?
要讀這個(gè)
http://wp.sigmod.org/?p=1629
[18] Michael Stonebraker: “The Case for Polystores,” wp.sigmod.org, July 13, 2015.
松耦合（loose coupling） 其實(shí)就是讓 individual component do one thing well, 而且它只需要保證接口不變就可以了震蒋， 內(nèi)部如何實(shí)現(xiàn)/優(yōu)化茸塞，別人都不用管
The big advantage of log-based integration is loose coupling between the various com‐ ponents, which manifests itself in two ways:
At a system level, asynchronous event streams make the system as a whole more robust to outages or performance degradation of individual components. If a consumer runs slow or fails, the event log can buffer messages (see “Disk space usage” on page 450), allowing the producer and any other consumers to continue running unaffected. The faulty consumer can catch up when it is fixed, so it doesn’t miss any data, and the fault is contained. By contrast, the synchronous interaction of distributed transactions tends to escalate local faults into large- scale failures (see “Limitations of distributed transactions” on page 363).
At a human level, unbundling data systems allows different software components and services to be developed, improved, and maintained independently from each other by different teams. Specialization allows each team to focus on doing one thing well, with well-defined interfaces to other teams’ systems. Event logs provide an interface that is powerful enough to capture fairly strong consistency properties (due to durability and ordering of events), but also general enough to be applicable to almost any kind of data.
這句話很關(guān)鍵，building for scale you don't needed is wasted effort
吳軍老師在他的專欄也講過(guò)查剖， Google 都是先 Build 10倍于現(xiàn)在的 workload钾虐， 然后再升級(jí)
As I said in the Preface, building for scale that you don’t need is wasted effort and may lock you into an inflexible design. In effect, it is a form of premature optimization.
有空可以看一下現(xiàn)在的 research 方向
Similarly, it would be great to be able to precompute and update caches more easily. Recall that a materialized view is essentially a precomputed cache, so you could imag‐ ine creating a cache by declaratively specifying materialized views for complex quer‐ ies, including recursive queries on graphs (see “Graph-Like Data Models” on page 49) and application logic. There is interesting early-stage research in this area, such as differential dataflow [24, 25], and I hope that these ideas will find their way into pro‐ duction systems.
The approach of unbundling databases by composing specialized storage and pro‐ cessing systems with application code is also becoming known as the “database inside-out” approach [26], after the title of a conference talk I gave in 2014 [27]. However, calling it a “new architecture” is too grandiose. I see it more as a design pattern, a starting point for discussion, and we give it a name simply so that we can better talk about it.
Martin 太謙虛了
These ideas are not mine; they are simply an amalgamation of other people’s ideas from which I think we should learn. In particular, there is a lot of overlap with data‐ flow languages such as Oz [28] and Juttle [29], functional reactive programming (FRP) languages such as Elm [30, 31], and logic programming languages such as Bloom [32]. The term unbundling in this context was proposed by Jay Kreps [7].
真的不是一個(gè)level的……
In the microservices approach, the code that processes the purchase would prob‐ ably query an exchange-rate service or database in order to obtain the current rate for a particular currency.
In the dataflow approach, the code that processes purchases would subscribe to a stream of exchange rate updates ahead of time, and record the current rate in a local database whenever it changes. When it comes to processing the purchase, it only needs to query the local database.
我覺(jué)得micro services 更流行的本質(zhì)原因可能是他是lazy approach， 就跟操作系統(tǒng)里面的思想一樣笋庄， 如果你不需要效扫， 不用eagerly update
咦？ 我想錯(cuò)了无切，network round trip 比我們想象的時(shí)間要長(zhǎng) 所以 eagerly update 速度反而更快荡短？
The second approach has replaced a synchronous network request to another service with a query to a local database (which may be on the same machine, even in the same process).ii Not only is the dataflow approach faster, but it is also more robust to the failure of another service. The fastest and most reliable network request is no net‐ work request at all! Instead of RPC, we now have a stream join between purchase events and exchange rate update events (see “Stream-table join (stream enrichment)” on page 473).
Martin 這里跟我想的不太一樣，可能是我還不夠了解dataflow的思想
Subscribing to a stream of changes, rather than querying the current state when needed, brings us closer to a spreadsheet-like model of computation: when some piece of data changes, any derived data that depends on it can swiftly be updated. There are still many open questions, for example around issues like time-dependent joins, but I believe that building applications around dataflow ideas is a very promis‐ ing direction to go in.
所以Martin 傾向于derived data eagerly done 哆键，也對(duì)掘托，因?yàn)閐erived data 早晚都要process， 什么時(shí)候process 看你app 具體需求了
Taken together, the write path and the read path encompass the whole journey of the data, from the point where it is collected to the point where it is consumed (probably by another human). The write path is the portion of the journey that is precomputed —i.e., that is done eagerly as soon as the data comes in, regardless of whether anyone has asked to see it. The read path is the portion of the journey that only happens when someone asks for it. If you are familiar with functional programming lan‐ guages, you might notice that the write path is similar to eager evaluation, and the read path is similar to lazy evaluation.
看來(lái)也要了解functional programming……
這點(diǎn)很重要籍嘹，以后application 很有可能就是stateful的闪盔，所以整個(gè)架構(gòu)都有可能變化
The huge popularity of web applications in the last two decades has led us to certain assumptions about application development that are easy to take for granted. In par‐ ticular, the client/server model—in which clients are largely stateless and servers have the authority over data—is so common that we almost forget that anything else exists. However, technology keeps moving on, and I think it is important to question the status quo from time to time.
Storm’s distributed RPC feature supports this usage pattern (see “Message passing and RPC” on page 468). For example, it has been used to compute the number of people who have seen a URL on Twitter—i.e., the union of the follower sets of every‐ one who has tweeted that URL [48].
[48] Nathan Marz: “Trident: A High-Level Abstraction for Realtime Computation,” blog.twitter.com, August 2, 2012.
