Notes

tags: Flink State Snapshots,Fault Tolerance via State Snapshots,Stateful Stream Processing Two implementations of state backends are available: RocksDB An embedded key/value store keeps its working state on disk. Overhead Accesses and updates involve serialization and deserialization. Java heap-based state backend Keeps its working state in memory, on the Java heap. Risk Large amount state will cause OOM. Conclusion Both of these state backends are able to do asynchronous snapshotting, meaning that they can take a snapshot without impeding the ongoing stream processing. ...

tags: Flink State Snapshots,Stateful Stream Processing source: https://nightlies.apache.org/flink/flink-docs-release-1.14/docs/learn-flink/fault%5Ftolerance/

tags: Stateful Stream Processing

tags: Stream processing,Flink source: https://nightlies.apache.org/flink/flink-docs-release-1.14/docs/learn-flink/overview/#stateful-stream-processing https://nightlies.apache.org/flink/flink-docs-release-1.14/docs/concepts/stateful-stream-processing/ This means that how one event is handled can depend on the accumulated effect of all the events that came before it. How the stateful streaming processing works on a distributed cluster? The set of parallel instances of a stateful operator is effectively a sharded key-value store. Each parallel instance is responsible for handling events for a specific group of keys, and the state for those keys is kept locally. ...

tags: Stream processing,Flink source: https://nightlies.apache.org/flink/flink-docs-release-1.14/docs/learn-flink/overview/#timely-stream-processing Flink timely stream processing support by using event timestamps that are recorded in data stream, rather than using the clocks of the machines processing the data.

tags: Flink Parallel Dataflows Redistributing streams (as between map() and keyBy/window above, as well as between keyBy/window and Sink) change the partitioning of streams. Each operator subtask sends data to different target subtasks, depending on the selected transformation. Examples are keyBy() (which re-partitions by hashing the key), broadcast(), or rebalance() (which re-partitions randomly). In a redistributing exchange the ordering among the elements is only preserved within each pair of sending and receiving subtasks (for example, subtask[1] of map() and subtask[2] of keyBy/window). So, for example, the redistribution between the keyBy/window and the Sink operators shown above introduces non-determinism regarding the order in which the aggregated results for different keys arrive at the Sink. ...

tags: Flink Parallel Dataflows One-to-one streams (for example between the Source and the map() operators in the figure above) preserve the partitioning and ordering of the elements. That means that subtask[1] of the map() operator will see the same elements in the same order as they were produced by subtask[1] of the Source operator.

tags: Flink Streams can transport data between two operators in a one-to-one (or forwarding) pattern, or in a redistributing pattern:

tags: Flink Stream processing, on the other hand, involves unbounded data streams. Conceptually, at least, the input may never end, and so you are forced to continuously process the data as it arrives.

tags: Spark Batch processing is the paradigm at work when you process a bounded data stream. In this mode of operation you can choose to ingest the entire dataset before producing any results, which means that it is possible, for example, to sort the data, compute global statistics, or produce a final report that summarizes all of the input.

tags: Flink,Flink State Snapshots,Flink Checkpoint,Flink Savepoint source: https://zhuanlan.zhihu.com/p/79526638

tags: Linux source: https://github.com/sunym1993/flash-linux0.11-talk

tags: Linux source: https://archive.org/details/199405-decusnew-orleans/1994050DECUSNewOrleansLinuxImplementationIssuesInLinux.ogg

tags: Operating system

tags: Ethereum,Proof-of-stake source: https://ethereum.org/en/eth2/shard-chains/ Sharding is the process of splitting a database horizontally to spread the load – it’s a common concept in computer science. In an Ethereum context, sharding will reduce network congestion and increase transactions per second by creating new chains, known as “shards”. This is important for reasons other than scalability.

tags: Ethereum,Proof-of-stake source: https://ethereum.org/en/eth2/beacon-chain/ Extra coordination for the Ethereum: Shard chains. The beacon chain receives state information from shards and makes it available for other shards, allowing the network to stay in sync. The beacon chain will also manage the validators from registering their stake deposits to issuing their rewards and penalties.

tags: Ethereum,Proof-of-stake source: https://ethereum.org/en/developers/docs/consensus-mechanisms/pos/#how-does-pos-work When you submit a transaction on a shard, a validator will be responsible for adding your transaction to a shard block. Validators are algorithmically chosen by Ethereum: The Beacon Chain to propose new blocks. Attestation If a validator isn’t chosen to propose a new shard block, they’ll have to attest to another validator’s proposal and confirm that everything looks as it should. It’s the attestation that is recorded in the beacon chain rather than the transaction itself. ...

tags: Ethereum

tags: Blockchain Proof,Solana,Proof-of-stake Solana is a Proof of Stake network. This short phrase - “Proof of Stake” - represents a much larger concept with considerable complexity behind it, and even more so for Solana, which adds the unique properties of Proof of History to the mix to enable fast, low-latency transactions while still maintaining censorship resistance.

tags: Blockchain,Blockchain Proof,Ethereum,Solana source: https://ethereum.org/en/developers/docs/consensus-mechanisms/pos/ Proof workflow: Users stake money(ETH) to become a validator. Validators are chosen at random to create blocks and are responsible for checking and confirming blocks they don’t create. user’s stake is also used as a way to incentivise good validator behavior. For example, a user can lose a portion of their stake for things like going offline (failing to validate) or their entire stake for deliberate collusion. ...

tags: Blockchain Proof,Blockchain,Ethereum source: https://ethereum.org/en/developers/docs/consensus-mechanisms/pow/ Wikipedia: https://en.wikipedia.org/wiki/Proof%5Fof%5Fwork A key feature of proof-of-work schemes is their asymmetry: the work – the computation – must be moderately hard (yet feasible) on the prover or requester side but easy to check for the verifier or service provider. With a hash function, let’s say SHA-1. For example, to do PoW, we need to generate a SHA-1 hash of the given data that must begins 52 binary zeros, that is 13 hexadecimal zeros: ...

tags: Blockchain

tags: Blockchain,Solana,Proof-of-stake,Proof-of-history source: “Shinobi Systems’ Solana Proof of Stake + Proof of History Primer.” Accessed January 5, 2022. https://www.shinobi-systems.com/primer.html.

tags: Video: Blockchain 101 - A Visual Demo，区块链, Online Tools source: https://andersbrownworth.com/blockchain/hash

tags: 区块链 source: https://youtu.be/%5F160oMzblY8 It’s like Git but not support merge. The progress of changing blocks like git rebase.

tags: C/C++ 只能通过抽象类的指针或引用调用动态解析子类函数，虚函数表示需要动态解析，纯虚函数必须被子类覆盖，否则无法实例化。

tags: Programming Language

tags: Programming Language

tags: Computer Systems,读书笔记

tags: Rust source: https://stackoverflow.com/a/34745885/2873718

tags: Rust Wrapper Types,Rust source: Rust Memory Container Cheat-sheet

tags: Rust,Rust Wrapper Types source: https://manishearth.github.io/blog/2015/05/27/wrapper-types-in-rust-choosing-your-guarantees/

tags: Tokio source: https://github.com/tony612/tokio-internals

tags: Raft source: https://web.stanford.edu/~ouster/cgi-bin/papers/OngaroPhD.pdf

tags: 分布式,一致性 source: https://lamport.azurewebsites.net/pubs/time-clocks.pdf The Big Problem The event order in a distributed system. The defination of a distributed system: the order of its events occurred is unpredictable, sometimes it’s impossiable to say a event occurred before another. How this paper try to solve this problem: use a “happened before” relation to define a partial ordering and distributed algorithm. Background

tags: Paxos source: https://blog.csdn.net/qq%5F35440678/article/details/78080431

tags: Rust,Raft source: https://github.com/tikv/raft-rs

tags: Raft,分布式，分布式共识，Online Tools source: http://thesecretlivesofdata.com/raft/

tags: 分布式共识,Online Tools,区块链 source: https://web3scout.github.io/forcecons-sim/

tags: org-roam, Org Mode source: https://seds.nl/notes/org%5Froam%5Fexport%5Fbacklinks%5Fon%5Fhugo/ https://seds.nl/notes/export%5Forg%5Froam%5Fbacklinks%5Fwith%5Fgohugo/ 利用 hugo 的 partial template layouts/partials/backlinks.html {{ $re := $.File.BaseFileName }} {{ $backlinks := slice }} {{ range .Site.AllPages }} {{ if and (findRE $re .RawContent) (not (eq $re .File.BaseFileName)) }} {{ $backlinks = $backlinks | append . }} {{ end }} {{ end }} <hr> {{ if gt (len $backlinks) 0 }} <div class="bl-section"> <h4>Links to this note</h4> <div class="backlinks"> <ul> {{ range $backlinks }} <li><a href="{{ .RelPermalink }}">{{ .Title }}</a></li> {{ end }} </ul> </div> </div> {{ else }} <div class="bl-section"> <h4>No notes link to this note</h4> </div> {{ end }} 然后插入到的 single.html 就行 ...

tags: Tools

tags: Taking Notes,Online Tools

tags: Learning,Taking Notes source: https://www.nateliason.com/blog/roam

tags: Learning,Taking Notes,RoamResearch source: https://fortelabs.co/blog/how-to-take-smart-notes/ Luhmann’s slip-box: build second brain context – its network of associations, relationships, and connections to other information. But Luhmann often remarked that he never forced himself to do anything he didn’t feel like doing: “I only do what is easy. I only write when I immediately know how to do it. If I falter for a moment, I put the matter aside and do something else” (Luhmann et al., 1987, 154f). ...

tags: Learning,Taking Notes,org-roam,Org Mode source: https://blog.jethro.dev/posts/how%5Fto%5Ftake%5Fsmart%5Fnotes%5Forg/ Notes aren’t a record of my thinking process. They are my thinking process. – Richard Feynman The primary purpose of note-taking should not be for storing ideas, but for developing them. When we take notes, we should ask: “In what context do I want to see this note again?” Note-taking for writing: Find topic/research question Research/find literature Read and take notes Draw conclusions / outline text Write Two types of notes: ...

证书 [译] 写给工程师：关于证书（certificate）和公钥基础设施（PKI）的一切（SmallStep, 2018）

tags: Learning English 一般现在时第三人称单音形规则 一般过去时音形规则