Notes

tags: Bigdata

tags: Bigdata,Streaming,Stateful Stream Processing source: Kreps, Jay. “Why Local State Is a Fundamental Primitive in Stream Processing - O’Reilly Radar.” Accessed January 5, 2022. http://radar.oreilly.com/2014/07/why-local-state-is-a-fundamental-primitive-in-stream-processing.html. Why local state is much faster than a distribut database. local state can easily restore by some middleware like Kafka: by writing changes to a Kafka topic.

tags: Bigdata,Flink,Dataflow Model,Streaming source: “Streaming 102: The World beyond Batch – O’Reilly.” Accessed January 5, 2022. https://www.oreilly.com/radar/the-world-beyond-batch-streaming-102/. Three more concepts: Watermarks: Useful for event time windowing. All input data with event times less than watermark have been observed. Triggers: Signal for a window to produce output. Accumulation: The way to handle multiple results that are observed for the same window. Streaming 101 Redux What: Transformations Where: windowing Make a temporal boundary for a unbounded data source. ...

tags: Bigdata,Streaming source: Akidau, Tyler, Robert Bradshaw, Craig Chambers, Slava Chernyak, Rafael J. Fernández-Moctezuma, Reuven Lax, Sam McVeety, et al. “The Dataflow Model: A Practical Approach to Balancing Correctness, Latency, and Cost in Massive-Scale, Unbounded, out-of-Order Data Processing.” Proceedings of the VLDB Endowment 8, no. 12 (August 2015): 1792–1803. https://doi.org/10.14778/2824032.2824076.

tags: Bigdata,Flink,Streaming source: Akidau, Tyler. “Streaming 101: The World beyond Batch.” O’Reilly Media, August 5, 2015. https://www.oreilly.com/radar/the-world-beyond-batch-streaming-101/. Streaming: a type of data processing engine that is designed with infinite data sets in mind. Other common uses of “streaming” that will be avoid in the rest of the post: Unbounded data: A type of ever-growing, essentially infinite data set. Unbounded data processing: An ongoing mode of data processing, applied to the aforementioned type of unbounded data. Low-latency, approximate, and/or speculative results: These types of results are most often associated with streaming engines. Limitations of streaming To beat batch at its own game, you really only need two things: ...

tags: DAO,Web3 source: Foundation, Ethereum. “DAOs, DACs, DAs and More: An Incomplete Terminology Guide.” Accessed January 5, 2022. https://blog.ethereum.org/2014/05/06/daos-dacs-das-and-more-an-incomplete-terminology-guide/.

tags: DAO,Online Tutorial source: https://aragon.org/dao

tags: Freelance source: Dan. “My Writing Finances, 2021 | Dan Moren.” Accessed January 5, 2022. https://dmoren.com/2022/01/03/my-writing-finances-2021/.

tags: Freelance source: “I Make $3K/Mo from a Browser Extension! (+ December 2021 Updates).” Accessed January 5, 2022. http://newsletter.tonydinh.com/issues/i-make-3k-mo-from-a-browser-extension-december-2021-updates-966892. HN: https://news.ycombinator.com/item?id=29803817

tags: Web3 source: Continuations by Albert Wenger. “Web3/Crypto: Why Bother?” Accessed January 5, 2022. https://continuations.com/post/671863718643105792/web3crypto-why-bother.

tags: Ethereum,MetaMask,Online Tools source: https://www.skiff.org/updates/skiff-ens

tags: Web3,Online Tools,Real Problems That Web3 Solves, Part 1 source: https://metamask.io/ MetaMask has implemented private key login via a Chrome extension and now has 21 million users. In theory, something like this could have existed a long time ago. In practice, the existence of blockchains and the need for better UX for cryptocurrency incentivized much more investmenets in these tools.

tags: Web3,Smart contracts source: Bill Prin’s Personal Page. “Real Problems That Web3 Solves, Part 1,” January 3, 2022. https://billprin.com/2022/01/03/real-problems-web3-solves.html. What exactly is the difference between Web3, blockchain, and cryptocurrency You can think of blockchain and cryptocurrency as technological implementation details, and Web3 as the communities, businesses, and social relationships that form on top of that technology. A similar analogy would be the original World Wide Web, which could have been construed as a rebrand of the underlying technologies of HTML over HTTP over TCP/IP. Those protocols have served as the foundation for virtually all web content for the last 30 years, but it’d certainly be unfair to describe “the Web” as merely a rebranding of HTML/HTTP/TCP/IP since what happens on top of those protocols is much bigger than just the protocols themselves. ...

tags: Blockchain,Ethereum source: https://ethereum.org/en/developers/docs/smart-contracts/ A type of Ethereum account. Some code deployed and running in some VM, like Ethereum Virtual Machine.

tags: Ethereum,Smart contracts,Blockchain,Web3 source: https://ethereum.org/en/dao/ Based on smart contracts and use tokens for voting. Tokens can be exchanged in the market.

tags: AI,Deep Learning source: “Neural Network From Scratch.” Accessed January 5, 2022. https://sirupsen.com/napkin/neural-net.

tags: Taking Notes,RoamResearch source: “My Personal Note Taking Journey.” Accessed January 5, 2022. https://phonetonote.com/blog/my-personal-note-taking-journey/.

tags: Zotero,Emacs I got an error when I’m inserting Zotero ref to Emacs by M-x org-zotxt-insert-reference-link RET [error] request--callback: peculiar error: 500 I got the error of zotxt by follow the instruction Debug Output Logging: (5)(+0000003): HTTP/1.0 500 Internal Server Error X-Zotero-Version: 5.0.96.3 X-Zotero-Connector-API-Version: 2 Content-Type: text/plain; charset=UTF-8 csl is nullTypeError: csl is null buildBibliographyResponse/responseData<@resource://gre/modules/addons/XPIProvider.jsm -> jar:file:///Users/wanghui/Library/Application%20Support/Zotero/Profiles/34hkbjfm.default/extensions/zotxt@e6h.org.xpi!/bootstrap.js:220:9 buildBibliographyResponse@resource://gre/modules/addons/XPIProvider.jsm -> jar:file:///Users/wanghui/Library/Application%20Support/Zotero/Profiles/34hkbjfm.default/extensions/zotxt@e6h.org.xpi!/bootstrap.js:219:24 buildResponse/<@resource://gre/modules/addons/XPIProvider.jsm -> jar:file:///Users/wanghui/Library/Application%20Support/Zotero/Profiles/34hkbjfm.default/extensions/zotxt@e6h.org.xpi!/bootstrap.js:156:20 tryCatcher@resource://zotero/loader.jsm -> resource://zotero/bluebird/util.js:16:16 module.exports/Promise.prototype._settlePromiseFromHandler@resource://zotero/loader.jsm -> resource://zotero/bluebird/promise.js:547:13 module.exports/Promise.prototype._settlePromise@resource://zotero/loader.jsm -> resource://zotero/bluebird/promise.js:604:13 module.exports/Promise.prototype._settlePromise0@resource://zotero/loader.jsm -> resource://zotero/bluebird/promise.js:649:5 module.exports/Promise.prototype._settlePromises@resource://zotero/loader.jsm -> resource://zotero/bluebird/promise.js:729:13 module.exports/Promise.prototype._fulfill@resource://zotero/loader.jsm -> resource://zotero/bluebird/promise.js:673:13 module.exports/PromiseArray.prototype._resolve@resource://zotero/loader.jsm -> resource://zotero/bluebird/promise_array.js:127:5 module.exports/MappingPromiseArray.prototype._promiseFulfilled@resource://zotero/loader.jsm -> resource://zotero/bluebird/map.js:108:13 module.exports/Promise.prototype._settlePromise@resource://zotero/loader.jsm -> resource://zotero/bluebird/promise.js:609:17 module.exports/Promise… (3613 chars) And I found this issue: zotxt api does not work for non-latin references. By looking at the reply, I think it’s the same issue I’m got: I did’nt install the style which I setted in my Emacs config: ...

tags: Org Mode,Taking Notes,Zotero,Research,Emacs source: “Scientific Writing with Zotero and Org Mode | Irreal.” Accessed January 5, 2022. https://irreal.org/blog/?p=5771.

tags: Org Mode,Taking Notes,Zotero,Research,Emacs source: “A Research Workflow with Zotero and Org Mode | Mkbehr.Com.” Accessed January 5, 2022. http://www.mkbehr.com/posts/a-research-workflow-with-zotero-and-org-mode/. Gluing zotero and Org mode together with zotxt(zotxt-emacs). Workflow: Store papers into zotero by its browser plugin, that may also download the PDF. Create a page in Emacs and link to zotero via zotxt-emacs C-c " ". When I want to read the paper. Go to the page in Emacs and type C-c " a. When I’m reading a paper and see a citation that might be useful, I look it up on the internet and repeat this process to store a note linking to it.

tags: Tools,Learning,Taking Notes

tags: Rust,优化,High Performance source: https://blog.datalust.co/deserializing-json-really-fast/

tags: TCP source: https://arthurchiao.art/blog/rfc1180-a-tcp-ip-tutorial-zh/

tags: Assembly,NASM Assembly Language Tutorials source: https://ratfactor.com/assembly-nights

tags: Web3 source: https://blog.wesleyac.com/posts/web3-centralized

tags: Algorithm source: https://malisper.me/an-algorithm-for-passing-programming-interviews/

tags: Blockchain,Web3 source: https://www.kooslooijesteijn.net/blog/web3

tags: Blockchain

tags: Go source: https://tip.golang.org/doc/fuzz/

tags: Freelance source: https://www.hellobonsai.com/ HN: https://news.ycombinator.com/item?id=29782097

tags: Computer Systems

tags: Computer Systems,Assembly,Linux,Online Tutorial source: “NASM Assembly Language Tutorials - Asmtutor.Com.” Accessed January 5, 2022. https://asmtutor.com/.

tags: Freelance

tags: Freelance,Microstartup source: https://news.ycombinator.com/item?id=29790964 Comments: Related: “Tell HN: My Microstartups make $500/day while I’m sleeping” (this): https://news.ycombinator.com/item?id=29790964 “AMA: I make $100K+ ARR from my microstartups” (3 months ago): https://news.ycombinator.com/item?id=28561132 “Show HN: I passed up an opportunity to make $200K from my microstartup” (2020): https://twitter.com/1HaKr/status/1301142901510995969 “Show HN: My Indie Hacker goal - Earn $100 a day to keep your desk job away” (2020): https://news.ycombinator.com/item?id=24304674 “Show HN: I made $9000 posting on Hacker News about my microstartup” (2020): https://news.ycombinator.com/submitted?id=1hakr ...

tags: Blockchain

tags: Privoxy,Over the Wall source: https://wiki.archlinux.org/title/Privoxy%5F(%E7%AE%80%E4%BD%93%E4%B8%AD%E6%96%87)#%E8%BD%AC%E5%8F%91%E5%8D%8F%E8%AE%AE

tags: Tools,Unix home: https://www.privoxy.org/

tags: Bigdata source: https://beam.apache.org/

tags: Flink State Snapshots source: https://nightlies.apache.org/flink/flink-docs-release-1.14/docs/concepts/stateful-stream-processing/#keyed-state Keyed state is maintained in what can be thought of as an embedded key/value store.

tags: Flink State Snapshots,Fault Tolerance via State Snapshots source: https://nightlies.apache.org/flink/flink-docs-release-1.14/docs/learn-flink/fault%5Ftolerance/#exactly-once-guarantees Depending on the choices you make, Flink possiable outcomes: Flink makes no effort to recover from failures (at most once) Nothing is lost, but you may experience duplicated results (at least once) Nothing is lost or duplicated (exactly once) Given that Flink recovers from faults by rewinding and replaying the source data streams, when the ideal situation is described as exactly once this does not mean that every event will be processed exactly once. Instead, it means that every event will affect the state being managed by Flink exactly once. ...

tags: 分布式 source: https://en.wikipedia.org/wiki/Chandy%E2%80%93Lamport%5Falgorithm

tags: Fault Tolerance via State Snapshots,Flink State Snapshots,Wikipedia: Chandy–Lamport algorithm source: https://nightlies.apache.org/flink/flink-docs-release-1.14/docs/learn-flink/fault%5Ftolerance/#how-does-state-snapshotting-work Workflow: Checkpoint coordinator (part of the job manager) instructs a task manager to begin a checkpoint. Insert numbered checkpoint barriers into their streams of all the sources record their offsets. checkpoint barriers flow through the job graph, indicating the part of the stream before and after each checkpoint. Checkpoint n will contain the state of each operator that resulted from having consumed every event before checkpoint barrier n, and none of the events after it. ...

tags: Flink State Snapshots,Fault Tolerance via State Snapshots a snapshot taken automatically by Flink for the purpose of being able to recover from faults. Checkpoints can be incremental, and are optimized for being restored quickly.

tags: Flink State Snapshots a snapshot triggered manually by a user (or an API call) for some operational purpose, such as a stateful redeploy/upgrade/rescaling operation. Savepoints are always complete, and are optimized for operational flexibility.

tags: Flink State Snapshots,Fault Tolerance via State Snapshots,Flink Checkpoint source: https://nightlies.apache.org/flink/flink-docs-release-1.14/docs/learn-flink/fault%5Ftolerance/#checkpoint-storage Flink periodically takes persistent snapshots of all the state in every operator and copies these snapshots somewhere more durable, such as a distributed file system. In the event of the failure, Flink can restore the complete state of your application and resume processing as though nothing had gone wrong. Two implementations: A distributed file system. JobManager’s heap.