4 <title>Sixth Data - Data storage and computing engine</title>
5 <!-- 2019-09-22 Sun 11:57 -->
7 <meta name="viewport" content="width=device-width, initial-scale=1">
8 <meta name="generator" content="Org-mode">
9 <meta name="author" content="Svjatoslav Agejenko">
10 <link href="https://bootswatch.com/3/darkly/bootstrap.min.css" rel="stylesheet">
11 <script src="https://cdnjs.cloudflare.com/ajax/libs/jquery/1.11.3/jquery.min.js"></script>
12 <script src="https://cdnjs.cloudflare.com/ajax/libs/twitter-bootstrap/3.3.5/js/bootstrap.min.js"></script>
13 <style type="text/css">
14 footer {background-color: #111 !important;}
15 pre {background-color: #111; color: #ccc;}
17 <style type="text/css">
18 /* org mode styles on top of twbs */
35 background-color: #f5f5f5;
62 display: inline-block;
74 .bs-docs-sidebar.affix {
77 @media (min-width: 768px) {
83 /* All levels of nav */
84 .bs-docs-sidebar .nav > li > a {
91 .bs-docs-sidebar .nav > li > a:hover,
92 .bs-docs-sidebar .nav > li > a:focus {
95 text-decoration: none;
96 background-color: transparent;
97 border-left: 1px solid #A1283B;
99 .bs-docs-sidebar .nav > .active > a,
100 .bs-docs-sidebar .nav > .active:hover > a,
101 .bs-docs-sidebar .nav > .active:focus > a {
105 background-color: transparent;
106 border-left: 2px solid #A1283B;
109 /* Nav: second level (shown on .active) */
110 .bs-docs-sidebar .nav .nav {
111 display: none; /* Hide by default, but at >768px, show it */
112 padding-bottom: 10px;
114 .bs-docs-sidebar .nav .nav > li > a {
121 .bs-docs-sidebar .nav .nav > li > a:hover,
122 .bs-docs-sidebar .nav .nav > li > a:focus {
125 .bs-docs-sidebar .nav .nav > .active > a,
126 .bs-docs-sidebar .nav .nav > .active:hover > a,
127 .bs-docs-sidebar .nav .nav > .active:focus > a {
132 /* Nav: third level (shown on .active) */
133 .bs-docs-sidebar .nav .nav .nav {
134 padding-bottom: 10px;
136 .bs-docs-sidebar .nav .nav .nav > li > a {
143 .bs-docs-sidebar .nav .nav .nav > li > a:hover,
144 .bs-docs-sidebar .nav .nav .nav > li > a:focus {
147 .bs-docs-sidebar .nav .nav .nav > .active > a,
148 .bs-docs-sidebar .nav .nav .nav > .active:hover > a,
149 .bs-docs-sidebar .nav .nav .nav > .active:focus > a {
154 /* Show and affix the side nav when space allows it */
155 @media (min-width: 992px) {
156 .bs-docs-sidebar .nav > .active > ul {
159 /* Widen the fixed sidebar */
160 .bs-docs-sidebar.affix,
161 .bs-docs-sidebar.affix-bottom {
164 .bs-docs-sidebar.affix {
165 position: fixed; /* Undo the static from mobile first approach */
168 .bs-docs-sidebar.affix-bottom {
169 position: absolute; /* Undo the static from mobile first approach */
171 .bs-docs-sidebar.affix .bs-docs-sidenav,.bs-docs-sidebar.affix-bottom .bs-docs-sidenav {
176 @media (min-width: 1200px) {
177 /* Widen the fixed sidebar again */
178 .bs-docs-sidebar.affix-bottom,
179 .bs-docs-sidebar.affix {
184 <script type="text/javascript">
188 $('.bs-docs-sidebar li').first().addClass('active');
190 $(document.body).scrollspy({target: '.bs-docs-sidebar'});
192 $('.bs-docs-sidebar').affix();
197 <div id="content" class="container">
198 <div class="row"><div class="col-md-9"><h1 class="title">Sixth Data - Data storage and computing engine</h1>
200 <div id="outline-container-sec-1" class="outline-2">
201 <h2 id="sec-1"><span class="section-number-2">1</span> General</h2>
202 <div class="outline-text-2" id="text-1">
204 <li>This is a subproject of <a href="https://www3.svjatoslav.eu/projects/sixth/">Sixth</a>
207 <li>This program is free software: you can redistribute it and/or modify
208 it under the terms of the <a href="https://www.gnu.org/licenses/lgpl.html">GNU Lesser General Public License</a> as
209 published by the Free Software Foundation, either version 3 of the
210 License, or (at your option) any later version.
215 <li>Svjatoslav Agejenko
217 <li>Homepage: <a href="https://svjatoslav.eu">https://svjatoslav.eu</a>
219 <li>Email: <a href="mailto://svjatoslav@svjatoslav.eu">mailto://svjatoslav@svjatoslav.eu</a>
224 <li><a href="https://www.svjatoslav.eu/projects/">Other software projects hosted at svjatoslav.eu</a>
229 <div id="outline-container-sec-1-1" class="outline-3">
230 <h3 id="sec-1-1"><span class="section-number-3">1.1</span> Source code</h3>
231 <div class="outline-text-3" id="text-1-1">
233 <li><a href="https://www2.svjatoslav.eu/gitweb/?p=sixth-data.git;a=snapshot;h=HEAD;sf=tgz">Download latest snapshot in TAR GZ format</a>
236 <li><a href="https://www2.svjatoslav.eu/gitweb/?p=sixth-data.git;a=summary">Browse Git repository online</a>
239 <li>Clone Git repository using command:
240 <pre class="example">
241 git clone https://www2.svjatoslav.eu/git/sixth-data.git
246 <li>See <a href="https://www3.svjatoslav.eu/projects/sixth-data/apidocs/">JavaDoc</a>.
253 <div id="outline-container-sec-2" class="outline-2">
254 <h2 id="sec-2"><a id="ID-f6764282-a6f6-44e6-8716-b428074dd093" name="ID-f6764282-a6f6-44e6-8716-b428074dd093"></a><span class="section-number-2">2</span> Vision / goal</h2>
255 <div class="outline-text-2" id="text-2">
257 Provide versioned, clustered, flexible, distributed, multi-dimensional
258 data storage engine for the <a href="http://www2.svjatoslav.eu/gitbrowse/sixth/doc/index.html">Sixth computation engine</a>.
262 <li>Speaking of traditional relational database and object oriented
263 business applications:
266 <li>I hate object-relational impedance mismatch.
269 <li>I don't like to convert data between persistent database and
270 runtime objects for every transaction. How about creating united
271 database/computation engine instead to:
274 <li>Eliminate constant moving and converting of data between 2 systems
275 and make computing happen close to where the data is stored.
278 <li>Abstract away difference between RAM VS persistent storage. Let
279 the system decide at runtime which data to keep in what kind of
288 <div id="outline-container-sec-3" class="outline-2">
289 <h2 id="sec-3"><span class="section-number-2">3</span> Inspiration</h2>
290 <div class="outline-text-2" id="text-3">
292 <li>Relational databases:
296 <li>Indexable / Quickly searchable.
301 <li>Git (version control system)
305 <li>Branchable / mergeable.
307 <li>Transparent cansistency, checksumming and deduplication.
309 <li>(Git as a database:
313 <a href="https://www.kenneth-truyers.net/2016/10/13/git-nosql-database/">https://www.kenneth-truyers.net/2016/10/13/git-nosql-database/</a> )
319 <div id="outline-container-sec-3-1" class="outline-3">
320 <h3 id="sec-3-1"><a id="ID-d2375acc-af14-4f18-8ad0-7949501178c5" name="ID-d2375acc-af14-4f18-8ad0-7949501178c5"></a><span class="section-number-3">3.1</span> Brain</h3>
321 <div class="outline-text-3" id="text-3-1">
323 <li>Brain appears to have more than 3D dimensional design:
324 <a href="https://singularityhub.com/2017/06/21/is-there-a-multidimensional-mathematical-world-hidden-in-the-brains-computation/">https://singularityhub.com/2017/06/21/is-there-a-multidimensional-mathematical-world-hidden-in-the-brains-computation/</a>
327 <li>Brain appears to use geometry to map thoughts and even sounds:
329 <li><a href="https://www.quantamagazine.org/the-brain-maps-out-ideas-and-memories-like-spaces-20190114/">https://www.quantamagazine.org/the-brain-maps-out-ideas-and-memories-like-spaces-20190114/</a>
331 <li><a href="https://www.quantamagazine.org/goals-and-rewards-redraw-the-brains-map-of-the-world-20190328">https://www.quantamagazine.org/goals-and-rewards-redraw-the-brains-map-of-the-world-20190328</a>
336 <li>It directly inspires <a href="#sec-4-1">Geometrical computation</a> idea and nicely fits
337 with <a href="#sec-3-2">CM-1 Connection Machine</a> design.
343 <div id="outline-container-sec-3-2" class="outline-3">
344 <h3 id="sec-3-2"><a id="ID-01aa65c1-3d44-44a8-9b90-58454bc6be80" name="ID-01aa65c1-3d44-44a8-9b90-58454bc6be80"></a><span class="section-number-3">3.2</span> CM-1 Connection Machine</h3>
345 <div class="outline-text-3" id="text-3-2">
347 <a href="https://en.wikipedia.org/wiki/Connection_Machine">https://en.wikipedia.org/wiki/Connection_Machine</a>
351 <li>see: <a href="#sec-4-1">Geometrical computation</a>
354 <li>Computation unit has local CPU and RAM.
357 <li>Data is pre-distributed across computation units.
360 <li>Machine's internal 12-dimensional hypercube network allows to
361 efficiently simulate arbitrary dimensional network topology between
362 computational units. So that when we are solving/simulating for
363 example 5 dimensional problem, we can arrange computational units
364 into virtual 5D network. See:
365 <a href="http://www.mission-base.com/tamiko/theory/cm_txts/di-ch2.html">http://www.mission-base.com/tamiko/theory/cm_txts/di-ch2.html</a>
372 <div id="outline-container-sec-4" class="outline-2">
373 <h2 id="sec-4"><span class="section-number-2">4</span> Ideas</h2>
374 <div class="outline-text-2" id="text-4">
375 </div><div id="outline-container-sec-4-1" class="outline-3">
376 <h3 id="sec-4-1"><a id="ID-171fe375-c737-41e6-b429-a414f6abc5d8" name="ID-171fe375-c737-41e6-b429-a414f6abc5d8"></a><span class="section-number-3">4.1</span> Geometrical computation</h3>
377 <div class="outline-text-3" id="text-4-1">
379 <li>Inspired by <a href="#sec-3-1">Brain</a>.
381 <li>Wits nicely with <a href="#sec-3-2">CM-1 Connection Machine</a> properties.
386 <div id="outline-container-sec-4-1-1" class="outline-4">
387 <h4 id="sec-4-1-1"><a id="ID-5d287158-53ea-44a2-a754-dd862366066a" name="ID-5d287158-53ea-44a2-a754-dd862366066a"></a><span class="section-number-4">4.1.1</span> Distributed computation and data storage</h4>
388 <div class="outline-text-4" id="text-4-1-1">
390 Maybe every problem can be translated to geometry (use any shapes and
391 as many dimensions as you need). Solution(s) to such problems would
392 then appear as relatively simple search/comparison/lookup results. As
393 a bonus, such geometrical *data storage* AND *computation* can be
394 naturally made in *parallel* and *distributed*. That's what neurons in
395 the brain appear to be doing ! :) . Learning means building/updating
396 the model (the hard part). Question answering is making (relatively
397 simple) lookups (geometrical queries) against the model.
401 <div id="outline-container-sec-4-1-2" class="outline-4">
402 <h4 id="sec-4-1-2"><a id="ID-a117c11e-97c1-4822-88b2-9fc10f96caec" name="ID-a117c11e-97c1-4822-88b2-9fc10f96caec"></a><span class="section-number-4">4.1.2</span> Mapping of hyperspace to traditional object-oriented model</h4>
403 <div class="outline-text-4" id="text-4-1-2">
405 Object oriented programming is inspired by the way human mind
406 operates. It allows programmer to express ideas to computer in a more
411 It is possible to map object model to geometrical hyperspace:
415 <li>Object is a point in space (universe). Each object member variable
416 translates to its own dimension. That is: if class declares 4
417 variables for an object, then corresponding object can be stored as
418 a single point inside 4 dimensional space. Variable values translate
419 to point coordinates in space. That is: Integer, floating point
420 number and even boolean and string can be translated to linear value
421 that can be used as a coordinate along particular dimension.
424 <li>Each class declares its own space (universe). All class instances
425 (objects) are points inside that particular universe. References
426 between objects of different types are hyperlinks (portals) between
432 <div id="outline-container-sec-4-1-3" class="outline-4">
433 <h4 id="sec-4-1-3"><a id="ID-b6b15bd2-c78b-4c51-a343-72843a515c29" name="ID-b6b15bd2-c78b-4c51-a343-72843a515c29"></a><span class="section-number-4">4.1.3</span> Handling of relations</h4>
434 <div class="outline-text-4" id="text-4-1-3">
436 Consider we want to create database of books and authors. Book can
437 have multiple authors, and single person can be author for multiple
438 books. It is possible to store how many hours of work each author has
439 contributed to every book, using hyperspace as follows:
443 <li>Every dimension corresponds to one particular book author. (10
444 authors in the database, would require 10 dimensional space)
446 <li>Point in space corresponds to one particular book.
448 <li>Point location along particular (author) dimension corresponds
449 to amount of work contributed by particular author for given
463 <li>Every dimension corresponds to one particular book.
465 <li>Point in space corresponds to one particular author in the entire
468 <li>Point location along particular (book) dimension corresponds to
469 amount of work contributed for book by given author (point).
480 <div id="outline-container-sec-4-2" class="outline-3">
481 <h3 id="sec-4-2"><span class="section-number-3">4.2</span> Layered architecture</h3>
482 <div class="outline-text-3" id="text-4-2">
484 <dt> layer 1 </dt><dd>disk / block storage / partition
487 <dt> layer 2 </dt><dd>key/value storage. Keys are unique and are dictated by
488 storage engine. Value is arbitrary but limited size byte
489 array. This layer is responsible for handling disk
490 defragmentation and consistency in case of crash
494 <dt> layer 3 </dt><dd>key/value storage. Keys are content hashes. Values are
495 arbitrary but limited size content byte arrays. This
496 layer effectively implements content addressable
497 storage. Content addressible storage enables GIT-like
498 behavior (possibility for competing branches, retaining
499 history, transparent deduplication)
502 <dt> layer 4 </dt><dd>Implements arbitrary dimensional multiverse.
505 <dt> layer 5 </dt><dd>Distributed computation engine.
511 <div id="outline-container-sec-5" class="outline-2">
512 <h2 id="sec-5"><span class="section-number-2">5</span> Current status</h2>
513 <div class="outline-text-2" id="text-5">
515 <li>More or less defined <a href="#sec-2">Vision / goal</a>.
518 <li>Collected some <a href="#sec-3-1">ideas</a>.
521 <li>Implemented very simple persistent key-value map.
523 <li>Long term goal is to use it as a backing storage engine and
524 implement more advanced features on top of this.
532 <div id="outline-container-sec-6" class="outline-2">
533 <h2 id="sec-6"><span class="section-number-2">6</span> See also</h2>
534 <div class="outline-text-2" id="text-6">
536 Interesting or competing projects with good ideas:
540 <li><a href="#sec-3-2">CM-1 Connection Machine</a>
545 <li>database in the form of a knowledge graph that uses machine
546 reasoning to simplify data processing challenges for AI
547 applications. <a href="https://grakn.ai/">https://grakn.ai/</a>
552 <li><a href="http://wiki.squeak.org/squeak/2665">Magma</a>
554 <li>Multi-user object database for Squeak
559 <li><a href="http://esug.org/data/ESUG2015/3%20wednesday/1100-1130%20SQL%20Queries%20on%20Smalltalk%20Objects/SQL%20Queries%20in%20Smalltalk%20(James%20Foster).pdf">Gemstone/S</a>
561 <li>Completely distributed smalltalk based computing
567 <li><a href="http://www.uruk.org/emu/Taos.html">TAOS</a>
569 <li>Completely distributed operating system/virtual machine:
574 <li><a href="https://github.com/vygr/ChrysaLisp">ChrysaLisp</a>
576 <li>Assembler/C-Script/Lisp 64 bit, MIMD, multi CPU, multi threaded,
577 multi core, multi user Parallel OS. With GUI, Terminal, OO
578 Assembler, Class libraries, C-Script compiler, Lisp interpreter,
579 Debugger, and more…
586 </div><div class="col-md-3"><nav id="table-of-contents">
587 <div id="text-table-of-contents" class="bs-docs-sidebar">
589 <li><a href="#sec-1">1. General</a>
591 <li><a href="#sec-1-1">1.1. Source code</a></li>
594 <li><a href="#sec-2">2. Vision / goal</a></li>
595 <li><a href="#sec-3">3. Inspiration</a>
597 <li><a href="#sec-3-1">3.1. Brain</a></li>
598 <li><a href="#sec-3-2">3.2. CM-1 Connection Machine</a></li>
601 <li><a href="#sec-4">4. Ideas</a>
603 <li><a href="#sec-4-1">4.1. Geometrical computation</a></li>
604 <li><a href="#sec-4-2">4.2. Layered architecture</a></li>
607 <li><a href="#sec-5">5. Current status</a></li>
608 <li><a href="#sec-6">6. See also</a></li>
613 <footer id="postamble" class="">
614 <div><p class="author">Author: Svjatoslav Agejenko</p>
615 <p class="date">Created: 2019-09-22 Sun 11:57</p>
616 <p class="creator"><a href="http://www.gnu.org/software/emacs/">Emacs</a> 26.1 (<a href="http://orgmode.org">Org-mode</a> 9.1.9)</p>