4 <title>Sixth Data - Data storage and computing engine</title>
5 <!-- 2020-01-29 K 22:47 -->
7 <meta name="viewport" content="width=device-width, initial-scale=1">
8 <meta name="generator" content="Org-mode">
9 <meta name="author" content="Svjatoslav Agejenko">
10 <link href="https://bootswatch.com/3/darkly/bootstrap.min.css" rel="stylesheet">
11 <script src="https://cdnjs.cloudflare.com/ajax/libs/jquery/1.11.3/jquery.min.js"></script>
12 <script src="https://cdnjs.cloudflare.com/ajax/libs/twitter-bootstrap/3.3.5/js/bootstrap.min.js"></script>
13 <style type="text/css">
14 footer {background-color: #111 !important;}
15 pre {background-color: #111; color: #ccc;}
17 <style type="text/css">
18 /* org mode styles on top of twbs */
35 background-color: #f5f5f5;
62 display: inline-block;
74 .bs-docs-sidebar.affix {
77 @media (min-width: 768px) {
83 /* All levels of nav */
84 .bs-docs-sidebar .nav > li > a {
91 .bs-docs-sidebar .nav > li > a:hover,
92 .bs-docs-sidebar .nav > li > a:focus {
95 text-decoration: none;
96 background-color: transparent;
97 border-left: 1px solid #A1283B;
99 .bs-docs-sidebar .nav > .active > a,
100 .bs-docs-sidebar .nav > .active:hover > a,
101 .bs-docs-sidebar .nav > .active:focus > a {
105 background-color: transparent;
106 border-left: 2px solid #A1283B;
109 /* Nav: second level (shown on .active) */
110 .bs-docs-sidebar .nav .nav {
111 display: none; /* Hide by default, but at >768px, show it */
112 padding-bottom: 10px;
114 .bs-docs-sidebar .nav .nav > li > a {
121 .bs-docs-sidebar .nav .nav > li > a:hover,
122 .bs-docs-sidebar .nav .nav > li > a:focus {
125 .bs-docs-sidebar .nav .nav > .active > a,
126 .bs-docs-sidebar .nav .nav > .active:hover > a,
127 .bs-docs-sidebar .nav .nav > .active:focus > a {
132 /* Nav: third level (shown on .active) */
133 .bs-docs-sidebar .nav .nav .nav {
134 padding-bottom: 10px;
136 .bs-docs-sidebar .nav .nav .nav > li > a {
143 .bs-docs-sidebar .nav .nav .nav > li > a:hover,
144 .bs-docs-sidebar .nav .nav .nav > li > a:focus {
147 .bs-docs-sidebar .nav .nav .nav > .active > a,
148 .bs-docs-sidebar .nav .nav .nav > .active:hover > a,
149 .bs-docs-sidebar .nav .nav .nav > .active:focus > a {
154 /* Show and affix the side nav when space allows it */
155 @media (min-width: 992px) {
156 .bs-docs-sidebar .nav > .active > ul {
159 /* Widen the fixed sidebar */
160 .bs-docs-sidebar.affix,
161 .bs-docs-sidebar.affix-bottom {
164 .bs-docs-sidebar.affix {
165 position: fixed; /* Undo the static from mobile first approach */
168 .bs-docs-sidebar.affix-bottom {
169 position: absolute; /* Undo the static from mobile first approach */
171 .bs-docs-sidebar.affix .bs-docs-sidenav,.bs-docs-sidebar.affix-bottom .bs-docs-sidenav {
176 @media (min-width: 1200px) {
177 /* Widen the fixed sidebar again */
178 .bs-docs-sidebar.affix-bottom,
179 .bs-docs-sidebar.affix {
184 <script type="text/javascript">
188 $('.bs-docs-sidebar li').first().addClass('active');
190 $(document.body).scrollspy({target: '.bs-docs-sidebar'});
192 $('.bs-docs-sidebar').affix();
197 <div id="content" class="container">
198 <div class="row"><div class="col-md-9"><h1 class="title">Sixth Data - Data storage and computing engine</h1>
200 <div id="outline-container-sec-1" class="outline-2">
201 <h2 id="sec-1"><span class="section-number-2">1</span> General</h2>
202 <div class="outline-text-2" id="text-1">
204 <li>This is a subproject of <a href="https://www3.svjatoslav.eu/projects/sixth/">Sixth</a> with the goal of providing data
205 storage and computation facilities.
208 <li>This program is free software: you can redistribute it and/or modify
209 it under the terms of the <a href="https://www.gnu.org/licenses/lgpl.html">GNU Lesser General Public License</a> as
210 published by the Free Software Foundation, either version 3 of the
211 License, or (at your option) any later version.
216 <li>Svjatoslav Agejenko
218 <li>Homepage: <a href="https://svjatoslav.eu">https://svjatoslav.eu</a>
220 <li>Email: <a href="mailto://svjatoslav@svjatoslav.eu">mailto://svjatoslav@svjatoslav.eu</a>
225 <li><a href="https://www.svjatoslav.eu/projects/">Other software projects hosted at svjatoslav.eu</a>
230 <div id="outline-container-sec-1-1" class="outline-3">
231 <h3 id="sec-1-1"><span class="section-number-3">1.1</span> Source code</h3>
232 <div class="outline-text-3" id="text-1-1">
234 <li><a href="https://www2.svjatoslav.eu/gitweb/?p=sixth-data.git;a=snapshot;h=HEAD;sf=tgz">Download latest snapshot in TAR GZ format</a>
237 <li><a href="https://www2.svjatoslav.eu/gitweb/?p=sixth-data.git;a=summary">Browse Git repository online</a>
240 <li>Clone Git repository using command:
241 <pre class="example">
242 git clone https://www2.svjatoslav.eu/git/sixth-data.git
251 <div id="outline-container-sec-2" class="outline-2">
252 <h2 id="sec-2"><a id="ID-f6764282-a6f6-44e6-8716-b428074dd093" name="ID-f6764282-a6f6-44e6-8716-b428074dd093"></a><span class="section-number-2">2</span> Vision / goal</h2>
253 <div class="outline-text-2" id="text-2">
255 Provide hackable, versioned, optimized, distributed, geometrical,
256 arbitrary dimensional (<a href="#sec-4">hypercube based</a>) data storage and computation
257 engine (<a href="#sec-3-1">as inspired by the brain</a>) for general purpose visual computing
258 environment called <a href="http://www2.svjatoslav.eu/gitbrowse/sixth/doc/index.html">Sixth</a>.
262 Because <a href="http://www.paulgraham.com/rootsoflisp.html">Lisp is hackable self defined programmable programming
263 language</a> it would be used to provide <a href="https://en.wikipedia.org/wiki/Imperative_programming">imperative programming</a> support.
267 <div id="outline-container-sec-3" class="outline-2">
268 <h2 id="sec-3"><a id="ID-0fa6354b-18c9-4120-bbf5-c7239aebecab" name="ID-0fa6354b-18c9-4120-bbf5-c7239aebecab"></a><span class="section-number-2">3</span> Inspiration</h2>
269 <div class="outline-text-2" id="text-3">
271 <li>see also: <a href="https://en.wikipedia.org/wiki/OLAP_cube">OLAP cube</a>.
275 <div id="outline-container-sec-3-1" class="outline-3">
276 <h3 id="sec-3-1"><a id="ID-d2375acc-af14-4f18-8ad0-7949501178c5" name="ID-d2375acc-af14-4f18-8ad0-7949501178c5"></a><span class="section-number-3">3.1</span> Brain</h3>
277 <div class="outline-text-3" id="text-3-1">
279 <li>Brain appears to be natural geometrical/parallel data storage and
280 computational engine:
282 <li><a href="https://www.quantamagazine.org/the-brain-maps-out-ideas-and-memories-like-spaces-20190114/">https://www.quantamagazine.org/the-brain-maps-out-ideas-and-memories-like-spaces-20190114/</a>
287 <li>Even more awesome is that brain appears to operate and is wired as
288 arbitrary/variable dimensional structure:
289 <a href="https://singularityhub.com/2017/06/21/is-there-a-multidimensional-mathematical-world-hidden-in-the-brains-computation/">https://singularityhub.com/2017/06/21/is-there-a-multidimensional-mathematical-world-hidden-in-the-brains-computation/</a>
292 <li>On top of this, this multidimensional space that brain represents
293 has dynamic/variable resolution/density:
295 <li><a href="https://www.quantamagazine.org/goals-and-rewards-redraw-the-brains-map-of-the-world-20190328">https://www.quantamagazine.org/goals-and-rewards-redraw-the-brains-map-of-the-world-20190328</a>
300 <li>Such properties allow parallel <a href="#sec-5">Geometrical computation</a> and
301 beautifully fits <a href="#sec-3-2">CM-1 Connection Machine</a> architecture (for extra
302 hardware accelerated solution).
308 <div id="outline-container-sec-3-2" class="outline-3">
309 <h3 id="sec-3-2"><a id="ID-01aa65c1-3d44-44a8-9b90-58454bc6be80" name="ID-01aa65c1-3d44-44a8-9b90-58454bc6be80"></a><span class="section-number-3">3.2</span> CM-1 Connection Machine</h3>
310 <div class="outline-text-3" id="text-3-2">
312 <a href="https://en.wikipedia.org/wiki/Connection_Machine">https://en.wikipedia.org/wiki/Connection_Machine</a>
316 Massively parallel (thousands of CPUs) connected via
317 machine's internal 12-dimensional hypercube network allows to
318 efficiently simulate arbitrary dimensional hypercube and network
319 topology between computational units. So that when we are
320 solving/simulating for example 5 dimensional problem, we can arrange
321 computational units into virtual 5D network. See:
322 <a href="http://www.mission-base.com/tamiko/theory/cm_txts/di-ch2.html">http://www.mission-base.com/tamiko/theory/cm_txts/di-ch2.html</a>
326 we can pre-distribute data across computation units and perform
327 parallel <a href="#sec-5">geometrical computation</a>.
333 <div id="outline-container-sec-4" class="outline-2">
334 <h2 id="sec-4"><a id="ID-96116550-a6a1-4700-bef7-865d0deee7ea" name="ID-96116550-a6a1-4700-bef7-865d0deee7ea"></a><span class="section-number-2">4</span> Reasons for hypercube as a so called first class citizen</h2>
335 <div class="outline-text-2" id="text-4">
337 <li>Hypercube is quite general purpose data structure that naturally
338 encapsulates wide variety data and problems.
341 <li>Nicely captures apparent <a href="#sec-3-1">properties of the brain</a>.
344 <li>Naturally supports distributed and parallel <a href="#sec-5">geometrical data storage
348 <li>Dedicated hardware like <a href="#sec-3-2">CM-1</a> can be built around hypercube concept
349 that results in data, computation process and hardware, all
350 beautifully fitting together while complementing each other
354 <li>Hypercube stored data (and computation process) has geometry by its
355 nature and should fit nicely with "3D first" user interface ideology
356 of the parent <a href="http://www2.svjatoslav.eu/gitbrowse/sixth/doc/index.html">Sixth</a> project.
361 <div id="outline-container-sec-5" class="outline-2">
362 <h2 id="sec-5"><a id="ID-171fe375-c737-41e6-b429-a414f6abc5d8" name="ID-171fe375-c737-41e6-b429-a414f6abc5d8"></a><span class="section-number-2">5</span> Geometrical computation idea</h2>
363 <div class="outline-text-2" id="text-5">
365 <div id="outline-container-sec-5-1" class="outline-3">
366 <h3 id="sec-5-1"><a id="ID-5d287158-53ea-44a2-a754-dd862366066a" name="ID-5d287158-53ea-44a2-a754-dd862366066a"></a><span class="section-number-3">5.1</span> Distributed computation and data storage</h3>
367 <div class="outline-text-3" id="text-5-1">
369 Lots of problems can be translated to geometry (use any shapes and as
370 many dimensions as you need). Solution(s) to such problems could be
371 then found via geometrical search/comparison/lookup results. As a
372 bonus, such geometrical *data storage* AND *computation* can be
373 naturally made in *parallel* and *distributed*.
377 Learning means building/updating/re-balancing the model (the hard
378 part). Question answering is making (relatively simple) lookups
379 (geometrical queries) against the model.
383 <div id="outline-container-sec-5-2" class="outline-3">
384 <h3 id="sec-5-2"><a id="ID-a117c11e-97c1-4822-88b2-9fc10f96caec" name="ID-a117c11e-97c1-4822-88b2-9fc10f96caec"></a><span class="section-number-3">5.2</span> Mapping hypercube to object-oriented model and relational database</h3>
385 <div class="outline-text-3" id="text-5-2">
387 Object oriented programming is inspired by the way human mind
388 operates. It allows programmer to express ideas to computer in a more
393 It is actually also possible to map object model and relational
394 database to geometrical hyperspace:
398 <li>Object or database table row is a point in hypercube arbitrary
399 dimensional space. Each object member variable or database table
400 column can be mapped to its own dimension in hypercube. That is: if
401 class declares 4 variables for an object, then corresponding object
402 can be stored as a single point inside 4 dimensional
403 hypercube. Variable values translate to point coordinates in that
404 hypercube. That is: numbers and string can be translated to linear
405 value that can be used as a coordinate along particular dimension.
408 <li>Each object class or database table declares its own hypercube that
409 contain instances (objects) of that class or rows of a table.
415 <div id="outline-container-sec-5-3" class="outline-3">
416 <h3 id="sec-5-3"><a id="ID-b6b15bd2-c78b-4c51-a343-72843a515c29" name="ID-b6b15bd2-c78b-4c51-a343-72843a515c29"></a><span class="section-number-3">5.3</span> Mapping entity relations in hypercube</h3>
417 <div class="outline-text-3" id="text-5-3">
419 Consider we want to create database of:
426 <li>Effort: Amount of time contributed by every author to every book
432 Information above can be represented as 3D cube where dimensions are:
444 Points in that cube would nicely capture many to many relations
445 between authors and the books.
450 <div id="outline-container-sec-6" class="outline-2">
451 <h2 id="sec-6"><span class="section-number-2">6</span> Current status</h2>
452 <div class="outline-text-2" id="text-6">
454 <li>More or less defined <a href="#sec-2">Vision / goal</a>.
457 <li>Collected some <a href="#sec-3">inspiring</a> <a href="#sec-5">ideas</a>.
460 <li>Implemented very simple persistent key-value map.
462 <li>Long term goal is to use it as a backing storage engine and
463 implement more advanced features on top of this via layered
472 <div id="outline-container-sec-7" class="outline-2">
473 <h2 id="sec-7"><span class="section-number-2">7</span> See also</h2>
474 <div class="outline-text-2" id="text-7">
476 Interesting or competing projects with good ideas:
480 <li><a href="#sec-3-2">CM-1 Connection Machine</a>
483 <li><a href="http://phantomos.org/">http://phantomos.org/</a>
485 <li>Programs run forever. System crash or reboot does not destroy
486 state of running program.
491 <li>Taichi: A Language for High-Performance Computation onSpatially
492 Sparse Data Structures
494 <li><a href="http://taichi.graphics/wp-content/uploads/2019/09/taichi_lang.pdf">http://taichi.graphics/wp-content/uploads/2019/09/taichi_lang.pdf</a>
501 <li>database in the form of a knowledge graph that uses machine
502 reasoning to simplify data processing challenges for AI
503 applications. <a href="https://grakn.ai/">https://grakn.ai/</a>
508 <li><a href="http://wiki.squeak.org/squeak/2665">Magma</a>
510 <li>Multi-user object database for Squeak
515 <li><a href="http://esug.org/data/ESUG2015/3%20wednesday/1100-1130%20SQL%20Queries%20on%20Smalltalk%20Objects/SQL%20Queries%20in%20Smalltalk%20(James%20Foster).pdf">Gemstone/S</a>
517 <li>Completely distributed smalltalk based computing
523 <li><a href="http://www.uruk.org/emu/Taos.html">TAOS</a>
525 <li>Completely distributed operating system/virtual machine:
530 <li><a href="https://github.com/vygr/ChrysaLisp">ChrysaLisp</a>
532 <li>Assembler/C-Script/Lisp 64 bit, MIMD, multi CPU, multi threaded,
533 multi core, multi user Parallel OS. With GUI, Terminal, OO
534 Assembler, Class libraries, C-Script compiler, Lisp interpreter,
535 Debugger, and more…
542 </div><div class="col-md-3"><nav id="table-of-contents">
543 <div id="text-table-of-contents" class="bs-docs-sidebar">
545 <li><a href="#sec-1">1. General</a>
547 <li><a href="#sec-1-1">1.1. Source code</a></li>
550 <li><a href="#sec-2">2. Vision / goal</a></li>
551 <li><a href="#sec-3">3. Inspiration</a>
553 <li><a href="#sec-3-1">3.1. Brain</a></li>
554 <li><a href="#sec-3-2">3.2. CM-1 Connection Machine</a></li>
557 <li><a href="#sec-4">4. Reasons for hypercube as a so called first class citizen</a></li>
558 <li><a href="#sec-5">5. Geometrical computation idea</a>
560 <li><a href="#sec-5-1">5.1. Distributed computation and data storage</a></li>
561 <li><a href="#sec-5-2">5.2. Mapping hypercube to object-oriented model and relational database</a></li>
562 <li><a href="#sec-5-3">5.3. Mapping entity relations in hypercube</a></li>
565 <li><a href="#sec-6">6. Current status</a></li>
566 <li><a href="#sec-7">7. See also</a></li>
571 <footer id="postamble" class="">
572 <div><p class="author">Author: Svjatoslav Agejenko</p>
573 <p class="date">Created: 2020-01-29 K 22:47</p>
574 <p class="creator"><a href="http://www.gnu.org/software/emacs/">Emacs</a> 26.1 (<a href="http://orgmode.org">Org-mode</a> 9.1.9)</p>