1 <?xml version="1.0" encoding="utf-8"?>
2 <!DOCTYPE html PUBLIC "-//W3C//DTD XHTML 1.0 Strict//EN"
3 "http://www.w3.org/TR/xhtml1/DTD/xhtml1-strict.dtd">
4 <html xmlns="http://www.w3.org/1999/xhtml" lang="en" xml:lang="en">
6 <!-- 2021-03-16 Tue 20:48 -->
7 <meta http-equiv="Content-Type" content="text/html;charset=utf-8" />
8 <meta name="viewport" content="width=device-width, initial-scale=1" />
9 <title>Sixth Data - Data storage and computing engine</title>
10 <meta name="generator" content="Org mode" />
11 <meta name="author" content="Svjatoslav Agejenko" />
12 <style type="text/css">
13 <!--/*--><![CDATA[/*><!--*/
14 .title { text-align: center;
15 margin-bottom: .2em; }
16 .subtitle { text-align: center;
20 .todo { font-family: monospace; color: red; }
21 .done { font-family: monospace; color: green; }
22 .priority { font-family: monospace; color: orange; }
23 .tag { background-color: #eee; font-family: monospace;
24 padding: 2px; font-size: 80%; font-weight: normal; }
25 .timestamp { color: #bebebe; }
26 .timestamp-kwd { color: #5f9ea0; }
27 .org-right { margin-left: auto; margin-right: 0px; text-align: right; }
28 .org-left { margin-left: 0px; margin-right: auto; text-align: left; }
29 .org-center { margin-left: auto; margin-right: auto; text-align: center; }
30 .underline { text-decoration: underline; }
31 #postamble p, #preamble p { font-size: 90%; margin: .2em; }
32 p.verse { margin-left: 3%; }
34 border: 1px solid #ccc;
35 box-shadow: 3px 3px 3px #eee;
37 font-family: monospace;
49 background-color: white;
53 border: 1px solid black;
55 pre.src:hover:before { display: inline;}
56 /* Languages per Org manual */
57 pre.src-asymptote:before { content: 'Asymptote'; }
58 pre.src-awk:before { content: 'Awk'; }
59 pre.src-C:before { content: 'C'; }
60 /* pre.src-C++ doesn't work in CSS */
61 pre.src-clojure:before { content: 'Clojure'; }
62 pre.src-css:before { content: 'CSS'; }
63 pre.src-D:before { content: 'D'; }
64 pre.src-ditaa:before { content: 'ditaa'; }
65 pre.src-dot:before { content: 'Graphviz'; }
66 pre.src-calc:before { content: 'Emacs Calc'; }
67 pre.src-emacs-lisp:before { content: 'Emacs Lisp'; }
68 pre.src-fortran:before { content: 'Fortran'; }
69 pre.src-gnuplot:before { content: 'gnuplot'; }
70 pre.src-haskell:before { content: 'Haskell'; }
71 pre.src-hledger:before { content: 'hledger'; }
72 pre.src-java:before { content: 'Java'; }
73 pre.src-js:before { content: 'Javascript'; }
74 pre.src-latex:before { content: 'LaTeX'; }
75 pre.src-ledger:before { content: 'Ledger'; }
76 pre.src-lisp:before { content: 'Lisp'; }
77 pre.src-lilypond:before { content: 'Lilypond'; }
78 pre.src-lua:before { content: 'Lua'; }
79 pre.src-matlab:before { content: 'MATLAB'; }
80 pre.src-mscgen:before { content: 'Mscgen'; }
81 pre.src-ocaml:before { content: 'Objective Caml'; }
82 pre.src-octave:before { content: 'Octave'; }
83 pre.src-org:before { content: 'Org mode'; }
84 pre.src-oz:before { content: 'OZ'; }
85 pre.src-plantuml:before { content: 'Plantuml'; }
86 pre.src-processing:before { content: 'Processing.js'; }
87 pre.src-python:before { content: 'Python'; }
88 pre.src-R:before { content: 'R'; }
89 pre.src-ruby:before { content: 'Ruby'; }
90 pre.src-sass:before { content: 'Sass'; }
91 pre.src-scheme:before { content: 'Scheme'; }
92 pre.src-screen:before { content: 'Gnu Screen'; }
93 pre.src-sed:before { content: 'Sed'; }
94 pre.src-sh:before { content: 'shell'; }
95 pre.src-sql:before { content: 'SQL'; }
96 pre.src-sqlite:before { content: 'SQLite'; }
97 /* additional languages in org.el's org-babel-load-languages alist */
98 pre.src-forth:before { content: 'Forth'; }
99 pre.src-io:before { content: 'IO'; }
100 pre.src-J:before { content: 'J'; }
101 pre.src-makefile:before { content: 'Makefile'; }
102 pre.src-maxima:before { content: 'Maxima'; }
103 pre.src-perl:before { content: 'Perl'; }
104 pre.src-picolisp:before { content: 'Pico Lisp'; }
105 pre.src-scala:before { content: 'Scala'; }
106 pre.src-shell:before { content: 'Shell Script'; }
107 pre.src-ebnf2ps:before { content: 'ebfn2ps'; }
108 /* additional language identifiers per "defun org-babel-execute"
110 pre.src-cpp:before { content: 'C++'; }
111 pre.src-abc:before { content: 'ABC'; }
112 pre.src-coq:before { content: 'Coq'; }
113 pre.src-groovy:before { content: 'Groovy'; }
114 /* additional language identifiers from org-babel-shell-names in
115 ob-shell.el: ob-shell is the only babel language using a lambda to put
116 the execution function name together. */
117 pre.src-bash:before { content: 'bash'; }
118 pre.src-csh:before { content: 'csh'; }
119 pre.src-ash:before { content: 'ash'; }
120 pre.src-dash:before { content: 'dash'; }
121 pre.src-ksh:before { content: 'ksh'; }
122 pre.src-mksh:before { content: 'mksh'; }
123 pre.src-posh:before { content: 'posh'; }
124 /* Additional Emacs modes also supported by the LaTeX listings package */
125 pre.src-ada:before { content: 'Ada'; }
126 pre.src-asm:before { content: 'Assembler'; }
127 pre.src-caml:before { content: 'Caml'; }
128 pre.src-delphi:before { content: 'Delphi'; }
129 pre.src-html:before { content: 'HTML'; }
130 pre.src-idl:before { content: 'IDL'; }
131 pre.src-mercury:before { content: 'Mercury'; }
132 pre.src-metapost:before { content: 'MetaPost'; }
133 pre.src-modula-2:before { content: 'Modula-2'; }
134 pre.src-pascal:before { content: 'Pascal'; }
135 pre.src-ps:before { content: 'PostScript'; }
136 pre.src-prolog:before { content: 'Prolog'; }
137 pre.src-simula:before { content: 'Simula'; }
138 pre.src-tcl:before { content: 'tcl'; }
139 pre.src-tex:before { content: 'TeX'; }
140 pre.src-plain-tex:before { content: 'Plain TeX'; }
141 pre.src-verilog:before { content: 'Verilog'; }
142 pre.src-vhdl:before { content: 'VHDL'; }
143 pre.src-xml:before { content: 'XML'; }
144 pre.src-nxml:before { content: 'XML'; }
145 /* add a generic configuration mode; LaTeX export needs an additional
146 (add-to-list 'org-latex-listings-langs '(conf " ")) in .emacs */
147 pre.src-conf:before { content: 'Configuration File'; }
149 table { border-collapse:collapse; }
150 caption.t-above { caption-side: top; }
151 caption.t-bottom { caption-side: bottom; }
152 td, th { vertical-align:top; }
153 th.org-right { text-align: center; }
154 th.org-left { text-align: center; }
155 th.org-center { text-align: center; }
156 td.org-right { text-align: right; }
157 td.org-left { text-align: left; }
158 td.org-center { text-align: center; }
159 dt { font-weight: bold; }
160 .footpara { display: inline; }
161 .footdef { margin-bottom: 1em; }
162 .figure { padding: 1em; }
163 .figure p { text-align: center; }
164 .equation-container {
170 vertical-align: middle;
175 vertical-align: middle;
179 border: 2px solid gray;
184 { text-align: right; font-size: 70%; white-space: nowrap; }
185 textarea { overflow-x: auto; }
186 .linenr { font-size: smaller }
187 .code-highlighted { background-color: #ffff00; }
188 .org-info-js_info-navigation { border-style: none; }
189 #org-info-js_console-label
190 { font-size: 10px; font-weight: bold; white-space: nowrap; }
191 .org-info-js_search-highlight
192 { background-color: #ffff00; color: #000000; font-weight: bold; }
193 .org-svg { width: 90%; }
196 <link href="https://bootswatch.com/3/darkly/bootstrap.min.css" rel="stylesheet">
197 <script src="https://cdnjs.cloudflare.com/ajax/libs/jquery/1.11.3/jquery.min.js"></script>
198 <script src="https://cdnjs.cloudflare.com/ajax/libs/twitter-bootstrap/3.3.5/js/bootstrap.min.js"></script>
199 <style type="text/css">
200 footer {background-color: #111 !important;}
201 pre {background-color: #111; color: #ccc;}
203 <link rel="stylesheet" type="text/css" href="https://thomasf.github.io/solarized-css/solarized-dark.min.css" />
204 <script type="text/javascript">
206 @licstart The following is the entire license notice for the
207 JavaScript code in this tag.
209 Copyright (C) 2012-2020 Free Software Foundation, Inc.
211 The JavaScript code in this tag is free software: you can
212 redistribute it and/or modify it under the terms of the GNU
213 General Public License (GNU GPL) as published by the Free Software
214 Foundation, either version 3 of the License, or (at your option)
215 any later version. The code is distributed WITHOUT ANY WARRANTY;
216 without even the implied warranty of MERCHANTABILITY or FITNESS
217 FOR A PARTICULAR PURPOSE. See the GNU GPL for more details.
219 As additional permission under GNU GPL version 3 section 7, you
220 may distribute non-source (e.g., minimized or compacted) forms of
221 that code without the copy of the GNU GPL normally required by
222 section 4, provided you include this license notice and a URL
223 through which recipients can access the Corresponding Source.
226 @licend The above is the entire license notice
227 for the JavaScript code in this tag.
229 <!--/*--><![CDATA[/*><!--*/
230 function CodeHighlightOn(elem, id)
232 var target = document.getElementById(id);
234 elem.cacheClassElem = elem.className;
235 elem.cacheClassTarget = target.className;
236 target.className = "code-highlighted";
237 elem.className = "code-highlighted";
240 function CodeHighlightOff(elem, id)
242 var target = document.getElementById(id);
243 if(elem.cacheClassElem)
244 elem.className = elem.cacheClassElem;
245 if(elem.cacheClassTarget)
246 target.className = elem.cacheClassTarget;
253 <h1 class="title">Sixth Data - Data storage and computing engine</h1>
254 <div id="table-of-contents">
255 <h2>Table of Contents</h2>
256 <div id="text-table-of-contents">
258 <li><a href="#orgeab6191">1. General</a>
260 <li><a href="#org7ecc9d8">1.1. Source code</a></li>
263 <li><a href="#org5efecb1">2. Vision / goal</a></li>
264 <li><a href="#orgf50bee5">3. Inspiration</a>
266 <li><a href="#orge517915">3.1. Brain</a></li>
267 <li><a href="#org4d6ef67">3.2. CM-1 Connection Machine</a></li>
270 <li><a href="#orge26c065">4. Reasons for hypercube as a so called first class citizen</a></li>
271 <li><a href="#org4f408df">5. Geometrical computation idea</a>
273 <li><a href="#org86e13fb">5.1. Distributed computation and data storage</a></li>
274 <li><a href="#orgc8bbf69">5.2. Mapping hypercube to object-oriented model and relational database</a></li>
275 <li><a href="#org6b2bdc0">5.3. Mapping entity relations in hypercube</a></li>
278 <li><a href="#org5c3a882">6. Current status</a></li>
279 <li><a href="#orgfccb4f4">7. See also</a>
281 <li><a href="#orgaf37e2e">7.1. Computation on multi dimensional data</a></li>
282 <li><a href="#org56bd168">7.2. Distributed, reliable, parallel computing systems</a></li>
283 <li><a href="#orgb6ed46a">7.3. Rules based machine reasoning</a></li>
290 <div id="outline-container-orgeab6191" class="outline-2">
291 <h2 id="orgeab6191"><span class="section-number-2">1</span> General</h2>
292 <div class="outline-text-2" id="text-1">
294 <li>This program is free software: released under Creative Commons Zero
299 <li>Svjatoslav Agejenko</li>
300 <li>Homepage: <a href="https://svjatoslav.eu">https://svjatoslav.eu</a></li>
301 <li>Email: <a href="mailto://svjatoslav@svjatoslav.eu">mailto://svjatoslav@svjatoslav.eu</a></li>
304 <li><a href="https://www.svjatoslav.eu/projects/">Other software projects hosted at svjatoslav.eu</a></li>
308 <div id="outline-container-org7ecc9d8" class="outline-3">
309 <h3 id="org7ecc9d8"><span class="section-number-3">1.1</span> Source code</h3>
310 <div class="outline-text-3" id="text-1-1">
312 <li><a href="https://www2.svjatoslav.eu/gitweb/?p=sixth-data.git;a=snapshot;h=HEAD;sf=tgz">Download latest snapshot in TAR GZ format</a></li>
314 <li><a href="https://www2.svjatoslav.eu/gitweb/?p=sixth-data.git;a=summary">Browse Git repository online</a></li>
317 Clone Git repository using command:
319 <pre class="example">
320 git clone https://www2.svjatoslav.eu/git/sixth-data.git
323 <li>See <a href="https://www3.svjatoslav.eu/projects/sixth-data/apidocs/">JavaDoc</a>.</li>
329 <div id="outline-container-org5efecb1" class="outline-2">
330 <h2 id="org5efecb1"><span class="section-number-2">2</span> Vision / goal</h2>
331 <div class="outline-text-2" id="text-2">
333 Provide hackable, versioned, optimized, distributed, geometrical,
334 arbitrary dimensional (<a href="#orge26c065">hypercube based</a>) data storage and computation
335 engine (<a href="#orge517915">as inspired by the brain</a>) for general purpose visual computing
336 environment called <a href="http://www2.svjatoslav.eu/gitbrowse/sixth/doc/index.html">Sixth</a>.
340 Because <a href="http://www.paulgraham.com/rootsoflisp.html">Lisp is hackable self defined programmable programming
341 language</a> it would be used to provide <a href="https://en.wikipedia.org/wiki/Imperative_programming">imperative programming</a> support.
345 <div id="outline-container-orgf50bee5" class="outline-2">
346 <h2 id="orgf50bee5"><span class="section-number-2">3</span> Inspiration</h2>
347 <div class="outline-text-2" id="text-3">
349 <li>see also: <a href="https://en.wikipedia.org/wiki/OLAP_cube">OLAP cube</a>.</li>
352 <div id="outline-container-orge517915" class="outline-3">
353 <h3 id="orge517915"><span class="section-number-3">3.1</span> Brain</h3>
354 <div class="outline-text-3" id="text-3-1">
356 <li>Brain appears to be natural geometrical/parallel data storage and
357 computational engine:
359 <li><a href="https://www.quantamagazine.org/the-brain-maps-out-ideas-and-memories-like-spaces-20190114/">https://www.quantamagazine.org/the-brain-maps-out-ideas-and-memories-like-spaces-20190114/</a></li>
362 <li>Even more awesome is that brain appears to operate and is wired as
363 arbitrary/variable dimensional structure:
364 <a href="https://singularityhub.com/2017/06/21/is-there-a-multidimensional-mathematical-world-hidden-in-the-brains-computation/">https://singularityhub.com/2017/06/21/is-there-a-multidimensional-mathematical-world-hidden-in-the-brains-computation/</a></li>
366 <li>On top of this, this multidimensional space that brain represents
367 has dynamic/variable resolution/density:
369 <li><a href="https://www.quantamagazine.org/goals-and-rewards-redraw-the-brains-map-of-the-world-20190328">https://www.quantamagazine.org/goals-and-rewards-redraw-the-brains-map-of-the-world-20190328</a></li>
372 <li>Such properties allow parallel <a href="#org4f408df">Geometrical computation</a> and
373 beautifully fits <a href="#org4d6ef67">CM-1 Connection Machine</a> architecture (for extra
374 hardware accelerated solution).</li>
379 <div id="outline-container-org4d6ef67" class="outline-3">
380 <h3 id="org4d6ef67"><span class="section-number-3">3.2</span> CM-1 Connection Machine</h3>
381 <div class="outline-text-3" id="text-3-2">
383 <a href="https://en.wikipedia.org/wiki/Connection_Machine">https://en.wikipedia.org/wiki/Connection_Machine</a>
387 Massively parallel (thousands of CPUs) connected via
388 machine's internal 12-dimensional hypercube network allows to
389 efficiently simulate arbitrary dimensional hypercube and network
390 topology between computational units. So that when we are
391 solving/simulating for example 5 dimensional problem, we can arrange
392 computational units into virtual 5D network. See:
393 <a href="http://www.mission-base.com/tamiko/theory/cm_txts/di-ch2.html">http://www.mission-base.com/tamiko/theory/cm_txts/di-ch2.html</a>
397 we can pre-distribute data across computation units and perform
398 parallel <a href="#org4f408df">geometrical computation</a>.
404 <div id="outline-container-orge26c065" class="outline-2">
405 <h2 id="orge26c065"><span class="section-number-2">4</span> Reasons for hypercube as a so called first class citizen</h2>
406 <div class="outline-text-2" id="text-4">
408 <li>Hypercube is quite general purpose data structure that naturally
409 encapsulates wide variety data and problems.</li>
411 <li>Nicely captures apparent <a href="#orge517915">properties of the brain</a>.</li>
413 <li>Naturally supports distributed and parallel <a href="#org4f408df">geometrical data storage
414 and computation.</a></li>
416 <li>Dedicated hardware like <a href="#org4d6ef67">CM-1</a> can be built around hypercube concept
417 that results in data, computation process and hardware, all
418 beautifully fitting together while complementing each other
421 <li>Hypercube stored data (and computation process) has geometry by its
422 nature and should fit nicely with "3D first" user interface ideology
423 of the parent <a href="http://www2.svjatoslav.eu/gitbrowse/sixth/doc/index.html">Sixth</a> project.</li>
427 <div id="outline-container-org4f408df" class="outline-2">
428 <h2 id="org4f408df"><span class="section-number-2">5</span> Geometrical computation idea</h2>
429 <div class="outline-text-2" id="text-5">
431 <div id="outline-container-org86e13fb" class="outline-3">
432 <h3 id="org86e13fb"><span class="section-number-3">5.1</span> Distributed computation and data storage</h3>
433 <div class="outline-text-3" id="text-5-1">
435 Lots of problems can be translated to geometry (use any shapes and as
436 many dimensions as you need). Solution(s) to such problems could be
437 then found via geometrical search/comparison/lookup results. As a
438 bonus, such geometrical <b>data storage</b> AND <b>computation</b> can be
439 naturally made in <b>parallel</b> and <b>distributed</b>.
443 Learning means building/updating/re-balancing the model (the hard
444 part). Question answering is making (relatively simple) lookups
445 (geometrical queries) against the model.
449 <div id="outline-container-orgc8bbf69" class="outline-3">
450 <h3 id="orgc8bbf69"><span class="section-number-3">5.2</span> Mapping hypercube to object-oriented model and relational database</h3>
451 <div class="outline-text-3" id="text-5-2">
453 Object oriented programming is inspired by the way human mind
454 operates. It allows programmer to express ideas to computer in a more
459 It is actually also possible to map object model and relational
460 database to geometrical hyperspace:
464 <li>Object or database table row is a point in hypercube arbitrary
465 dimensional space. Each object member variable or database table
466 column can be mapped to its own dimension in hypercube. That is: if
467 class declares 4 variables for an object, then corresponding object
468 can be stored as a single point inside 4 dimensional
469 hypercube. Variable values translate to point coordinates in that
470 hypercube. That is: numbers and string can be translated to linear
471 value that can be used as a coordinate along particular dimension.</li>
473 <li>Each object class or database table declares its own hypercube that
474 contain instances (objects) of that class or rows of a table.</li>
479 <div id="outline-container-org6b2bdc0" class="outline-3">
480 <h3 id="org6b2bdc0"><span class="section-number-3">5.3</span> Mapping entity relations in hypercube</h3>
481 <div class="outline-text-3" id="text-5-3">
483 Consider we want to create database of:
488 <li>Effort: Amount of time contributed by every author to every book
489 that he/she wrote.</li>
493 Information above can be represented as 3D cube where dimensions are:
502 Points in that cube would nicely capture many to many relations
503 between authors and the books.
508 <div id="outline-container-org5c3a882" class="outline-2">
509 <h2 id="org5c3a882"><span class="section-number-2">6</span> Current status</h2>
510 <div class="outline-text-2" id="text-6">
512 <li>More or less defined <a href="#org5efecb1">Vision / goal</a>.</li>
514 <li>Collected some <a href="#orgf50bee5">inspiring</a> <a href="#org4f408df">ideas</a>.</li>
516 <li>Implemented very simple persistent key-value map.
518 <li>Long term goal is to use it as a backing storage engine and
519 implement more advanced features on top of this via layered
526 <div id="outline-container-orgfccb4f4" class="outline-2">
527 <h2 id="orgfccb4f4"><span class="section-number-2">7</span> See also</h2>
528 <div class="outline-text-2" id="text-7">
530 Interesting or competing projects with good ideas:
534 <li>flexible user interface building for interacting with different data
536 <li><a href="https://gtoolkit.com/">Glamorous Toolkit</a>
538 <li>Moldable development environment. It is a live notebook. It is a
539 flexible search interface. It is a fancy code editor. It is a
540 software analysis platform. It is a data visualization engine. All
547 <div id="outline-container-orgaf37e2e" class="outline-3">
548 <h3 id="orgaf37e2e"><span class="section-number-3">7.1</span> Computation on multi dimensional data</h3>
549 <div class="outline-text-3" id="text-7-1">
551 <li><a href="https://rd-alliance.org/system/files/Array-Databases_final-report.pdf">Array Databases: Concepts, Standards, Implementations</a></li>
553 <li><a href="https://tiledb.com/">TileDB</a>
555 <li>Analyze and share complex multi-dimensional data at scale</li>
558 <li><a href="#org4d6ef67">CM-1 Connection Machine</a></li>
562 <div id="outline-container-org56bd168" class="outline-3">
563 <h3 id="org56bd168"><span class="section-number-3">7.2</span> Distributed, reliable, parallel computing systems</h3>
564 <div class="outline-text-3" id="text-7-2">
566 <li><a href="https://github.com/vygr/ChrysaLisp">ChrysaLisp</a>
568 <li>Assembler/C-Script/Lisp 64 bit, MIMD, multi CPU, multi threaded,
569 multi core, multi user Parallel OS. With GUI, Terminal, OO
570 Assembler, Class libraries, C-Script compiler, Lisp interpreter,
571 Debugger, and more…</li>
574 <li><a href="http://esug.org/data/ESUG2015/3%20wednesday/1100-1130%20SQL%20Queries%20on%20Smalltalk%20Objects/SQL%20Queries%20in%20Smalltalk%20(James%20Foster).pdf">Gemstone/S</a>
576 <li>Completely distributed smalltalk based computing
580 <li><a href="http://phantomos.org/">http://phantomos.org/</a>
582 <li>Programs run forever. System crash or reboot does not destroy
583 state of running program.</li>
586 <li><a href="http://wiki.squeak.org/squeak/2665">Magma</a>
588 <li>Multi-user object database for Squeak</li>
591 <li><a href="http://www.uruk.org/emu/Taos.html">TAOS</a>
593 <li>Completely distributed operating system/virtual machine:</li>
599 <div id="outline-container-orgb6ed46a" class="outline-3">
600 <h3 id="orgb6ed46a"><span class="section-number-3">7.3</span> Rules based machine reasoning</h3>
601 <div class="outline-text-3" id="text-7-3">
605 <li>database in the form of a knowledge graph that uses machine
606 reasoning to simplify data processing challenges for AI
607 applications. <a href="https://grakn.ai/">https://grakn.ai/</a></li>
610 <li><a href="https://en.wikipedia.org/wiki/Prolog">Prolog programming language</a></li>
616 <div id="postamble" class="status">
617 <p class="author">Author: Svjatoslav Agejenko</p>
618 <p class="date">Created: 2021-03-16 Tue 20:48</p>
619 <p class="validation"><a href="http://validator.w3.org/check?uri=referer">Validate</a></p>