1 <?xml version="1.0" encoding="utf-8"?>
2 <!DOCTYPE html PUBLIC "-//W3C//DTD XHTML 1.0 Strict//EN"
3 "http://www.w3.org/TR/xhtml1/DTD/xhtml1-strict.dtd">
4 <html xmlns="http://www.w3.org/1999/xhtml" lang="en" xml:lang="en">
6 <!-- 2021-04-09 Fri 16:55 -->
7 <meta http-equiv="Content-Type" content="text/html;charset=utf-8" />
8 <meta name="viewport" content="width=device-width, initial-scale=1" />
9 <title>Sixth Data - Data storage and computing engine</title>
10 <meta name="generator" content="Org mode" />
11 <meta name="author" content="Svjatoslav Agejenko" />
12 <style type="text/css">
13 <!--/*--><![CDATA[/*><!--*/
14 .title { text-align: center;
15 margin-bottom: .2em; }
16 .subtitle { text-align: center;
20 .todo { font-family: monospace; color: red; }
21 .done { font-family: monospace; color: green; }
22 .priority { font-family: monospace; color: orange; }
23 .tag { background-color: #eee; font-family: monospace;
24 padding: 2px; font-size: 80%; font-weight: normal; }
25 .timestamp { color: #bebebe; }
26 .timestamp-kwd { color: #5f9ea0; }
27 .org-right { margin-left: auto; margin-right: 0px; text-align: right; }
28 .org-left { margin-left: 0px; margin-right: auto; text-align: left; }
29 .org-center { margin-left: auto; margin-right: auto; text-align: center; }
30 .underline { text-decoration: underline; }
31 #postamble p, #preamble p { font-size: 90%; margin: .2em; }
32 p.verse { margin-left: 3%; }
34 border: 1px solid #ccc;
35 box-shadow: 3px 3px 3px #eee;
37 font-family: monospace;
49 background-color: white;
53 border: 1px solid black;
55 pre.src:hover:before { display: inline; margin-top: 14px;}
56 /* Languages per Org manual */
57 pre.src-asymptote:before { content: 'Asymptote'; }
58 pre.src-awk:before { content: 'Awk'; }
59 pre.src-C:before { content: 'C'; }
60 /* pre.src-C++ doesn't work in CSS */
61 pre.src-clojure:before { content: 'Clojure'; }
62 pre.src-css:before { content: 'CSS'; }
63 pre.src-D:before { content: 'D'; }
64 pre.src-ditaa:before { content: 'ditaa'; }
65 pre.src-dot:before { content: 'Graphviz'; }
66 pre.src-calc:before { content: 'Emacs Calc'; }
67 pre.src-emacs-lisp:before { content: 'Emacs Lisp'; }
68 pre.src-fortran:before { content: 'Fortran'; }
69 pre.src-gnuplot:before { content: 'gnuplot'; }
70 pre.src-haskell:before { content: 'Haskell'; }
71 pre.src-hledger:before { content: 'hledger'; }
72 pre.src-java:before { content: 'Java'; }
73 pre.src-js:before { content: 'Javascript'; }
74 pre.src-latex:before { content: 'LaTeX'; }
75 pre.src-ledger:before { content: 'Ledger'; }
76 pre.src-lisp:before { content: 'Lisp'; }
77 pre.src-lilypond:before { content: 'Lilypond'; }
78 pre.src-lua:before { content: 'Lua'; }
79 pre.src-matlab:before { content: 'MATLAB'; }
80 pre.src-mscgen:before { content: 'Mscgen'; }
81 pre.src-ocaml:before { content: 'Objective Caml'; }
82 pre.src-octave:before { content: 'Octave'; }
83 pre.src-org:before { content: 'Org mode'; }
84 pre.src-oz:before { content: 'OZ'; }
85 pre.src-plantuml:before { content: 'Plantuml'; }
86 pre.src-processing:before { content: 'Processing.js'; }
87 pre.src-python:before { content: 'Python'; }
88 pre.src-R:before { content: 'R'; }
89 pre.src-ruby:before { content: 'Ruby'; }
90 pre.src-sass:before { content: 'Sass'; }
91 pre.src-scheme:before { content: 'Scheme'; }
92 pre.src-screen:before { content: 'Gnu Screen'; }
93 pre.src-sed:before { content: 'Sed'; }
94 pre.src-sh:before { content: 'shell'; }
95 pre.src-sql:before { content: 'SQL'; }
96 pre.src-sqlite:before { content: 'SQLite'; }
97 /* additional languages in org.el's org-babel-load-languages alist */
98 pre.src-forth:before { content: 'Forth'; }
99 pre.src-io:before { content: 'IO'; }
100 pre.src-J:before { content: 'J'; }
101 pre.src-makefile:before { content: 'Makefile'; }
102 pre.src-maxima:before { content: 'Maxima'; }
103 pre.src-perl:before { content: 'Perl'; }
104 pre.src-picolisp:before { content: 'Pico Lisp'; }
105 pre.src-scala:before { content: 'Scala'; }
106 pre.src-shell:before { content: 'Shell Script'; }
107 pre.src-ebnf2ps:before { content: 'ebfn2ps'; }
108 /* additional language identifiers per "defun org-babel-execute"
110 pre.src-cpp:before { content: 'C++'; }
111 pre.src-abc:before { content: 'ABC'; }
112 pre.src-coq:before { content: 'Coq'; }
113 pre.src-groovy:before { content: 'Groovy'; }
114 /* additional language identifiers from org-babel-shell-names in
115 ob-shell.el: ob-shell is the only babel language using a lambda to put
116 the execution function name together. */
117 pre.src-bash:before { content: 'bash'; }
118 pre.src-csh:before { content: 'csh'; }
119 pre.src-ash:before { content: 'ash'; }
120 pre.src-dash:before { content: 'dash'; }
121 pre.src-ksh:before { content: 'ksh'; }
122 pre.src-mksh:before { content: 'mksh'; }
123 pre.src-posh:before { content: 'posh'; }
124 /* Additional Emacs modes also supported by the LaTeX listings package */
125 pre.src-ada:before { content: 'Ada'; }
126 pre.src-asm:before { content: 'Assembler'; }
127 pre.src-caml:before { content: 'Caml'; }
128 pre.src-delphi:before { content: 'Delphi'; }
129 pre.src-html:before { content: 'HTML'; }
130 pre.src-idl:before { content: 'IDL'; }
131 pre.src-mercury:before { content: 'Mercury'; }
132 pre.src-metapost:before { content: 'MetaPost'; }
133 pre.src-modula-2:before { content: 'Modula-2'; }
134 pre.src-pascal:before { content: 'Pascal'; }
135 pre.src-ps:before { content: 'PostScript'; }
136 pre.src-prolog:before { content: 'Prolog'; }
137 pre.src-simula:before { content: 'Simula'; }
138 pre.src-tcl:before { content: 'tcl'; }
139 pre.src-tex:before { content: 'TeX'; }
140 pre.src-plain-tex:before { content: 'Plain TeX'; }
141 pre.src-verilog:before { content: 'Verilog'; }
142 pre.src-vhdl:before { content: 'VHDL'; }
143 pre.src-xml:before { content: 'XML'; }
144 pre.src-nxml:before { content: 'XML'; }
145 /* add a generic configuration mode; LaTeX export needs an additional
146 (add-to-list 'org-latex-listings-langs '(conf " ")) in .emacs */
147 pre.src-conf:before { content: 'Configuration File'; }
149 table { border-collapse:collapse; }
150 caption.t-above { caption-side: top; }
151 caption.t-bottom { caption-side: bottom; }
152 td, th { vertical-align:top; }
153 th.org-right { text-align: center; }
154 th.org-left { text-align: center; }
155 th.org-center { text-align: center; }
156 td.org-right { text-align: right; }
157 td.org-left { text-align: left; }
158 td.org-center { text-align: center; }
159 dt { font-weight: bold; }
160 .footpara { display: inline; }
161 .footdef { margin-bottom: 1em; }
162 .figure { padding: 1em; }
163 .figure p { text-align: center; }
164 .equation-container {
170 vertical-align: middle;
175 vertical-align: middle;
179 border: 2px solid gray;
184 { text-align: right; font-size: 70%; white-space: nowrap; }
185 textarea { overflow-x: auto; }
186 .linenr { font-size: smaller }
187 .code-highlighted { background-color: #ffff00; }
188 .org-info-js_info-navigation { border-style: none; }
189 #org-info-js_console-label
190 { font-size: 10px; font-weight: bold; white-space: nowrap; }
191 .org-info-js_search-highlight
192 { background-color: #ffff00; color: #000000; font-weight: bold; }
193 .org-svg { width: 90%; }
196 <link href="https://bootswatch.com/3/darkly/bootstrap.min.css" rel="stylesheet">
197 <script src="https://cdnjs.cloudflare.com/ajax/libs/jquery/1.11.3/jquery.min.js"></script>
198 <script src="https://cdnjs.cloudflare.com/ajax/libs/twitter-bootstrap/3.3.5/js/bootstrap.min.js"></script>
199 <style type="text/css">
200 footer {background-color: #111 !important;}
201 pre {background-color: #111; color: #ccc;}
203 <link rel="stylesheet" type="text/css" href="https://thomasf.github.io/solarized-css/solarized-dark.min.css" />
204 <script type="text/javascript">
205 // @license magnet:?xt=urn:btih:e95b018ef3580986a04669f1b5879592219e2a7a&dn=public-domain.txt Public Domain
206 <!--/*--><![CDATA[/*><!--*/
207 function CodeHighlightOn(elem, id)
209 var target = document.getElementById(id);
211 elem.classList.add("code-highlighted");
212 target.classList.add("code-highlighted");
215 function CodeHighlightOff(elem, id)
217 var target = document.getElementById(id);
219 elem.classList.remove("code-highlighted");
220 target.classList.remove("code-highlighted");
229 <h1 class="title">Sixth Data - Data storage and computing engine</h1>
230 <div id="table-of-contents">
231 <h2>Table of Contents</h2>
232 <div id="text-table-of-contents">
234 <li><a href="#orgf6f7c88">1. General</a>
236 <li><a href="#org131e0e7">1.1. Source code</a></li>
239 <li><a href="#orgdcc87be">2. Vision / goal</a></li>
240 <li><a href="#org272be25">3. Inspiration</a>
242 <li><a href="#orgd088440">3.1. Brain</a></li>
243 <li><a href="#org341bc63">3.2. CM-1 Connection Machine</a></li>
246 <li><a href="#org73746b7">4. Reasons for hypercube as a so called first class citizen</a></li>
247 <li><a href="#org62f4a7e">5. Geometrical computation idea</a>
249 <li><a href="#org2eee867">5.1. Distributed computation and data storage</a></li>
250 <li><a href="#orgf3009e9">5.2. Mapping hypercube to object-oriented model and relational database</a></li>
251 <li><a href="#orgbbab085">5.3. Mapping entity relations in hypercube</a></li>
254 <li><a href="#org93c7c97">6. Current status</a></li>
255 <li><a href="#orga7b6014">7. See also</a>
257 <li><a href="#orgae272cb">7.1. Computation on multi dimensional data</a></li>
258 <li><a href="#org472df24">7.2. Distributed, reliable, parallel computing systems</a></li>
259 <li><a href="#orgf75aa32">7.3. Rules based machine reasoning</a></li>
266 <div id="outline-container-orgf6f7c88" class="outline-2">
267 <h2 id="orgf6f7c88"><span class="section-number-2">1</span> General</h2>
268 <div class="outline-text-2" id="text-1">
270 <li>This program is free software: released under Creative Commons Zero
275 <li>Svjatoslav Agejenko</li>
276 <li>Homepage: <a href="https://svjatoslav.eu">https://svjatoslav.eu</a></li>
277 <li>Email: <a href="mailto://svjatoslav@svjatoslav.eu">mailto://svjatoslav@svjatoslav.eu</a></li>
280 <li><a href="https://www.svjatoslav.eu/projects/">Other software projects hosted at svjatoslav.eu</a></li>
284 <div id="outline-container-org131e0e7" class="outline-3">
285 <h3 id="org131e0e7"><span class="section-number-3">1.1</span> Source code</h3>
286 <div class="outline-text-3" id="text-1-1">
288 <li><a href="https://www2.svjatoslav.eu/gitweb/?p=sixth-data.git;a=snapshot;h=HEAD;sf=tgz">Download latest snapshot in TAR GZ format</a></li>
290 <li><a href="https://www2.svjatoslav.eu/gitweb/?p=sixth-data.git;a=summary">Browse Git repository online</a></li>
293 Clone Git repository using command:
295 <pre class="example">
296 git clone https://www2.svjatoslav.eu/git/sixth-data.git
299 <li>See <a href="https://www3.svjatoslav.eu/projects/sixth-data/apidocs/">JavaDoc</a>.</li>
305 <div id="outline-container-orgdcc87be" class="outline-2">
306 <h2 id="orgdcc87be"><span class="section-number-2">2</span> Vision / goal</h2>
307 <div class="outline-text-2" id="text-2">
309 Provide hackable, versioned, optimized, distributed, geometrical,
310 arbitrary dimensional (<a href="#org73746b7">hypercube based</a>) data storage and computation
311 engine (<a href="#orgd088440">as inspired by the brain</a>) for general purpose visual computing
312 environment called <a href="http://www2.svjatoslav.eu/gitbrowse/sixth/doc/index.html">Sixth</a>.
316 Because <a href="http://www.paulgraham.com/rootsoflisp.html">Lisp is hackable self defined programmable programming
317 language</a> it would be used to provide <a href="https://en.wikipedia.org/wiki/Imperative_programming">imperative programming</a> support.
321 <div id="outline-container-org272be25" class="outline-2">
322 <h2 id="org272be25"><span class="section-number-2">3</span> Inspiration</h2>
323 <div class="outline-text-2" id="text-3">
325 <li>see also: <a href="https://en.wikipedia.org/wiki/OLAP_cube">OLAP cube</a>.</li>
328 <div id="outline-container-orgd088440" class="outline-3">
329 <h3 id="orgd088440"><span class="section-number-3">3.1</span> Brain</h3>
330 <div class="outline-text-3" id="text-3-1">
332 <li>Brain appears to be natural geometrical/parallel data storage and
333 computational engine:
335 <li><a href="https://www.quantamagazine.org/the-brain-maps-out-ideas-and-memories-like-spaces-20190114/">https://www.quantamagazine.org/the-brain-maps-out-ideas-and-memories-like-spaces-20190114/</a></li>
336 <li><a href="https://www.simonsfoundation.org/2021/04/07/geometrical-thinking-offers-a-window-into-computation/">Geometrical Thinking Offers a Window Into Computation</a></li>
339 <li>Even more awesome is that brain appears to operate and is wired as
340 arbitrary/variable dimensional structure:
341 <a href="https://singularityhub.com/2017/06/21/is-there-a-multidimensional-mathematical-world-hidden-in-the-brains-computation/">https://singularityhub.com/2017/06/21/is-there-a-multidimensional-mathematical-world-hidden-in-the-brains-computation/</a></li>
343 <li>On top of this, this multidimensional space that brain represents
344 has dynamic/variable resolution/density:
346 <li><a href="https://www.quantamagazine.org/goals-and-rewards-redraw-the-brains-map-of-the-world-20190328">https://www.quantamagazine.org/goals-and-rewards-redraw-the-brains-map-of-the-world-20190328</a></li>
349 <li>Such properties allow parallel <a href="#org62f4a7e">Geometrical computation</a> and
350 beautifully fits <a href="#org341bc63">CM-1 Connection Machine</a> architecture (for extra
351 hardware accelerated solution).</li>
357 <div id="outline-container-org341bc63" class="outline-3">
358 <h3 id="org341bc63"><span class="section-number-3">3.2</span> CM-1 Connection Machine</h3>
359 <div class="outline-text-3" id="text-3-2">
361 <a href="https://en.wikipedia.org/wiki/Connection_Machine">https://en.wikipedia.org/wiki/Connection_Machine</a>
365 Massively parallel (thousands of CPUs) connected via
366 machine's internal 12-dimensional hypercube network allows to
367 efficiently simulate arbitrary dimensional hypercube and network
368 topology between computational units. So that when we are
369 solving/simulating for example 5 dimensional problem, we can arrange
370 computational units into virtual 5D network. See:
371 <a href="http://www.mission-base.com/tamiko/theory/cm_txts/di-ch2.html">http://www.mission-base.com/tamiko/theory/cm_txts/di-ch2.html</a>
375 we can pre-distribute data across computation units and perform
376 parallel <a href="#org62f4a7e">geometrical computation</a>.
382 <div id="outline-container-org73746b7" class="outline-2">
383 <h2 id="org73746b7"><span class="section-number-2">4</span> Reasons for hypercube as a so called first class citizen</h2>
384 <div class="outline-text-2" id="text-4">
386 <li>Hypercube is quite general purpose data structure that naturally
387 encapsulates wide variety data and problems.</li>
389 <li>Nicely captures apparent <a href="#orgd088440">properties of the brain</a>.</li>
391 <li>Naturally supports distributed and parallel <a href="#org62f4a7e">geometrical data storage
392 and computation.</a></li>
394 <li>Dedicated hardware like <a href="#org341bc63">CM-1</a> can be built around hypercube concept
395 that results in data, computation process and hardware, all
396 beautifully fitting together while complementing each other
399 <li>Hypercube stored data (and computation process) has geometry by its
400 nature and should fit nicely with "3D first" user interface ideology
401 of the parent <a href="http://www2.svjatoslav.eu/gitbrowse/sixth/doc/index.html">Sixth</a> project.</li>
405 <div id="outline-container-org62f4a7e" class="outline-2">
406 <h2 id="org62f4a7e"><span class="section-number-2">5</span> Geometrical computation idea</h2>
407 <div class="outline-text-2" id="text-5">
409 <div id="outline-container-org2eee867" class="outline-3">
410 <h3 id="org2eee867"><span class="section-number-3">5.1</span> Distributed computation and data storage</h3>
411 <div class="outline-text-3" id="text-5-1">
413 Lots of problems can be translated to geometry (use any shapes and as
414 many dimensions as you need). Solution(s) to such problems could be
415 then found via geometrical search/comparison/lookup results. As a
416 bonus, such geometrical <b>data storage</b> AND <b>computation</b> can be
417 naturally made in <b>parallel</b> and <b>distributed</b>.
421 Learning means building/updating/re-balancing the model (the hard
422 part). Question answering is making (relatively simple) lookups
423 (geometrical queries) against the model.
427 <div id="outline-container-orgf3009e9" class="outline-3">
428 <h3 id="orgf3009e9"><span class="section-number-3">5.2</span> Mapping hypercube to object-oriented model and relational database</h3>
429 <div class="outline-text-3" id="text-5-2">
431 Object oriented programming is inspired by the way human mind
432 operates. It allows programmer to express ideas to computer in a more
437 It is actually also possible to map object model and relational
438 database to geometrical hyperspace:
442 <li>Object or database table row is a point in hypercube arbitrary
443 dimensional space. Each object member variable or database table
444 column can be mapped to its own dimension in hypercube. That is: if
445 class declares 4 variables for an object, then corresponding object
446 can be stored as a single point inside 4 dimensional
447 hypercube. Variable values translate to point coordinates in that
448 hypercube. That is: numbers and string can be translated to linear
449 value that can be used as a coordinate along particular dimension.</li>
451 <li>Each object class or database table declares its own hypercube that
452 contain instances (objects) of that class or rows of a table.</li>
457 <div id="outline-container-orgbbab085" class="outline-3">
458 <h3 id="orgbbab085"><span class="section-number-3">5.3</span> Mapping entity relations in hypercube</h3>
459 <div class="outline-text-3" id="text-5-3">
461 Consider we want to create database of:
466 <li>Effort: Amount of time contributed by every author to every book
467 that he/she wrote.</li>
471 Information above can be represented as 3D cube where dimensions are:
480 Points in that cube would nicely capture many to many relations
481 between authors and the books.
486 <div id="outline-container-org93c7c97" class="outline-2">
487 <h2 id="org93c7c97"><span class="section-number-2">6</span> Current status</h2>
488 <div class="outline-text-2" id="text-6">
490 <li>More or less defined <a href="#orgdcc87be">Vision / goal</a>.</li>
492 <li>Collected some <a href="#org272be25">inspiring</a> <a href="#org62f4a7e">ideas</a>.</li>
494 <li>Implemented very simple persistent key-value map.
496 <li>Long term goal is to use it as a backing storage engine and
497 implement more advanced features on top of this via layered
504 <div id="outline-container-orga7b6014" class="outline-2">
505 <h2 id="orga7b6014"><span class="section-number-2">7</span> See also</h2>
506 <div class="outline-text-2" id="text-7">
508 Interesting or competing projects with good ideas:
512 <li>flexible user interface building for interacting with different data
514 <li><a href="https://gtoolkit.com/">Glamorous Toolkit</a>
516 <li>Moldable development environment. It is a live notebook. It is a
517 flexible search interface. It is a fancy code editor. It is a
518 software analysis platform. It is a data visualization engine. All
525 <div id="outline-container-orgae272cb" class="outline-3">
526 <h3 id="orgae272cb"><span class="section-number-3">7.1</span> Computation on multi dimensional data</h3>
527 <div class="outline-text-3" id="text-7-1">
529 <li><a href="https://rd-alliance.org/system/files/Array-Databases_final-report.pdf">Array Databases: Concepts, Standards, Implementations</a></li>
531 <li><a href="https://tiledb.com/">TileDB</a>
533 <li>Analyze and share complex multi-dimensional data at scale</li>
536 <li><a href="#org341bc63">CM-1 Connection Machine</a></li>
538 <li><a href="https://lisp-stat.dev/">Lisp-Stat: An environment for Statistical Computing</a></li>
543 <div id="outline-container-org472df24" class="outline-3">
544 <h3 id="org472df24"><span class="section-number-3">7.2</span> Distributed, reliable, parallel computing systems</h3>
545 <div class="outline-text-3" id="text-7-2">
547 <li><a href="https://github.com/vygr/ChrysaLisp">ChrysaLisp</a>
549 <li>Assembler/C-Script/Lisp 64 bit, MIMD, multi CPU, multi threaded,
550 multi core, multi user Parallel OS. With GUI, Terminal, OO
551 Assembler, Class libraries, C-Script compiler, Lisp interpreter,
552 Debugger, and more…</li>
555 <li><a href="http://esug.org/data/ESUG2015/3%20wednesday/1100-1130%20SQL%20Queries%20on%20Smalltalk%20Objects/SQL%20Queries%20in%20Smalltalk%20(James%20Foster).pdf">Gemstone/S</a>
557 <li>Completely distributed smalltalk based computing
561 <li><a href="http://phantomos.org/">http://phantomos.org/</a>
563 <li>Programs run forever. System crash or reboot does not destroy
564 state of running program.</li>
567 <li><a href="http://wiki.squeak.org/squeak/2665">Magma</a>
569 <li>Multi-user object database for Squeak</li>
572 <li><a href="http://www.uruk.org/emu/Taos.html">TAOS</a>
574 <li>Completely distributed operating system/virtual machine:</li>
580 <div id="outline-container-orgf75aa32" class="outline-3">
581 <h3 id="orgf75aa32"><span class="section-number-3">7.3</span> Rules based machine reasoning</h3>
582 <div class="outline-text-3" id="text-7-3">
586 <li>database in the form of a knowledge graph that uses machine
587 reasoning to simplify data processing challenges for AI
588 applications. <a href="https://grakn.ai/">https://grakn.ai/</a></li>
591 <li><a href="https://en.wikipedia.org/wiki/Prolog">Prolog programming language</a></li>
597 <div id="postamble" class="status">
598 <p class="author">Author: Svjatoslav Agejenko</p>
599 <p class="date">Created: 2021-04-09 Fri 16:55</p>
600 <p class="validation"><a href="https://validator.w3.org/check?uri=referer">Validate</a></p>