1 <?xml version="1.0" encoding="utf-8"?>
2 <!DOCTYPE html PUBLIC "-//W3C//DTD XHTML 1.0 Strict//EN"
3 "http://www.w3.org/TR/xhtml1/DTD/xhtml1-strict.dtd">
4 <html xmlns="http://www.w3.org/1999/xhtml" lang="en" xml:lang="en">
6 <!-- 2021-04-01 Thu 19:11 -->
7 <meta http-equiv="Content-Type" content="text/html;charset=utf-8" />
8 <meta name="viewport" content="width=device-width, initial-scale=1" />
9 <title>Sixth Data - Data storage and computing engine</title>
10 <meta name="generator" content="Org mode" />
11 <meta name="author" content="Svjatoslav Agejenko" />
12 <style type="text/css">
13 <!--/*--><![CDATA[/*><!--*/
14 .title { text-align: center;
15 margin-bottom: .2em; }
16 .subtitle { text-align: center;
20 .todo { font-family: monospace; color: red; }
21 .done { font-family: monospace; color: green; }
22 .priority { font-family: monospace; color: orange; }
23 .tag { background-color: #eee; font-family: monospace;
24 padding: 2px; font-size: 80%; font-weight: normal; }
25 .timestamp { color: #bebebe; }
26 .timestamp-kwd { color: #5f9ea0; }
27 .org-right { margin-left: auto; margin-right: 0px; text-align: right; }
28 .org-left { margin-left: 0px; margin-right: auto; text-align: left; }
29 .org-center { margin-left: auto; margin-right: auto; text-align: center; }
30 .underline { text-decoration: underline; }
31 #postamble p, #preamble p { font-size: 90%; margin: .2em; }
32 p.verse { margin-left: 3%; }
34 border: 1px solid #ccc;
35 box-shadow: 3px 3px 3px #eee;
37 font-family: monospace;
49 background-color: white;
53 border: 1px solid black;
55 pre.src:hover:before { display: inline; margin-top: 14px;}
56 /* Languages per Org manual */
57 pre.src-asymptote:before { content: 'Asymptote'; }
58 pre.src-awk:before { content: 'Awk'; }
59 pre.src-C:before { content: 'C'; }
60 /* pre.src-C++ doesn't work in CSS */
61 pre.src-clojure:before { content: 'Clojure'; }
62 pre.src-css:before { content: 'CSS'; }
63 pre.src-D:before { content: 'D'; }
64 pre.src-ditaa:before { content: 'ditaa'; }
65 pre.src-dot:before { content: 'Graphviz'; }
66 pre.src-calc:before { content: 'Emacs Calc'; }
67 pre.src-emacs-lisp:before { content: 'Emacs Lisp'; }
68 pre.src-fortran:before { content: 'Fortran'; }
69 pre.src-gnuplot:before { content: 'gnuplot'; }
70 pre.src-haskell:before { content: 'Haskell'; }
71 pre.src-hledger:before { content: 'hledger'; }
72 pre.src-java:before { content: 'Java'; }
73 pre.src-js:before { content: 'Javascript'; }
74 pre.src-latex:before { content: 'LaTeX'; }
75 pre.src-ledger:before { content: 'Ledger'; }
76 pre.src-lisp:before { content: 'Lisp'; }
77 pre.src-lilypond:before { content: 'Lilypond'; }
78 pre.src-lua:before { content: 'Lua'; }
79 pre.src-matlab:before { content: 'MATLAB'; }
80 pre.src-mscgen:before { content: 'Mscgen'; }
81 pre.src-ocaml:before { content: 'Objective Caml'; }
82 pre.src-octave:before { content: 'Octave'; }
83 pre.src-org:before { content: 'Org mode'; }
84 pre.src-oz:before { content: 'OZ'; }
85 pre.src-plantuml:before { content: 'Plantuml'; }
86 pre.src-processing:before { content: 'Processing.js'; }
87 pre.src-python:before { content: 'Python'; }
88 pre.src-R:before { content: 'R'; }
89 pre.src-ruby:before { content: 'Ruby'; }
90 pre.src-sass:before { content: 'Sass'; }
91 pre.src-scheme:before { content: 'Scheme'; }
92 pre.src-screen:before { content: 'Gnu Screen'; }
93 pre.src-sed:before { content: 'Sed'; }
94 pre.src-sh:before { content: 'shell'; }
95 pre.src-sql:before { content: 'SQL'; }
96 pre.src-sqlite:before { content: 'SQLite'; }
97 /* additional languages in org.el's org-babel-load-languages alist */
98 pre.src-forth:before { content: 'Forth'; }
99 pre.src-io:before { content: 'IO'; }
100 pre.src-J:before { content: 'J'; }
101 pre.src-makefile:before { content: 'Makefile'; }
102 pre.src-maxima:before { content: 'Maxima'; }
103 pre.src-perl:before { content: 'Perl'; }
104 pre.src-picolisp:before { content: 'Pico Lisp'; }
105 pre.src-scala:before { content: 'Scala'; }
106 pre.src-shell:before { content: 'Shell Script'; }
107 pre.src-ebnf2ps:before { content: 'ebfn2ps'; }
108 /* additional language identifiers per "defun org-babel-execute"
110 pre.src-cpp:before { content: 'C++'; }
111 pre.src-abc:before { content: 'ABC'; }
112 pre.src-coq:before { content: 'Coq'; }
113 pre.src-groovy:before { content: 'Groovy'; }
114 /* additional language identifiers from org-babel-shell-names in
115 ob-shell.el: ob-shell is the only babel language using a lambda to put
116 the execution function name together. */
117 pre.src-bash:before { content: 'bash'; }
118 pre.src-csh:before { content: 'csh'; }
119 pre.src-ash:before { content: 'ash'; }
120 pre.src-dash:before { content: 'dash'; }
121 pre.src-ksh:before { content: 'ksh'; }
122 pre.src-mksh:before { content: 'mksh'; }
123 pre.src-posh:before { content: 'posh'; }
124 /* Additional Emacs modes also supported by the LaTeX listings package */
125 pre.src-ada:before { content: 'Ada'; }
126 pre.src-asm:before { content: 'Assembler'; }
127 pre.src-caml:before { content: 'Caml'; }
128 pre.src-delphi:before { content: 'Delphi'; }
129 pre.src-html:before { content: 'HTML'; }
130 pre.src-idl:before { content: 'IDL'; }
131 pre.src-mercury:before { content: 'Mercury'; }
132 pre.src-metapost:before { content: 'MetaPost'; }
133 pre.src-modula-2:before { content: 'Modula-2'; }
134 pre.src-pascal:before { content: 'Pascal'; }
135 pre.src-ps:before { content: 'PostScript'; }
136 pre.src-prolog:before { content: 'Prolog'; }
137 pre.src-simula:before { content: 'Simula'; }
138 pre.src-tcl:before { content: 'tcl'; }
139 pre.src-tex:before { content: 'TeX'; }
140 pre.src-plain-tex:before { content: 'Plain TeX'; }
141 pre.src-verilog:before { content: 'Verilog'; }
142 pre.src-vhdl:before { content: 'VHDL'; }
143 pre.src-xml:before { content: 'XML'; }
144 pre.src-nxml:before { content: 'XML'; }
145 /* add a generic configuration mode; LaTeX export needs an additional
146 (add-to-list 'org-latex-listings-langs '(conf " ")) in .emacs */
147 pre.src-conf:before { content: 'Configuration File'; }
149 table { border-collapse:collapse; }
150 caption.t-above { caption-side: top; }
151 caption.t-bottom { caption-side: bottom; }
152 td, th { vertical-align:top; }
153 th.org-right { text-align: center; }
154 th.org-left { text-align: center; }
155 th.org-center { text-align: center; }
156 td.org-right { text-align: right; }
157 td.org-left { text-align: left; }
158 td.org-center { text-align: center; }
159 dt { font-weight: bold; }
160 .footpara { display: inline; }
161 .footdef { margin-bottom: 1em; }
162 .figure { padding: 1em; }
163 .figure p { text-align: center; }
164 .equation-container {
170 vertical-align: middle;
175 vertical-align: middle;
179 border: 2px solid gray;
184 { text-align: right; font-size: 70%; white-space: nowrap; }
185 textarea { overflow-x: auto; }
186 .linenr { font-size: smaller }
187 .code-highlighted { background-color: #ffff00; }
188 .org-info-js_info-navigation { border-style: none; }
189 #org-info-js_console-label
190 { font-size: 10px; font-weight: bold; white-space: nowrap; }
191 .org-info-js_search-highlight
192 { background-color: #ffff00; color: #000000; font-weight: bold; }
193 .org-svg { width: 90%; }
196 <link href="https://bootswatch.com/3/darkly/bootstrap.min.css" rel="stylesheet">
197 <script src="https://cdnjs.cloudflare.com/ajax/libs/jquery/1.11.3/jquery.min.js"></script>
198 <script src="https://cdnjs.cloudflare.com/ajax/libs/twitter-bootstrap/3.3.5/js/bootstrap.min.js"></script>
199 <style type="text/css">
200 footer {background-color: #111 !important;}
201 pre {background-color: #111; color: #ccc;}
203 <link rel="stylesheet" type="text/css" href="https://thomasf.github.io/solarized-css/solarized-dark.min.css" />
204 <script type="text/javascript">
205 // @license magnet:?xt=urn:btih:e95b018ef3580986a04669f1b5879592219e2a7a&dn=public-domain.txt Public Domain
206 <!--/*--><![CDATA[/*><!--*/
207 function CodeHighlightOn(elem, id)
209 var target = document.getElementById(id);
211 elem.classList.add("code-highlighted");
212 target.classList.add("code-highlighted");
215 function CodeHighlightOff(elem, id)
217 var target = document.getElementById(id);
219 elem.classList.remove("code-highlighted");
220 target.classList.remove("code-highlighted");
229 <h1 class="title">Sixth Data - Data storage and computing engine</h1>
230 <div id="table-of-contents">
231 <h2>Table of Contents</h2>
232 <div id="text-table-of-contents">
234 <li><a href="#org9953820">1. General</a>
236 <li><a href="#orgd7c1eb0">1.1. Source code</a></li>
239 <li><a href="#orgc70e97a">2. Vision / goal</a></li>
240 <li><a href="#org48ec5f6">3. Inspiration</a>
242 <li><a href="#orge6af731">3.1. Brain</a></li>
243 <li><a href="#orgb44a34b">3.2. CM-1 Connection Machine</a></li>
246 <li><a href="#org4c94c80">4. Reasons for hypercube as a so called first class citizen</a></li>
247 <li><a href="#orgef7f56d">5. Geometrical computation idea</a>
249 <li><a href="#orga27fcd3">5.1. Distributed computation and data storage</a></li>
250 <li><a href="#org4b4ad58">5.2. Mapping hypercube to object-oriented model and relational database</a></li>
251 <li><a href="#orgaff635e">5.3. Mapping entity relations in hypercube</a></li>
254 <li><a href="#org1319d8e">6. Current status</a></li>
255 <li><a href="#org154c515">7. See also</a>
257 <li><a href="#org2320db5">7.1. Computation on multi dimensional data</a></li>
258 <li><a href="#orge57f4df">7.2. Distributed, reliable, parallel computing systems</a></li>
259 <li><a href="#orgf5f08c4">7.3. Rules based machine reasoning</a></li>
266 <div id="outline-container-org9953820" class="outline-2">
267 <h2 id="org9953820"><span class="section-number-2">1</span> General</h2>
268 <div class="outline-text-2" id="text-1">
270 <li>This program is free software: released under Creative Commons Zero
275 <li>Svjatoslav Agejenko</li>
276 <li>Homepage: <a href="https://svjatoslav.eu">https://svjatoslav.eu</a></li>
277 <li>Email: <a href="mailto://svjatoslav@svjatoslav.eu">mailto://svjatoslav@svjatoslav.eu</a></li>
280 <li><a href="https://www.svjatoslav.eu/projects/">Other software projects hosted at svjatoslav.eu</a></li>
284 <div id="outline-container-orgd7c1eb0" class="outline-3">
285 <h3 id="orgd7c1eb0"><span class="section-number-3">1.1</span> Source code</h3>
286 <div class="outline-text-3" id="text-1-1">
288 <li><a href="https://www2.svjatoslav.eu/gitweb/?p=sixth-data.git;a=snapshot;h=HEAD;sf=tgz">Download latest snapshot in TAR GZ format</a></li>
290 <li><a href="https://www2.svjatoslav.eu/gitweb/?p=sixth-data.git;a=summary">Browse Git repository online</a></li>
293 Clone Git repository using command:
295 <pre class="example">
296 git clone https://www2.svjatoslav.eu/git/sixth-data.git
299 <li>See <a href="https://www3.svjatoslav.eu/projects/sixth-data/apidocs/">JavaDoc</a>.</li>
305 <div id="outline-container-orgc70e97a" class="outline-2">
306 <h2 id="orgc70e97a"><span class="section-number-2">2</span> Vision / goal</h2>
307 <div class="outline-text-2" id="text-2">
309 Provide hackable, versioned, optimized, distributed, geometrical,
310 arbitrary dimensional (<a href="#org4c94c80">hypercube based</a>) data storage and computation
311 engine (<a href="#orge6af731">as inspired by the brain</a>) for general purpose visual computing
312 environment called <a href="http://www2.svjatoslav.eu/gitbrowse/sixth/doc/index.html">Sixth</a>.
316 Because <a href="http://www.paulgraham.com/rootsoflisp.html">Lisp is hackable self defined programmable programming
317 language</a> it would be used to provide <a href="https://en.wikipedia.org/wiki/Imperative_programming">imperative programming</a> support.
321 <div id="outline-container-org48ec5f6" class="outline-2">
322 <h2 id="org48ec5f6"><span class="section-number-2">3</span> Inspiration</h2>
323 <div class="outline-text-2" id="text-3">
325 <li>see also: <a href="https://en.wikipedia.org/wiki/OLAP_cube">OLAP cube</a>.</li>
328 <div id="outline-container-orge6af731" class="outline-3">
329 <h3 id="orge6af731"><span class="section-number-3">3.1</span> Brain</h3>
330 <div class="outline-text-3" id="text-3-1">
332 <li>Brain appears to be natural geometrical/parallel data storage and
333 computational engine:
335 <li><a href="https://www.quantamagazine.org/the-brain-maps-out-ideas-and-memories-like-spaces-20190114/">https://www.quantamagazine.org/the-brain-maps-out-ideas-and-memories-like-spaces-20190114/</a></li>
338 <li>Even more awesome is that brain appears to operate and is wired as
339 arbitrary/variable dimensional structure:
340 <a href="https://singularityhub.com/2017/06/21/is-there-a-multidimensional-mathematical-world-hidden-in-the-brains-computation/">https://singularityhub.com/2017/06/21/is-there-a-multidimensional-mathematical-world-hidden-in-the-brains-computation/</a></li>
342 <li>On top of this, this multidimensional space that brain represents
343 has dynamic/variable resolution/density:
345 <li><a href="https://www.quantamagazine.org/goals-and-rewards-redraw-the-brains-map-of-the-world-20190328">https://www.quantamagazine.org/goals-and-rewards-redraw-the-brains-map-of-the-world-20190328</a></li>
348 <li>Such properties allow parallel <a href="#orgef7f56d">Geometrical computation</a> and
349 beautifully fits <a href="#orgb44a34b">CM-1 Connection Machine</a> architecture (for extra
350 hardware accelerated solution).</li>
355 <div id="outline-container-orgb44a34b" class="outline-3">
356 <h3 id="orgb44a34b"><span class="section-number-3">3.2</span> CM-1 Connection Machine</h3>
357 <div class="outline-text-3" id="text-3-2">
359 <a href="https://en.wikipedia.org/wiki/Connection_Machine">https://en.wikipedia.org/wiki/Connection_Machine</a>
363 Massively parallel (thousands of CPUs) connected via
364 machine's internal 12-dimensional hypercube network allows to
365 efficiently simulate arbitrary dimensional hypercube and network
366 topology between computational units. So that when we are
367 solving/simulating for example 5 dimensional problem, we can arrange
368 computational units into virtual 5D network. See:
369 <a href="http://www.mission-base.com/tamiko/theory/cm_txts/di-ch2.html">http://www.mission-base.com/tamiko/theory/cm_txts/di-ch2.html</a>
373 we can pre-distribute data across computation units and perform
374 parallel <a href="#orgef7f56d">geometrical computation</a>.
380 <div id="outline-container-org4c94c80" class="outline-2">
381 <h2 id="org4c94c80"><span class="section-number-2">4</span> Reasons for hypercube as a so called first class citizen</h2>
382 <div class="outline-text-2" id="text-4">
384 <li>Hypercube is quite general purpose data structure that naturally
385 encapsulates wide variety data and problems.</li>
387 <li>Nicely captures apparent <a href="#orge6af731">properties of the brain</a>.</li>
389 <li>Naturally supports distributed and parallel <a href="#orgef7f56d">geometrical data storage
390 and computation.</a></li>
392 <li>Dedicated hardware like <a href="#orgb44a34b">CM-1</a> can be built around hypercube concept
393 that results in data, computation process and hardware, all
394 beautifully fitting together while complementing each other
397 <li>Hypercube stored data (and computation process) has geometry by its
398 nature and should fit nicely with "3D first" user interface ideology
399 of the parent <a href="http://www2.svjatoslav.eu/gitbrowse/sixth/doc/index.html">Sixth</a> project.</li>
403 <div id="outline-container-orgef7f56d" class="outline-2">
404 <h2 id="orgef7f56d"><span class="section-number-2">5</span> Geometrical computation idea</h2>
405 <div class="outline-text-2" id="text-5">
407 <div id="outline-container-orga27fcd3" class="outline-3">
408 <h3 id="orga27fcd3"><span class="section-number-3">5.1</span> Distributed computation and data storage</h3>
409 <div class="outline-text-3" id="text-5-1">
411 Lots of problems can be translated to geometry (use any shapes and as
412 many dimensions as you need). Solution(s) to such problems could be
413 then found via geometrical search/comparison/lookup results. As a
414 bonus, such geometrical <b>data storage</b> AND <b>computation</b> can be
415 naturally made in <b>parallel</b> and <b>distributed</b>.
419 Learning means building/updating/re-balancing the model (the hard
420 part). Question answering is making (relatively simple) lookups
421 (geometrical queries) against the model.
425 <div id="outline-container-org4b4ad58" class="outline-3">
426 <h3 id="org4b4ad58"><span class="section-number-3">5.2</span> Mapping hypercube to object-oriented model and relational database</h3>
427 <div class="outline-text-3" id="text-5-2">
429 Object oriented programming is inspired by the way human mind
430 operates. It allows programmer to express ideas to computer in a more
435 It is actually also possible to map object model and relational
436 database to geometrical hyperspace:
440 <li>Object or database table row is a point in hypercube arbitrary
441 dimensional space. Each object member variable or database table
442 column can be mapped to its own dimension in hypercube. That is: if
443 class declares 4 variables for an object, then corresponding object
444 can be stored as a single point inside 4 dimensional
445 hypercube. Variable values translate to point coordinates in that
446 hypercube. That is: numbers and string can be translated to linear
447 value that can be used as a coordinate along particular dimension.</li>
449 <li>Each object class or database table declares its own hypercube that
450 contain instances (objects) of that class or rows of a table.</li>
455 <div id="outline-container-orgaff635e" class="outline-3">
456 <h3 id="orgaff635e"><span class="section-number-3">5.3</span> Mapping entity relations in hypercube</h3>
457 <div class="outline-text-3" id="text-5-3">
459 Consider we want to create database of:
464 <li>Effort: Amount of time contributed by every author to every book
465 that he/she wrote.</li>
469 Information above can be represented as 3D cube where dimensions are:
478 Points in that cube would nicely capture many to many relations
479 between authors and the books.
484 <div id="outline-container-org1319d8e" class="outline-2">
485 <h2 id="org1319d8e"><span class="section-number-2">6</span> Current status</h2>
486 <div class="outline-text-2" id="text-6">
488 <li>More or less defined <a href="#orgc70e97a">Vision / goal</a>.</li>
490 <li>Collected some <a href="#org48ec5f6">inspiring</a> <a href="#orgef7f56d">ideas</a>.</li>
492 <li>Implemented very simple persistent key-value map.
494 <li>Long term goal is to use it as a backing storage engine and
495 implement more advanced features on top of this via layered
502 <div id="outline-container-org154c515" class="outline-2">
503 <h2 id="org154c515"><span class="section-number-2">7</span> See also</h2>
504 <div class="outline-text-2" id="text-7">
506 Interesting or competing projects with good ideas:
510 <li>flexible user interface building for interacting with different data
512 <li><a href="https://gtoolkit.com/">Glamorous Toolkit</a>
514 <li>Moldable development environment. It is a live notebook. It is a
515 flexible search interface. It is a fancy code editor. It is a
516 software analysis platform. It is a data visualization engine. All
523 <div id="outline-container-org2320db5" class="outline-3">
524 <h3 id="org2320db5"><span class="section-number-3">7.1</span> Computation on multi dimensional data</h3>
525 <div class="outline-text-3" id="text-7-1">
527 <li><a href="https://rd-alliance.org/system/files/Array-Databases_final-report.pdf">Array Databases: Concepts, Standards, Implementations</a></li>
529 <li><a href="https://tiledb.com/">TileDB</a>
531 <li>Analyze and share complex multi-dimensional data at scale</li>
534 <li><a href="#orgb44a34b">CM-1 Connection Machine</a></li>
536 <li><a href="https://lisp-stat.dev/">Lisp-Stat: An environment for Statistical Computing</a></li>
541 <div id="outline-container-orge57f4df" class="outline-3">
542 <h3 id="orge57f4df"><span class="section-number-3">7.2</span> Distributed, reliable, parallel computing systems</h3>
543 <div class="outline-text-3" id="text-7-2">
545 <li><a href="https://github.com/vygr/ChrysaLisp">ChrysaLisp</a>
547 <li>Assembler/C-Script/Lisp 64 bit, MIMD, multi CPU, multi threaded,
548 multi core, multi user Parallel OS. With GUI, Terminal, OO
549 Assembler, Class libraries, C-Script compiler, Lisp interpreter,
550 Debugger, and more…</li>
553 <li><a href="http://esug.org/data/ESUG2015/3%20wednesday/1100-1130%20SQL%20Queries%20on%20Smalltalk%20Objects/SQL%20Queries%20in%20Smalltalk%20(James%20Foster).pdf">Gemstone/S</a>
555 <li>Completely distributed smalltalk based computing
559 <li><a href="http://phantomos.org/">http://phantomos.org/</a>
561 <li>Programs run forever. System crash or reboot does not destroy
562 state of running program.</li>
565 <li><a href="http://wiki.squeak.org/squeak/2665">Magma</a>
567 <li>Multi-user object database for Squeak</li>
570 <li><a href="http://www.uruk.org/emu/Taos.html">TAOS</a>
572 <li>Completely distributed operating system/virtual machine:</li>
578 <div id="outline-container-orgf5f08c4" class="outline-3">
579 <h3 id="orgf5f08c4"><span class="section-number-3">7.3</span> Rules based machine reasoning</h3>
580 <div class="outline-text-3" id="text-7-3">
584 <li>database in the form of a knowledge graph that uses machine
585 reasoning to simplify data processing challenges for AI
586 applications. <a href="https://grakn.ai/">https://grakn.ai/</a></li>
589 <li><a href="https://en.wikipedia.org/wiki/Prolog">Prolog programming language</a></li>
595 <div id="postamble" class="status">
596 <p class="author">Author: Svjatoslav Agejenko</p>
597 <p class="date">Created: 2021-04-01 Thu 19:11</p>
598 <p class="validation"><a href="https://validator.w3.org/check?uri=referer">Validate</a></p>