1 #+TITLE: Sixth Data - Data storage and computing engine
3 * (document settings) :noexport:
4 ** use dark style for TWBS-HTML exporter
5 #+HTML_HEAD: <link href="https://bootswatch.com/3/darkly/bootstrap.min.css" rel="stylesheet">
6 #+HTML_HEAD: <script src="https://cdnjs.cloudflare.com/ajax/libs/jquery/1.11.3/jquery.min.js"></script>
7 #+HTML_HEAD: <script src="https://cdnjs.cloudflare.com/ajax/libs/twitter-bootstrap/3.3.5/js/bootstrap.min.js"></script>
8 #+HTML_HEAD: <style type="text/css">
9 #+HTML_HEAD: footer {background-color: #111 !important;}
10 #+HTML_HEAD: pre {background-color: #111; color: #ccc;}
14 - This is a subproject of [[https://www3.svjatoslav.eu/projects/sixth/][Sixth]] with the goal of providing data
15 storage and computation facilities.
17 - This program is free software: you can redistribute it and/or modify
18 it under the terms of the [[https://www.gnu.org/licenses/lgpl.html][GNU Lesser General Public License]] as
19 published by the Free Software Foundation, either version 3 of the
20 License, or (at your option) any later version.
24 - Homepage: https://svjatoslav.eu
25 - Email: mailto://svjatoslav@svjatoslav.eu
27 - [[https://www.svjatoslav.eu/projects/][Other software projects hosted at svjatoslav.eu]]
30 - [[https://www2.svjatoslav.eu/gitweb/?p=sixth-data.git;a=snapshot;h=HEAD;sf=tgz][Download latest snapshot in TAR GZ format]]
32 - [[https://www2.svjatoslav.eu/gitweb/?p=sixth-data.git;a=summary][Browse Git repository online]]
34 - Clone Git repository using command:
35 : git clone https://www2.svjatoslav.eu/git/sixth-data.git
39 :ID: f6764282-a6f6-44e6-8716-b428074dd093
41 Provide hackable, versioned, optimized, distributed, geometrical,
42 arbitrary dimensional ([[id:96116550-a6a1-4700-bef7-865d0deee7ea][hypercube based]]) data storage and computation
43 engine ([[id:d2375acc-af14-4f18-8ad0-7949501178c5][as inspired by the brain]]) for general purpose visual computing
44 environment called [[http://www2.svjatoslav.eu/gitbrowse/sixth/doc/index.html][Sixth]].
46 Because [[http://www.paulgraham.com/rootsoflisp.html][Lisp is hackable self defined programmable programming
47 language]] it would be used to provide [[https://en.wikipedia.org/wiki/Imperative_programming][imperative programming]] support.
50 :ID: 0fa6354b-18c9-4120-bbf5-c7239aebecab
52 + see also: [[https://en.wikipedia.org/wiki/OLAP_cube][OLAP cube]].
55 :ID: d2375acc-af14-4f18-8ad0-7949501178c5
57 + Brain appears to be natural geometrical/parallel data storage and
59 + https://www.quantamagazine.org/the-brain-maps-out-ideas-and-memories-like-spaces-20190114/
61 + Even more awesome is that brain appears to operate and is wired as
62 arbitrary/variable dimensional structure:
63 https://singularityhub.com/2017/06/21/is-there-a-multidimensional-mathematical-world-hidden-in-the-brains-computation/
65 + On top of this, this multidimensional space that brain represents
66 has dynamic/variable resolution/density:
67 + https://www.quantamagazine.org/goals-and-rewards-redraw-the-brains-map-of-the-world-20190328
69 + Such properties allow parallel [[id:171fe375-c737-41e6-b429-a414f6abc5d8][Geometrical computation]] and
70 beautifully fits [[id:01aa65c1-3d44-44a8-9b90-58454bc6be80][CM-1 Connection Machine]] architecture (for extra
71 hardware accelerated solution).
73 ** CM-1 Connection Machine
75 :ID: 01aa65c1-3d44-44a8-9b90-58454bc6be80
77 https://en.wikipedia.org/wiki/Connection_Machine
79 Massively parallel (thousands of CPUs) connected via
80 machine's internal 12-dimensional hypercube network allows to
81 efficiently simulate arbitrary dimensional hypercube and network
82 topology between computational units. So that when we are
83 solving/simulating for example 5 dimensional problem, we can arrange
84 computational units into virtual 5D network. See:
85 http://www.mission-base.com/tamiko/theory/cm_txts/di-ch2.html
87 we can pre-distribute data across computation units and perform
88 parallel [[id:171fe375-c737-41e6-b429-a414f6abc5d8][geometrical computation]].
90 * Reasons for hypercube as a so called first class citizen
92 :ID: 96116550-a6a1-4700-bef7-865d0deee7ea
94 + Hypercube is quite general purpose data structure that naturally
95 encapsulates wide variety data and problems.
97 + Nicely captures apparent [[id:d2375acc-af14-4f18-8ad0-7949501178c5][properties of the brain]].
99 + Naturally supports distributed and parallel [[id:171fe375-c737-41e6-b429-a414f6abc5d8][geometrical data storage
102 + Dedicated hardware like [[id:01aa65c1-3d44-44a8-9b90-58454bc6be80][CM-1]] can be built around hypercube concept
103 that results in data, computation process and hardware, all
104 beautifully fitting together while complementing each other
107 + Hypercube stored data (and computation process) has geometry by its
108 nature and should fit nicely with "3D first" user interface ideology
109 of the parent [[http://www2.svjatoslav.eu/gitbrowse/sixth/doc/index.html][Sixth]] project.
110 * Geometrical computation idea
112 :ID: 171fe375-c737-41e6-b429-a414f6abc5d8
114 ** Distributed computation and data storage
116 :ID: 5d287158-53ea-44a2-a754-dd862366066a
118 Lots of problems can be translated to geometry (use any shapes and as
119 many dimensions as you need). Solution(s) to such problems could be
120 then found via geometrical search/comparison/lookup results. As a
121 bonus, such geometrical *data storage* AND *computation* can be
122 naturally made in *parallel* and *distributed*.
124 Learning means building/updating/re-balancing the model (the hard
125 part). Question answering is making (relatively simple) lookups
126 (geometrical queries) against the model.
127 ** Mapping hypercube to object-oriented model and relational database
129 :ID: a117c11e-97c1-4822-88b2-9fc10f96caec
131 Object oriented programming is inspired by the way human mind
132 operates. It allows programmer to express ideas to computer in a more
135 It is actually also possible to map object model and relational
136 database to geometrical hyperspace:
138 + Object or database table row is a point in hypercube arbitrary
139 dimensional space. Each object member variable or database table
140 column can be mapped to its own dimension in hypercube. That is: if
141 class declares 4 variables for an object, then corresponding object
142 can be stored as a single point inside 4 dimensional
143 hypercube. Variable values translate to point coordinates in that
144 hypercube. That is: numbers and string can be translated to linear
145 value that can be used as a coordinate along particular dimension.
147 + Each object class or database table declares its own hypercube that
148 contain instances (objects) of that class or rows of a table.
150 ** Mapping entity relations in hypercube
152 :ID: b6b15bd2-c78b-4c51-a343-72843a515c29
154 Consider we want to create database of:
157 + Effort: Amount of time contributed by every author to every book
160 Information above can be represented as 3D cube where dimensions are:
165 Points in that cube would nicely capture many to many relations
166 between authors and the books.
168 - More or less defined [[id:f6764282-a6f6-44e6-8716-b428074dd093][Vision / goal]].
170 - Collected some [[id:0fa6354b-18c9-4120-bbf5-c7239aebecab][inspiring]] [[id:171fe375-c737-41e6-b429-a414f6abc5d8][ideas]].
172 - Implemented very simple persistent key-value map.
173 - Long term goal is to use it as a backing storage engine and
174 implement more advanced features on top of this via layered
178 Interesting or competing projects with good ideas:
180 + [[id:01aa65c1-3d44-44a8-9b90-58454bc6be80][CM-1 Connection Machine]]
182 + http://phantomos.org/
183 + Programs run forever. System crash or reboot does not destroy
184 state of running program.
186 + Taichi: A Language for High-Performance Computation onSpatially
187 Sparse Data Structures
188 + http://taichi.graphics/wp-content/uploads/2019/09/taichi_lang.pdf
191 + database in the form of a knowledge graph that uses machine
192 reasoning to simplify data processing challenges for AI
193 applications. https://grakn.ai/
195 + [[http://wiki.squeak.org/squeak/2665][Magma]]
196 + Multi-user object database for Squeak
198 + [[http://esug.org/data/ESUG2015/3%20wednesday/1100-1130%20SQL%20Queries%20on%20Smalltalk%20Objects/SQL%20Queries%20in%20Smalltalk%20(James%20Foster).pdf][Gemstone/S]]
199 + Completely distributed smalltalk based computing
202 + [[http://www.uruk.org/emu/Taos.html][TAOS]]
203 + Completely distributed operating system/virtual machine:
205 + [[https://github.com/vygr/ChrysaLisp][ChrysaLisp]]
206 + Assembler/C-Script/Lisp 64 bit, MIMD, multi CPU, multi threaded,
207 multi core, multi user Parallel OS. With GUI, Terminal, OO
208 Assembler, Class libraries, C-Script compiler, Lisp interpreter,
209 Debugger, and more...