1 #+TITLE: Sixth Data - Data storage and computing engine
3 * (document settings) :noexport:
4 ** use dark style for TWBS-HTML exporter
5 #+HTML_HEAD: <link href="https://bootswatch.com/3/darkly/bootstrap.min.css" rel="stylesheet">
6 #+HTML_HEAD: <script src="https://cdnjs.cloudflare.com/ajax/libs/jquery/1.11.3/jquery.min.js"></script>
7 #+HTML_HEAD: <script src="https://cdnjs.cloudflare.com/ajax/libs/twitter-bootstrap/3.3.5/js/bootstrap.min.js"></script>
8 #+HTML_HEAD: <style type="text/css">
9 #+HTML_HEAD: footer {background-color: #111 !important;}
10 #+HTML_HEAD: pre {background-color: #111; color: #ccc;}
14 - This program is free software: released under Creative Commons Zero
19 - Homepage: https://svjatoslav.eu
20 - Email: mailto://svjatoslav@svjatoslav.eu
22 - [[https://www.svjatoslav.eu/projects/][Other software projects hosted at svjatoslav.eu]]
25 - [[https://www2.svjatoslav.eu/gitweb/?p=sixth-data.git;a=snapshot;h=HEAD;sf=tgz][Download latest snapshot in TAR GZ format]]
27 - [[https://www2.svjatoslav.eu/gitweb/?p=sixth-data.git;a=summary][Browse Git repository online]]
29 - Clone Git repository using command:
30 : git clone https://www2.svjatoslav.eu/git/sixth-data.git
32 - See [[https://www3.svjatoslav.eu/projects/sixth-data/apidocs/][JavaDoc]].
36 :ID: f6764282-a6f6-44e6-8716-b428074dd093
38 Provide hackable, versioned, optimized, distributed, geometrical,
39 arbitrary dimensional ([[id:96116550-a6a1-4700-bef7-865d0deee7ea][hypercube based]]) data storage and computation
40 engine ([[id:d2375acc-af14-4f18-8ad0-7949501178c5][as inspired by the brain]]) for general purpose visual computing
41 environment called [[http://www2.svjatoslav.eu/gitbrowse/sixth/doc/index.html][Sixth]].
43 Because [[http://www.paulgraham.com/rootsoflisp.html][Lisp is hackable self defined programmable programming
44 language]] it would be used to provide [[https://en.wikipedia.org/wiki/Imperative_programming][imperative programming]] support.
47 :ID: 0fa6354b-18c9-4120-bbf5-c7239aebecab
49 + see also: [[https://en.wikipedia.org/wiki/OLAP_cube][OLAP cube]].
52 :ID: d2375acc-af14-4f18-8ad0-7949501178c5
54 + Brain appears to be natural geometrical/parallel data storage and
56 + https://www.quantamagazine.org/the-brain-maps-out-ideas-and-memories-like-spaces-20190114/
57 + [[https://www.simonsfoundation.org/2021/04/07/geometrical-thinking-offers-a-window-into-computation/][Geometrical Thinking Offers a Window Into Computation]]
59 + Even more awesome is that brain appears to operate and is wired as
60 arbitrary/variable dimensional structure:
61 https://singularityhub.com/2017/06/21/is-there-a-multidimensional-mathematical-world-hidden-in-the-brains-computation/
63 + On top of this, this multidimensional space that brain represents
64 has dynamic/variable resolution/density:
65 + https://www.quantamagazine.org/goals-and-rewards-redraw-the-brains-map-of-the-world-20190328
67 + Such properties allow parallel [[id:171fe375-c737-41e6-b429-a414f6abc5d8][Geometrical computation]] and
68 beautifully fits [[id:01aa65c1-3d44-44a8-9b90-58454bc6be80][CM-1 Connection Machine]] architecture (for extra
69 hardware accelerated solution).
72 ** CM-1 Connection Machine
74 :ID: 01aa65c1-3d44-44a8-9b90-58454bc6be80
76 https://en.wikipedia.org/wiki/Connection_Machine
78 Massively parallel (thousands of CPUs) connected via
79 machine's internal 12-dimensional hypercube network allows to
80 efficiently simulate arbitrary dimensional hypercube and network
81 topology between computational units. So that when we are
82 solving/simulating for example 5 dimensional problem, we can arrange
83 computational units into virtual 5D network. See:
84 http://www.mission-base.com/tamiko/theory/cm_txts/di-ch2.html
86 we can pre-distribute data across computation units and perform
87 parallel [[id:171fe375-c737-41e6-b429-a414f6abc5d8][geometrical computation]].
89 * Reasons for hypercube as a so called first class citizen
91 :ID: 96116550-a6a1-4700-bef7-865d0deee7ea
93 + Hypercube is quite general purpose data structure that naturally
94 encapsulates wide variety data and problems.
96 + Nicely captures apparent [[id:d2375acc-af14-4f18-8ad0-7949501178c5][properties of the brain]].
98 + Naturally supports distributed and parallel [[id:171fe375-c737-41e6-b429-a414f6abc5d8][geometrical data storage
101 + Dedicated hardware like [[id:01aa65c1-3d44-44a8-9b90-58454bc6be80][CM-1]] can be built around hypercube concept
102 that results in data, computation process and hardware, all
103 beautifully fitting together while complementing each other
106 + Hypercube stored data (and computation process) has geometry by its
107 nature and should fit nicely with "3D first" user interface ideology
108 of the parent [[http://www2.svjatoslav.eu/gitbrowse/sixth/doc/index.html][Sixth]] project.
109 * Geometrical computation idea
111 :ID: 171fe375-c737-41e6-b429-a414f6abc5d8
113 ** Distributed computation and data storage
115 :ID: 5d287158-53ea-44a2-a754-dd862366066a
117 Lots of problems can be translated to geometry (use any shapes and as
118 many dimensions as you need). Solution(s) to such problems could be
119 then found via geometrical search/comparison/lookup results. As a
120 bonus, such geometrical *data storage* AND *computation* can be
121 naturally made in *parallel* and *distributed*.
123 Learning means building/updating/re-balancing the model (the hard
124 part). Question answering is making (relatively simple) lookups
125 (geometrical queries) against the model.
126 ** Mapping hypercube to object-oriented model and relational database
128 :ID: a117c11e-97c1-4822-88b2-9fc10f96caec
130 Object oriented programming is inspired by the way human mind
131 operates. It allows programmer to express ideas to computer in a more
134 It is actually also possible to map object model and relational
135 database to geometrical hyperspace:
137 + Object or database table row is a point in hypercube arbitrary
138 dimensional space. Each object member variable or database table
139 column can be mapped to its own dimension in hypercube. That is: if
140 class declares 4 variables for an object, then corresponding object
141 can be stored as a single point inside 4 dimensional
142 hypercube. Variable values translate to point coordinates in that
143 hypercube. That is: numbers and string can be translated to linear
144 value that can be used as a coordinate along particular dimension.
146 + Each object class or database table declares its own hypercube that
147 contain instances (objects) of that class or rows of a table.
149 ** Mapping entity relations in hypercube
151 :ID: b6b15bd2-c78b-4c51-a343-72843a515c29
153 Consider we want to create database of:
156 + Effort: Amount of time contributed by every author to every book
159 Information above can be represented as 3D cube where dimensions are:
164 Points in that cube would nicely capture many to many relations
165 between authors and the books.
167 - More or less defined [[id:f6764282-a6f6-44e6-8716-b428074dd093][Vision / goal]].
169 - Collected some [[id:0fa6354b-18c9-4120-bbf5-c7239aebecab][inspiring]] [[id:171fe375-c737-41e6-b429-a414f6abc5d8][ideas]].
171 - Implemented very simple persistent key-value map.
172 - Long term goal is to use it as a backing storage engine and
173 implement more advanced features on top of this via layered
177 Interesting or competing projects with good ideas:
179 + flexible user interface building for interacting with different data
180 + [[https://gtoolkit.com/][Glamorous Toolkit]]
181 + Moldable development environment. It is a live notebook. It is a
182 flexible search interface. It is a fancy code editor. It is a
183 software analysis platform. It is a data visualization engine. All
186 ** Computation on multi dimensional data
187 + [[https://rd-alliance.org/system/files/Array-Databases_final-report.pdf][Array Databases: Concepts, Standards, Implementations]]
189 + [[https://tiledb.com/][TileDB]]
190 + Analyze and share complex multi-dimensional data at scale
192 + [[id:01aa65c1-3d44-44a8-9b90-58454bc6be80][CM-1 Connection Machine]]
194 + [[https://lisp-stat.dev/][Lisp-Stat: An environment for Statistical Computing]]
196 ** Distributed, reliable, parallel computing systems
197 + [[https://github.com/vygr/ChrysaLisp][ChrysaLisp]]
198 + Assembler/C-Script/Lisp 64 bit, MIMD, multi CPU, multi threaded,
199 multi core, multi user Parallel OS. With GUI, Terminal, OO
200 Assembler, Class libraries, C-Script compiler, Lisp interpreter,
201 Debugger, and more...
203 + [[http://esug.org/data/ESUG2015/3%20wednesday/1100-1130%20SQL%20Queries%20on%20Smalltalk%20Objects/SQL%20Queries%20in%20Smalltalk%20(James%20Foster).pdf][Gemstone/S]]
204 + Completely distributed smalltalk based computing
207 + http://phantomos.org/
208 + Programs run forever. System crash or reboot does not destroy
209 state of running program.
211 + [[http://wiki.squeak.org/squeak/2665][Magma]]
212 + Multi-user object database for Squeak
214 + [[http://www.uruk.org/emu/Taos.html][TAOS]]
215 + Completely distributed operating system/virtual machine:
217 ** Rules based machine reasoning
219 + database in the form of a knowledge graph that uses machine
220 reasoning to simplify data processing challenges for AI
221 applications. https://grakn.ai/
223 + [[https://en.wikipedia.org/wiki/Prolog][Prolog programming language]]