1 #+TITLE: Sixth Data - Data storage and computing engine
3 * (document settings) :noexport:
4 ** use dark style for TWBS-HTML exporter
5 #+HTML_HEAD: <link href="https://bootswatch.com/3/darkly/bootstrap.min.css" rel="stylesheet">
6 #+HTML_HEAD: <script src="https://cdnjs.cloudflare.com/ajax/libs/jquery/1.11.3/jquery.min.js"></script>
7 #+HTML_HEAD: <script src="https://cdnjs.cloudflare.com/ajax/libs/twitter-bootstrap/3.3.5/js/bootstrap.min.js"></script>
8 #+HTML_HEAD: <style type="text/css">
9 #+HTML_HEAD: footer {background-color: #111 !important;}
10 #+HTML_HEAD: pre {background-color: #111; color: #ccc;}
14 - This is a subproject of [[http://www3.svjatoslav.eu/projects/sixth/][Sixth]]
16 - This program is free software: you can redistribute it and/or modify
17 it under the terms of the [[https://www.gnu.org/licenses/lgpl.html][GNU Lesser General Public License]] as
18 published by the Free Software Foundation, either version 3 of the
19 License, or (at your option) any later version.
23 - Homepage: http://svjatoslav.eu
24 - Email: mailto://svjatoslav@svjatoslav.eu
26 - [[http://www.svjatoslav.eu/projects/][Other software projects hosted at svjatoslav.eu]]
29 - [[http://www2.svjatoslav.eu/gitweb/?p=sixth-data.git;a=snapshot;h=HEAD;sf=tgz][Download latest snapshot in TAR GZ format]]
31 - [[http://www2.svjatoslav.eu/gitweb/?p=sixth-data.git;a=summary][Browse Git repository online]]
33 - Clone Git repository using command:
34 : git clone http://www2.svjatoslav.eu/git/sixth-data.git
38 :ID: f6764282-a6f6-44e6-8716-b428074dd093
40 Provide versioned, clustered, flexible, distributed, multi-dimensional
41 data storage engine for the [[http://www2.svjatoslav.eu/gitbrowse/sixth/doc/index.html][Sixth computation engine]].
43 + Speaking of traditional relational database and object oriented
44 business applications:
46 + I hate object-relational impedance mismatch.
48 + I don't like to convert data between persistent database and
49 runtime objects for every transaction. How about creating united
50 database/computation engine instead to:
52 + Eliminate constant moving and converting of data between 2 systems
53 and make computing happen close to where the data is stored.
55 + Abstract away difference between RAM VS persistent storage. Let
56 the system decide at runtime which data to keep in what kind of
60 + Relational databases:
62 + Indexable / Quickly searchable.
64 + Git (version control system)
66 + Branchable / mergeable.
67 + Transparent cansistency, checksumming and deduplication.
69 https://www.kenneth-truyers.net/2016/10/13/git-nosql-database/ )
73 :ID: d2375acc-af14-4f18-8ad0-7949501178c5
75 + Appears to have more than 3D dimensional design. Food for
77 + https://singularityhub.com/2017/06/21/is-there-a-multidimensional-mathematical-world-hidden-in-the-brains-computation/
79 + It directly inspires following ideas
80 + [[id:5d287158-53ea-44a2-a754-dd862366066a][Distributed comutation and data storage]]
81 + [[id:a117c11e-97c1-4822-88b2-9fc10f96caec][Mapping of hyperspace to traditional object-oriented model]]
82 + [[id:b6b15bd2-c78b-4c51-a343-72843a515c29][Handling of relations]]
84 ** Distributed computation and data storage
86 :ID: 5d287158-53ea-44a2-a754-dd862366066a
88 Maybe every problem can be translated to geometry (use any shapes and
89 as many dimensions as you need). Solution(s) to such problems would
90 then appear as relatively simple search/comparison/lookup results. As
91 a bonus, such geometrical *data storage* AND *computation* can be
92 naturally made in *parallel* and *distributed*. That's what neurons in
93 the brain appear to be doing ! :) . Learning means building/updating
94 the model (the hard part). Question answering is making (relatively
95 simple) lookups (geometrical queries) against the model.
96 ** Mapping of hyperspace to traditional object-oriented model
98 :ID: a117c11e-97c1-4822-88b2-9fc10f96caec
100 Object oriented programming is inspired by the way human mind
101 operates. It allows programmer to express ideas to computer in a more
104 It is possible to map object model to geometrical hyperspace:
106 + Object is a point in space (universe). Each object member variable
107 translates to its own dimension. That is: if class declares 4
108 variables for an object, then corresponding object can be stored as
109 a single point inside 4 dimensional space. Variable values translate
110 to point coordinates in space. That is: Integer, floating point
111 number and even boolean and string can be translated to linear value
112 that can be used as a coordinate along particular dimension.
114 + Each class declares its own space (universe). All class instances
115 (objects) are points inside that particular universe. References
116 between objects of different types are hyperlinks (portals) between
118 ** Handling of relations
120 :ID: b6b15bd2-c78b-4c51-a343-72843a515c29
122 Consider we want to create database of books and authors. Book can
123 have multiple authors, and single person can be author for multiple
124 books. It is possible to store how many hours of work each author has
125 contributed to every book, using hyperspace as follows:
127 + Every dimension corresponds to one particular book author. (10
128 authors in the database, would require 10 dimensional space)
129 + Point in space corresponds to one particular book.
130 + Point location along particular (author) dimension corresponds
131 to amount of work contributed by particular author for given
136 + Every dimension corresponds to one particular book.
137 + Point in space corresponds to one particular author in the entire
139 + Point location along particular (book) dimension corresponds to
140 amount of work contributed for book by given author (point).
142 ** Layered architecture
143 + layer 1 :: disk / block storage / partition
145 + layer 2 :: key/value storage. Keys are unique and are dictated by
146 storage engine. Value is arbitrary but limited size byte
147 array. This layer is responsible for handling disk
148 defragmentation and consistency in case of crash
151 + layer 3 :: key/value storage. Keys are content hashes. Values are
152 arbitrary but limited size content byte arrays. This
153 layer effectively implements content addressable
154 storage. Content addressible storage enables GIT-like
155 behavior (possibility for competing branches, retaining
156 history, transparent deduplication)
158 + layer 4 :: Implements arbitrary dimensional multiverse.
160 + layer 5 :: Distributed computation engine.
162 - More or less defined [[id:f6764282-a6f6-44e6-8716-b428074dd093][Vision / goal]].
164 - Collected some [[id:d2375acc-af14-4f18-8ad0-7949501178c5][ideas]].
166 - Implemented very simple persistent key-value map.
167 - Long term goal is to use it as a backing storage engine and
168 implement more advanced features on top of this.
171 Interesting or competing projects with good ideas:
174 + database in the form of a knowledge graph that uses machine
175 reasoning to simplify data processing challenges for AI
176 applications. https://grakn.ai/
178 + [[http://wiki.squeak.org/squeak/2665][Magma]]
179 + Multi-user object database for Squeak
181 + [[http://esug.org/data/ESUG2015/3%20wednesday/1100-1130%20SQL%20Queries%20on%20Smalltalk%20Objects/SQL%20Queries%20in%20Smalltalk%20(James%20Foster).pdf][Gemstone/S]]
182 + Completely distributed smalltalk based computing
185 + [[http://www.uruk.org/emu/Taos.html][TAOS]]
186 + Completely distributed operating system/virtual machine:
188 + [[https://github.com/vygr/ChrysaLisp][ChrysaLisp]]
189 + Assembler/C-Script/Lisp 64 bit, MIMD, multi CPU, multi threaded,
190 multi core, multi user Parallel OS. With GUI, Terminal, OO
191 Assembler, Class libraries, C-Script compiler, Lisp interpreter,
192 Debugger, and more...