added Taichi reference

[sixth-data.git] / doc / index.org

diff --git a/doc/index.org b/doc/index.org
index caa29aa..b81105a 100644 (file)
--- a/doc/index.org
+++ b/doc/index.org
@@ -1,31 +1,39 @@
-#+TITLE: Sixth - system for data storage, computation, exploration and interaction
+#+TITLE: Sixth Data - Data storage and computing engine

 
 

------
-- This is a subproject of [[http://www2.svjatoslav.eu/gitbrowse/sixth/doc/index.html][Sixth]]
+* (document settings) :noexport:
+** use dark style for TWBS-HTML exporter
+#+HTML_HEAD: <link href="https://bootswatch.com/3/darkly/bootstrap.min.css" rel="stylesheet">
+#+HTML_HEAD: <script src="https://cdnjs.cloudflare.com/ajax/libs/jquery/1.11.3/jquery.min.js"></script>
+#+HTML_HEAD: <script src="https://cdnjs.cloudflare.com/ajax/libs/twitter-bootstrap/3.3.5/js/bootstrap.min.js"></script>
+#+HTML_HEAD: <style type="text/css">
+#+HTML_HEAD:   footer {background-color: #111 !important;}
+#+HTML_HEAD:   pre {background-color: #111; color: #ccc;}
+#+HTML_HEAD: </style>

 
 

-- [[http://www2.svjatoslav.eu/gitweb/?p=sixth-data.git;a=snapshot;h=HEAD;sf=tgz][download latest snapshot]]
+* General
+- This is a subproject of [[https://www3.svjatoslav.eu/projects/sixth/][Sixth]]

 
 

-- This program is free software; you can redistribute it and/or modify
-  it under the terms of version 3 of the [[https://www.gnu.org/licenses/lgpl.html][GNU Lesser General Public
-  License]] or later as published by the Free Software Foundation.
+- This program is free software: you can redistribute it and/or modify
+  it under the terms of the [[https://www.gnu.org/licenses/lgpl.html][GNU Lesser General Public License]] as
+  published by the Free Software Foundation, either version 3 of the
+  License, or (at your option) any later version.

 
 - Program author:
   - Svjatoslav Agejenko
 
 - Program author:
   - Svjatoslav Agejenko

-  - Homepage: http://svjatoslav.eu
+  - Homepage: https://svjatoslav.eu

   - Email: mailto://svjatoslav@svjatoslav.eu
 
   - Email: mailto://svjatoslav@svjatoslav.eu
 

-- [[http://www.svjatoslav.eu/programs.jsp][other applications hosted at svjatoslav.eu]]
+- [[https://www.svjatoslav.eu/projects/][Other software projects hosted at svjatoslav.eu]]

 
 

+** Source code
+- [[https://www2.svjatoslav.eu/gitweb/?p=sixth-data.git;a=snapshot;h=HEAD;sf=tgz][Download latest snapshot in TAR GZ format]]

 
 

-* (document settings)                                              :noexport:
-** use dark style for TWBS-HTML exporter
-#+HTML_HEAD: <link href="https://bootswatch.com/darkly/bootstrap.min.css" rel="stylesheet">
-#+HTML_HEAD: <script src="https://cdnjs.cloudflare.com/ajax/libs/jquery/1.11.2/jquery.min.js"></script>
-#+HTML_HEAD: <script src="https://cdnjs.cloudflare.com/ajax/libs/twitter-bootstrap/3.3.1/js/bootstrap.min.js"></script>"
-#+HTML_HEAD: <style type="text/css">
-#+HTML_HEAD:   footer {background-color: #111 !important;}
-#+HTML_HEAD:   pre {background-color: #111; color: #ccc;}
-#+HTML_HEAD: </style>
+- [[https://www2.svjatoslav.eu/gitweb/?p=sixth-data.git;a=summary][Browse Git repository online]]
+
+- Clone Git repository using command:
+  : git clone https://www2.svjatoslav.eu/git/sixth-data.git
+
+- See [[https://www3.svjatoslav.eu/projects/sixth-data/apidocs/][JavaDoc]].

 
 * Vision / goal
   :PROPERTIES:
 
 * Vision / goal
   :PROPERTIES:


@@ -36,15 +44,19 @@ data storage engine for the [[http://www2.svjatoslav.eu/gitbrowse/sixth/doc/inde
 
 + Speaking of traditional relational database and object oriented
   business applications:
 
 + Speaking of traditional relational database and object oriented
   business applications:

+

   + I hate object-relational impedance mismatch.
 
   + I don't like to convert data between persistent database and
   + I hate object-relational impedance mismatch.
 
   + I don't like to convert data between persistent database and

-  runtime objects for every transaction. How about creating united
-  database/computation engine instead to:
-  + Eliminate constant moving and converting of data between 2 systems.
+    runtime objects for every transaction. How about creating united
+    database/computation engine instead to:
+
+  + Eliminate constant moving and converting of data between 2 systems
+    and make computing happen close to where the data is stored.
+

   + Abstract away difference between RAM VS persistent storage. Let
   + Abstract away difference between RAM VS persistent storage. Let

-  the system decide at runtime which data to keep in what kind of
-  memory.
+    the system decide at runtime which data to keep in what kind of
+    memory.

 
 * Inspiration
 + Relational databases:
 
 * Inspiration
 + Relational databases:


@@ -62,17 +74,44 @@ data storage engine for the [[http://www2.svjatoslav.eu/gitbrowse/sixth/doc/inde
   :PROPERTIES:
   :ID:       d2375acc-af14-4f18-8ad0-7949501178c5
   :END:
   :PROPERTIES:
   :ID:       d2375acc-af14-4f18-8ad0-7949501178c5
   :END:

-+ Appears to have more than 3D dimensional design. Food for
-  thought...)
-  + https://singularityhub.com/2017/06/21/is-there-a-multidimensional-mathematical-world-hidden-in-the-brains-computation/
++ Brain appears to have more than 3D dimensional design:
+  https://singularityhub.com/2017/06/21/is-there-a-multidimensional-mathematical-world-hidden-in-the-brains-computation/
+
++ Brain appears to use geometry to map thoughts and even sounds:
+  + https://www.quantamagazine.org/the-brain-maps-out-ideas-and-memories-like-spaces-20190114/
+  + https://www.quantamagazine.org/goals-and-rewards-redraw-the-brains-map-of-the-world-20190328
+
++ It directly inspires [[id:171fe375-c737-41e6-b429-a414f6abc5d8][Geometrical computation]] idea and nicely fits
+  with [[id:01aa65c1-3d44-44a8-9b90-58454bc6be80][CM-1 Connection Machine]] design.
+
+** CM-1 Connection Machine
+:PROPERTIES:
+:ID:       01aa65c1-3d44-44a8-9b90-58454bc6be80
+:END:
+https://en.wikipedia.org/wiki/Connection_Machine
+
++ see: [[id:171fe375-c737-41e6-b429-a414f6abc5d8][Geometrical computation]]
+
++ Computation unit has local CPU and RAM.
+
++ Data is pre-distributed across computation units.

 
 

-+ It directly inspires following ideas
-  + [[id:5d287158-53ea-44a2-a754-dd862366066a][Distributed comutation and data storage]]
-  + [[id:a117c11e-97c1-4822-88b2-9fc10f96caec][Mapping of hyperspace to traditional object-oriented model]]
-  + [[id:b6b15bd2-c78b-4c51-a343-72843a515c29][Handling of relations]]
++ Machine's internal 12-dimensional hypercube network allows to
+  efficiently simulate arbitrary dimensional network topology between
+  computational units. So that when we are solving/simulating for
+  example 5 dimensional problem, we can arrange computational units
+  into virtual 5D network. See:
+  http://www.mission-base.com/tamiko/theory/cm_txts/di-ch2.html

 
 * Ideas
 
 * Ideas

-** Distributed computation and data storage
+** Geometrical computation
+:PROPERTIES:
+:ID:       171fe375-c737-41e6-b429-a414f6abc5d8
+:END:
++ Inspired by [[id:d2375acc-af14-4f18-8ad0-7949501178c5][Brain]].
++ Wits nicely with [[id:01aa65c1-3d44-44a8-9b90-58454bc6be80][CM-1 Connection Machine]] properties.
+
+*** Distributed computation and data storage

    :PROPERTIES:
    :ID:       5d287158-53ea-44a2-a754-dd862366066a
    :END:
    :PROPERTIES:
    :ID:       5d287158-53ea-44a2-a754-dd862366066a
    :END:


@@ -84,7 +123,7 @@ naturally made in *parallel* and *distributed*. That's what neurons in
 the brain appear to be doing ! :) . Learning means building/updating
 the model (the hard part). Question answering is making (relatively
 simple) lookups (geometrical queries) against the model.
 the brain appear to be doing ! :) . Learning means building/updating
 the model (the hard part). Question answering is making (relatively
 simple) lookups (geometrical queries) against the model.

-** Mapping of hyperspace to traditional object-oriented model
+*** Mapping of hyperspace to traditional object-oriented model

    :PROPERTIES:
    :ID:       a117c11e-97c1-4822-88b2-9fc10f96caec
    :END:
    :PROPERTIES:
    :ID:       a117c11e-97c1-4822-88b2-9fc10f96caec
    :END:


@@ -106,7 +145,7 @@ It is possible to map object model to geometrical hyperspace:
   (objects) are points inside that particular universe. References
   between objects of different types are hyperlinks (portals) between
   different universes.
   (objects) are points inside that particular universe. References
   between objects of different types are hyperlinks (portals) between
   different universes.

-** Handling of relations
+*** Handling of relations

    :PROPERTIES:
    :ID:       b6b15bd2-c78b-4c51-a343-72843a515c29
    :END:
    :PROPERTIES:
    :ID:       b6b15bd2-c78b-4c51-a343-72843a515c29
    :END:


@@ -130,7 +169,25 @@ Alternatively:
     + Point location along particular (book) dimension corresponds to
       amount of work contributed for book by given author (point).
 
     + Point location along particular (book) dimension corresponds to
       amount of work contributed for book by given author (point).
 

+** Layered architecture
++ layer 1 :: disk / block storage / partition

 
 

++ layer 2 :: key/value storage. Keys are unique and are dictated by
+             storage engine. Value is arbitrary but limited size byte
+             array. This layer is responsible for handling disk
+             defragmentation and consistency in case of crash
+             recovery.
+
++ layer 3 :: key/value storage. Keys are content hashes. Values are
+             arbitrary but limited size content byte arrays. This
+             layer effectively implements content addressable
+             storage. Content addressible storage enables GIT-like
+             behavior (possibility for competing branches, retaining
+             history, transparent deduplication)
+
++ layer 4 :: Implements arbitrary dimensional multiverse.
+
++ layer 5 :: Distributed computation engine.

 * Current status
 - More or less defined [[id:f6764282-a6f6-44e6-8716-b428074dd093][Vision / goal]].
 
 * Current status
 - More or less defined [[id:f6764282-a6f6-44e6-8716-b428074dd093][Vision / goal]].
 


@@ -143,16 +200,29 @@ Alternatively:
 * See also
 Interesting or competing projects with good ideas:
 
 * See also
 Interesting or competing projects with good ideas:
 

-+ GRAKN.AI: database in the form of a knowledge graph that uses
-  machine reasoning to simplify data processing challenges for AI
-  applications.
-  + https://grakn.ai/
++ [[id:01aa65c1-3d44-44a8-9b90-58454bc6be80][CM-1 Connection Machine]]
+
++ Taichi: A Language for High-Performance Computation onSpatially
+  Sparse Data Structures
+  + http://taichi.graphics/wp-content/uploads/2019/09/taichi_lang.pdf
+
++ GRAKN.AI
+  + database in the form of a knowledge graph that uses machine
+    reasoning to simplify data processing challenges for AI
+    applications. https://grakn.ai/
+
++ [[http://wiki.squeak.org/squeak/2665][Magma]]
+  + Multi-user object database for Squeak

 
 

-+ Gemstone/S based on Smalltalk.
-  + http://esug.org/data/ESUG2015/3%20wednesday/1100-1130%20SQL%20Queries%20on%20Smalltalk%20Objects/SQL%20Queries%20in%20Smalltalk%20(James%20Foster).pdf
++ [[http://esug.org/data/ESUG2015/3%20wednesday/1100-1130%20SQL%20Queries%20on%20Smalltalk%20Objects/SQL%20Queries%20in%20Smalltalk%20(James%20Foster).pdf][Gemstone/S]]
+  + Completely distributed smalltalk based computing
+    system.

 
 

-+ Magma distributed database in Smalltalk.
-  + http://wiki.squeak.org/squeak/2665
++ [[http://www.uruk.org/emu/Taos.html][TAOS]]
+  + Completely distributed operating system/virtual machine:

 
 

-+ ZetaVM
-  + https://github.com/zetavm/zetavm
++ [[https://github.com/vygr/ChrysaLisp][ChrysaLisp]]
+  + Assembler/C-Script/Lisp 64 bit, MIMD, multi CPU, multi threaded,
+    multi core, multi user Parallel OS. With GUI, Terminal, OO
+    Assembler, Class libraries, C-Script compiler, Lisp interpreter,
+    Debugger, and more...