<!doctype html>
<html lang="en">
<head>
-<title>Sixth - system for data storage, computation, exploration and interaction</title>
-<!-- 2017-06-13 Tue 22:14 -->
+<title>Sixth Data - Data storage and computing engine</title>
+<!-- 2020-06-20 Sat 03:28 -->
<meta charset="utf-8">
<meta name="viewport" content="width=device-width, initial-scale=1">
<meta name="generator" content="Org-mode">
<meta name="author" content="Svjatoslav Agejenko">
-<link href="https://bootswatch.com/darkly/bootstrap.min.css" rel="stylesheet">
-<script src="https://cdnjs.cloudflare.com/ajax/libs/jquery/1.11.2/jquery.min.js"></script>
-<script src="https://cdnjs.cloudflare.com/ajax/libs/twitter-bootstrap/3.3.1/js/bootstrap.min.js"></script>"
+<link href="https://bootswatch.com/3/darkly/bootstrap.min.css" rel="stylesheet">
+<script src="https://cdnjs.cloudflare.com/ajax/libs/jquery/1.11.3/jquery.min.js"></script>
+<script src="https://cdnjs.cloudflare.com/ajax/libs/twitter-bootstrap/3.3.5/js/bootstrap.min.js"></script>
<style type="text/css">
footer {background-color: #111 !important;}
pre {background-color: #111; color: #ccc;}
</head>
<body>
<div id="content" class="container">
-<div class="row"><div class="col-md-9"><h1 class="title">Sixth - system for data storage, computation, exploration and interaction</h1>
-<hr >
-<ul class="org-ul">
-<li>This is a subproject of <a href="http://www2.svjatoslav.eu/gitbrowse/sixth/doc/index.html">Sixth</a>
-</li>
+<div class="row"><div class="col-md-9"><h1 class="title">Sixth Data - Data storage and computing engine</h1>
-<li><a href="http://www2.svjatoslav.eu/gitweb/?p=sixth-data.git;a=snapshot;h=HEAD;sf=tgz">download latest snapshot</a>
-</li>
-
-<li>This program is free software; you can redistribute it and/or modify
-it under the terms of version 3 of the <a href="https://www.gnu.org/licenses/lgpl.html">GNU Lesser General Public
-License</a> or later as published by the Free Software Foundation.
+<div id="outline-container-sec-1" class="outline-2">
+<h2 id="sec-1"><span class="section-number-2">1</span> General</h2>
+<div class="outline-text-2" id="text-1">
+<ul class="org-ul">
+<li>This program is free software: released under Creative Commons Zero
+(CC0) license
</li>
<li>Program author:
<ul class="org-ul">
<li>Svjatoslav Agejenko
</li>
-<li>Homepage: <a href="http://svjatoslav.eu">http://svjatoslav.eu</a>
+<li>Homepage: <a href="https://svjatoslav.eu">https://svjatoslav.eu</a>
</li>
<li>Email: <a href="mailto://svjatoslav@svjatoslav.eu">mailto://svjatoslav@svjatoslav.eu</a>
</li>
</ul>
</li>
-<li><a href="http://svjatoslav.eu/programs.jsp">other applications hosted at svjatoslav.eu</a>
+<li><a href="https://www.svjatoslav.eu/projects/">Other software projects hosted at svjatoslav.eu</a>
</li>
</ul>
+</div>
-
-<div id="outline-container-sec-1" class="outline-2">
-<h2 id="sec-1"><span class="section-number-2">1</span> Vision / goal</h2>
-<div class="outline-text-2" id="text-1">
-<p>
-Provide versioned, clustered, flexible, object-relational database
-functionality for the <a href="http://www2.svjatoslav.eu/gitbrowse/sixth/doc/index.html">Sixth computation engine</a>.
-</p>
-
+<div id="outline-container-sec-1-1" class="outline-3">
+<h3 id="sec-1-1"><span class="section-number-3">1.1</span> Source code</h3>
+<div class="outline-text-3" id="text-1-1">
<ul class="org-ul">
-<li>I hate object-relational impedance mismatch.
+<li><a href="https://www2.svjatoslav.eu/gitweb/?p=sixth-data.git;a=snapshot;h=HEAD;sf=tgz">Download latest snapshot in TAR GZ format</a>
</li>
-<li>I don't like to convert data between persistent database and runtime
-objects for every transaction. How about creating united
-database/computation engine instead to:
-<ul class="org-ul">
-<li>Eliminate constant moving and converting of data between 2 systems.
+<li><a href="https://www2.svjatoslav.eu/gitweb/?p=sixth-data.git;a=summary">Browse Git repository online</a>
</li>
-<li>Abstract away difference between RAM VS persistent storage. Let
-the system decide at runtime which data to keep in what kind of
-memory.
+
+<li>Clone Git repository using command:
+<pre class="example">
+git clone https://www2.svjatoslav.eu/git/sixth-data.git
+
+</pre>
</li>
-</ul>
+
+<li>See <a href="https://www3.svjatoslav.eu/projects/sixth-data/apidocs/">JavaDoc</a>.
</li>
</ul>
</div>
+</div>
+</div>
-<div id="outline-container-sec-1-1" class="outline-3">
-<h3 id="sec-1-1"><span class="section-number-3">1.1</span> Inspiration</h3>
-<div class="outline-text-3" id="text-1-1">
-<ul class="org-ul">
-<li>Relational databases:
+<div id="outline-container-sec-2" class="outline-2">
+<h2 id="sec-2"><a id="ID-f6764282-a6f6-44e6-8716-b428074dd093" name="ID-f6764282-a6f6-44e6-8716-b428074dd093"></a><span class="section-number-2">2</span> Vision / goal</h2>
+<div class="outline-text-2" id="text-2">
+<p>
+Provide hackable, versioned, optimized, distributed, geometrical,
+arbitrary dimensional (<a href="#sec-4">hypercube based</a>) data storage and computation
+engine (<a href="#sec-3-1">as inspired by the brain</a>) for general purpose visual computing
+environment called <a href="http://www2.svjatoslav.eu/gitbrowse/sixth/doc/index.html">Sixth</a>.
+</p>
+
+<p>
+Because <a href="http://www.paulgraham.com/rootsoflisp.html">Lisp is hackable self defined programmable programming
+language</a> it would be used to provide <a href="https://en.wikipedia.org/wiki/Imperative_programming">imperative programming</a> support.
+</p>
+</div>
+</div>
+<div id="outline-container-sec-3" class="outline-2">
+<h2 id="sec-3"><a id="ID-0fa6354b-18c9-4120-bbf5-c7239aebecab" name="ID-0fa6354b-18c9-4120-bbf5-c7239aebecab"></a><span class="section-number-2">3</span> Inspiration</h2>
+<div class="outline-text-2" id="text-3">
<ul class="org-ul">
-<li>Transactional.
-</li>
-<li>Indexable / Quickly searchable.
+<li>see also: <a href="https://en.wikipedia.org/wiki/OLAP_cube">OLAP cube</a>.
</li>
</ul>
-</li>
-
-<li>Git (version control system)
+</div>
+<div id="outline-container-sec-3-1" class="outline-3">
+<h3 id="sec-3-1"><a id="ID-d2375acc-af14-4f18-8ad0-7949501178c5" name="ID-d2375acc-af14-4f18-8ad0-7949501178c5"></a><span class="section-number-3">3.1</span> Brain</h3>
+<div class="outline-text-3" id="text-3-1">
<ul class="org-ul">
-<li>Versionable
+<li>Brain appears to be natural geometrical/parallel data storage and
+computational engine:
+<ul class="org-ul">
+<li><a href="https://www.quantamagazine.org/the-brain-maps-out-ideas-and-memories-like-spaces-20190114/">https://www.quantamagazine.org/the-brain-maps-out-ideas-and-memories-like-spaces-20190114/</a>
</li>
-<li>Branchable / mergeable.
+</ul>
</li>
-<li>Transparent cansistency, checksumming and deduplication.
+
+<li>Even more awesome is that brain appears to operate and is wired as
+arbitrary/variable dimensional structure:
+<a href="https://singularityhub.com/2017/06/21/is-there-a-multidimensional-mathematical-world-hidden-in-the-brains-computation/">https://singularityhub.com/2017/06/21/is-there-a-multidimensional-mathematical-world-hidden-in-the-brains-computation/</a>
</li>
-<li>(Git as a database:
-<a href="https://www.kenneth-truyers.net/2016/10/13/git-nosql-database/">https://www.kenneth-truyers.net/2016/10/13/git-nosql-database/</a> )
+
+<li>On top of this, this multidimensional space that brain represents
+has dynamic/variable resolution/density:
+<ul class="org-ul">
+<li><a href="https://www.quantamagazine.org/goals-and-rewards-redraw-the-brains-map-of-the-world-20190328">https://www.quantamagazine.org/goals-and-rewards-redraw-the-brains-map-of-the-world-20190328</a>
</li>
</ul>
</li>
+
+<li>Such properties allow parallel <a href="#sec-5">Geometrical computation</a> and
+beautifully fits <a href="#sec-3-2">CM-1 Connection Machine</a> architecture (for extra
+hardware accelerated solution).
+</li>
</ul>
</div>
</div>
-<div id="outline-container-sec-1-2" class="outline-3">
-<h3 id="sec-1-2"><span class="section-number-3">1.2</span> Solution (the big idea)</h3>
-<div class="outline-text-3" id="text-1-2">
+<div id="outline-container-sec-3-2" class="outline-3">
+<h3 id="sec-3-2"><a id="ID-01aa65c1-3d44-44a8-9b90-58454bc6be80" name="ID-01aa65c1-3d44-44a8-9b90-58454bc6be80"></a><span class="section-number-3">3.2</span> CM-1 Connection Machine</h3>
+<div class="outline-text-3" id="text-3-2">
<p>
-I see 4D data structure.
+<a href="https://en.wikipedia.org/wiki/Connection_Machine">https://en.wikipedia.org/wiki/Connection_Machine</a>
</p>
-
-<figure>
-<p><img src="data model.png" class="img-responsive" alt="data model.png">
+<p>
+Massively parallel (thousands of CPUs) connected via
+machine's internal 12-dimensional hypercube network allows to
+efficiently simulate arbitrary dimensional hypercube and network
+topology between computational units. So that when we are
+solving/simulating for example 5 dimensional problem, we can arrange
+computational units into virtual 5D network. See:
+<a href="http://www.mission-base.com/tamiko/theory/cm_txts/di-ch2.html">http://www.mission-base.com/tamiko/theory/cm_txts/di-ch2.html</a>
</p>
-</figure>
<p>
-Dimensions:
+we can pre-distribute data across computation units and perform
+parallel <a href="#sec-5">geometrical computation</a>.
</p>
+</div>
+</div>
+</div>
+
+<div id="outline-container-sec-4" class="outline-2">
+<h2 id="sec-4"><a id="ID-96116550-a6a1-4700-bef7-865d0deee7ea" name="ID-96116550-a6a1-4700-bef7-865d0deee7ea"></a><span class="section-number-2">4</span> Reasons for hypercube as a so called first class citizen</h2>
+<div class="outline-text-2" id="text-4">
<ul class="org-ul">
-<li>List of all the objecs in the system (rows).
+<li>Hypercube is quite general purpose data structure that naturally
+encapsulates wide variety data and problems.
</li>
-<li>List of all declared unique object fields (columns).
+
+<li>Nicely captures apparent <a href="#sec-3-1">properties of the brain</a>.
</li>
-<li>List of all historical transactions/commits/versions (think of
-sheets of paper).
+
+<li>Naturally supports distributed and parallel <a href="#sec-5">geometrical data storage
+and computation.</a>
</li>
-<li>List of all concurrently running branches/threads. Branches can
-appear and merge over time as needed.
+
+<li>Dedicated hardware like <a href="#sec-3-2">CM-1</a> can be built around hypercube concept
+that results in data, computation process and hardware, all
+beautifully fitting together while complementing each other
+strengths.
</li>
-<li>(Every cell is concrete field value within an object)
+
+<li>Hypercube stored data (and computation process) has geometry by its
+nature and should fit nicely with "3D first" user interface ideology
+of the parent <a href="http://www2.svjatoslav.eu/gitbrowse/sixth/doc/index.html">Sixth</a> project.
</li>
</ul>
+</div>
+</div>
+<div id="outline-container-sec-5" class="outline-2">
+<h2 id="sec-5"><a id="ID-171fe375-c737-41e6-b429-a414f6abc5d8" name="ID-171fe375-c737-41e6-b429-a414f6abc5d8"></a><span class="section-number-2">5</span> Geometrical computation idea</h2>
+<div class="outline-text-2" id="text-5">
+</div>
+<div id="outline-container-sec-5-1" class="outline-3">
+<h3 id="sec-5-1"><a id="ID-5d287158-53ea-44a2-a754-dd862366066a" name="ID-5d287158-53ea-44a2-a754-dd862366066a"></a><span class="section-number-3">5.1</span> Distributed computation and data storage</h3>
+<div class="outline-text-3" id="text-5-1">
+<p>
+Lots of problems can be translated to geometry (use any shapes and as
+many dimensions as you need). Solution(s) to such problems could be
+then found via geometrical search/comparison/lookup results. As a
+bonus, such geometrical *data storage* AND *computation* can be
+naturally made in *parallel* and *distributed*.
+</p>
+
+<p>
+Learning means building/updating/re-balancing the model (the hard
+part). Question answering is making (relatively simple) lookups
+(geometrical queries) against the model.
+</p>
+</div>
+</div>
+<div id="outline-container-sec-5-2" class="outline-3">
+<h3 id="sec-5-2"><a id="ID-a117c11e-97c1-4822-88b2-9fc10f96caec" name="ID-a117c11e-97c1-4822-88b2-9fc10f96caec"></a><span class="section-number-3">5.2</span> Mapping hypercube to object-oriented model and relational database</h3>
+<div class="outline-text-3" id="text-5-2">
+<p>
+Object oriented programming is inspired by the way human mind
+operates. It allows programmer to express ideas to computer in a more
+human-like terms.
+</p>
<p>
-Partitioning/clustering:
+It is actually also possible to map object model and relational
+database to geometrical hyperspace:
</p>
+
<ul class="org-ul">
-<li>Why not to partition/(load balance) as required across networked
-physical computers along arbitrary dimension(s) declared above ?
+<li>Object or database table row is a point in hypercube arbitrary
+dimensional space. Each object member variable or database table
+column can be mapped to its own dimension in hypercube. That is: if
+class declares 4 variables for an object, then corresponding object
+can be stored as a single point inside 4 dimensional
+hypercube. Variable values translate to point coordinates in that
+hypercube. That is: numbers and string can be translated to linear
+value that can be used as a coordinate along particular dimension.
+</li>
+
+<li>Each object class or database table declares its own hypercube that
+contain instances (objects) of that class or rows of a table.
</li>
</ul>
+</div>
+</div>
+<div id="outline-container-sec-5-3" class="outline-3">
+<h3 id="sec-5-3"><a id="ID-b6b15bd2-c78b-4c51-a343-72843a515c29" name="ID-b6b15bd2-c78b-4c51-a343-72843a515c29"></a><span class="section-number-3">5.3</span> Mapping entity relations in hypercube</h3>
+<div class="outline-text-3" id="text-5-3">
<p>
-Indexing (for fast searching):
+Consider we want to create database of:
</p>
<ul class="org-ul">
-<li>Why not to index along arbitrary dimensions (as required) ?
+<li>Books.
+</li>
+<li>Authors.
+</li>
+<li>Effort: Amount of time contributed by every author to every book
+that he/she wrote.
</li>
</ul>
<p>
-Further optimizations:
+Information above can be represented as 3D cube where dimensions are:
</p>
<ul class="org-ul">
-<li>In current early stage, trying to focus on minimum possible set of
-features that would provide maximum possible set of power/benefit :)
+<li>X: Book
+</li>
+<li>Y: Author
</li>
-<li>Once featres are locked. Anything can be optimised. Optimization for
-size (deduplication) can be solved using Git style content
-addressible storage mechanism.
+<li>Z: Effort
</li>
</ul>
+
+<p>
+Points in that cube would nicely capture many to many relations
+between authors and the books.
+</p>
</div>
</div>
</div>
-<div id="outline-container-sec-2" class="outline-2">
-<h2 id="sec-2"><span class="section-number-2">2</span> Current status</h2>
-<div class="outline-text-2" id="text-2">
+<div id="outline-container-sec-6" class="outline-2">
+<h2 id="sec-6"><span class="section-number-2">6</span> Current status</h2>
+<div class="outline-text-2" id="text-6">
<ul class="org-ul">
+<li>More or less defined <a href="#sec-2">Vision / goal</a>.
+</li>
+
+<li>Collected some <a href="#sec-3">inspiring</a> <a href="#sec-5">ideas</a>.
+</li>
+
<li>Implemented very simple persistent key-value map.
+<ul class="org-ul">
+<li>Long term goal is to use it as a backing storage engine and
+implement more advanced features on top of this via layered
+architecture.
</li>
</ul>
+</li>
+</ul>
+</div>
+</div>
+<div id="outline-container-sec-7" class="outline-2">
+<h2 id="sec-7"><span class="section-number-2">7</span> See also</h2>
+<div class="outline-text-2" id="text-7">
<p>
-Long term goal is to implement more advanced features on top of this.
+Interesting or competing projects with good ideas:
</p>
-</div>
-</div>
-<div id="outline-container-sec-3" class="outline-2">
-<h2 id="sec-3"><span class="section-number-2">3</span> <span class="label label-primary TODO">TODO</span> </h2>
-<div class="outline-text-2" id="text-3">
-</div><div id="outline-container-sec-3-1" class="outline-3">
-<h3 id="sec-3-1"><span class="section-number-3">3.1</span> check out Magma</h3>
-<div class="outline-text-3" id="text-3-1">
<ul class="org-ul">
-<li><a href="http://wiki.squeak.org/squeak/2665">http://wiki.squeak.org/squeak/2665</a>
+<li><a href="https://github.com/vygr/ChrysaLisp">ChrysaLisp</a>
+<ul class="org-ul">
+<li>Assembler/C-Script/Lisp 64 bit, MIMD, multi CPU, multi threaded,
+multi core, multi user Parallel OS. With GUI, Terminal, OO
+Assembler, Class libraries, C-Script compiler, Lisp interpreter,
+Debugger, and more…
+</li>
+</ul>
+</li>
+
+<li><a href="#sec-3-2">CM-1 Connection Machine</a>
+</li>
+
+<li><a href="http://esug.org/data/ESUG2015/3%20wednesday/1100-1130%20SQL%20Queries%20on%20Smalltalk%20Objects/SQL%20Queries%20in%20Smalltalk%20(James%20Foster).pdf">Gemstone/S</a>
+<ul class="org-ul">
+<li>Completely distributed smalltalk based computing
+system.
+</li>
+</ul>
+</li>
+
+<li><a href="https://gtoolkit.com/">Glamorous Toolkit</a>
+<ul class="org-ul">
+<li>Moldable development environment. It is a live notebook. It is a
+flexible search interface. It is a fancy code editor. It is a
+software analysis platform. It is a data visualization engine. All
+in one.
+</li>
+</ul>
+</li>
+
+<li>GRAKN.AI
+<ul class="org-ul">
+<li>database in the form of a knowledge graph that uses machine
+reasoning to simplify data processing challenges for AI
+applications. <a href="https://grakn.ai/">https://grakn.ai/</a>
+</li>
+</ul>
+</li>
+
+<li><a href="http://phantomos.org/">http://phantomos.org/</a>
+<ul class="org-ul">
+<li>Programs run forever. System crash or reboot does not destroy
+state of running program.
+</li>
+</ul>
+</li>
+
+<li><a href="http://wiki.squeak.org/squeak/2665">Magma</a>
+<ul class="org-ul">
+<li>Multi-user object database for Squeak
+</li>
+</ul>
+</li>
+
+<li>Taichi: A Language for High-Performance Computation onSpatially
+Sparse Data Structures
+<ul class="org-ul">
+<li><a href="http://taichi.graphics/wp-content/uploads/2019/09/taichi_lang.pdf">http://taichi.graphics/wp-content/uploads/2019/09/taichi_lang.pdf</a>
+</li>
+</ul>
+</li>
+
+<li><a href="http://www.uruk.org/emu/Taos.html">TAOS</a>
+<ul class="org-ul">
+<li>Completely distributed operating system/virtual machine:
+</li>
+</ul>
+</li>
+
+<li><a href="https://tiledb.com/">TileDB</a>
+<ul class="org-ul">
+<li>Analyze and share complex multi-dimensional data at scale
+</li>
+</ul>
</li>
</ul>
-</div>
</div>
</div>
</div><div class="col-md-3"><nav id="table-of-contents">
<div id="text-table-of-contents" class="bs-docs-sidebar">
<ul class="nav">
-<li><a href="#sec-1">1. Vision / goal</a>
+<li><a href="#sec-1">1. General</a>
+<ul class="nav">
+<li><a href="#sec-1-1">1.1. Source code</a></li>
+</ul>
+</li>
+<li><a href="#sec-2">2. Vision / goal</a></li>
+<li><a href="#sec-3">3. Inspiration</a>
<ul class="nav">
-<li><a href="#sec-1-1">1.1. Inspiration</a></li>
-<li><a href="#sec-1-2">1.2. Solution (the big idea)</a></li>
+<li><a href="#sec-3-1">3.1. Brain</a></li>
+<li><a href="#sec-3-2">3.2. CM-1 Connection Machine</a></li>
</ul>
</li>
-<li><a href="#sec-2">2. Current status</a></li>
-<li><a href="#sec-3">3. </a>
+<li><a href="#sec-4">4. Reasons for hypercube as a so called first class citizen</a></li>
+<li><a href="#sec-5">5. Geometrical computation idea</a>
<ul class="nav">
-<li><a href="#sec-3-1">3.1. check out Magma</a></li>
+<li><a href="#sec-5-1">5.1. Distributed computation and data storage</a></li>
+<li><a href="#sec-5-2">5.2. Mapping hypercube to object-oriented model and relational database</a></li>
+<li><a href="#sec-5-3">5.3. Mapping entity relations in hypercube</a></li>
</ul>
</li>
+<li><a href="#sec-6">6. Current status</a></li>
+<li><a href="#sec-7">7. See also</a></li>
</ul>
</div>
</nav>
</div></div></div>
<footer id="postamble" class="">
<div><p class="author">Author: Svjatoslav Agejenko</p>
-<p class="date">Created: 2017-06-13 Tue 22:14</p>
-<p class="creator"><a href="http://www.gnu.org/software/emacs/">Emacs</a> 25.1.1 (<a href="http://orgmode.org">Org-mode</a> 8.2.10)</p>
+<p class="date">Created: 2020-06-20 Sat 03:28</p>
+<p class="creator"><a href="http://www.gnu.org/software/emacs/">Emacs</a> 26.1 (<a href="http://orgmode.org">Org-mode</a> 9.1.9)</p>
</div>
</footer>
</body>
projects / sixth-data.git / blobdiff