-#+TITLE: Sixth - system for data storage, computation, exploration and interaction
+#+TITLE: Sixth Data - Data storage and computing engine
------
-- This is a subproject of [[http://www2.svjatoslav.eu/gitbrowse/sixth/doc/index.html][Sixth]]
-
-- [[http://www2.svjatoslav.eu/gitweb/?p=sixth-data.git;a=snapshot;h=HEAD;sf=tgz][download latest snapshot]]
+* (document settings) :noexport:
+** use dark style for TWBS-HTML exporter
+#+HTML_HEAD: <link href="https://bootswatch.com/3/darkly/bootstrap.min.css" rel="stylesheet">
+#+HTML_HEAD: <script src="https://cdnjs.cloudflare.com/ajax/libs/jquery/1.11.3/jquery.min.js"></script>
+#+HTML_HEAD: <script src="https://cdnjs.cloudflare.com/ajax/libs/twitter-bootstrap/3.3.5/js/bootstrap.min.js"></script>
+#+HTML_HEAD: <style type="text/css">
+#+HTML_HEAD: footer {background-color: #111 !important;}
+#+HTML_HEAD: pre {background-color: #111; color: #ccc;}
+#+HTML_HEAD: </style>
-- This program is free software; you can redistribute it and/or modify
- it under the terms of version 3 of the [[https://www.gnu.org/licenses/lgpl.html][GNU Lesser General Public
- License]] or later as published by the Free Software Foundation.
+* General
+- This program is free software: released under Creative Commons Zero
+ (CC0) license
- Program author:
- Svjatoslav Agejenko
- - Homepage: http://svjatoslav.eu
+ - Homepage: https://svjatoslav.eu
- Email: mailto://svjatoslav@svjatoslav.eu
-- [[http://www.svjatoslav.eu/programs.jsp][other applications hosted at svjatoslav.eu]]
+- [[https://www.svjatoslav.eu/projects/][Other software projects hosted at svjatoslav.eu]]
+** Source code
+- [[https://www2.svjatoslav.eu/gitweb/?p=sixth-data.git;a=snapshot;h=HEAD;sf=tgz][Download latest snapshot in TAR GZ format]]
-* (document settings) :noexport:
-** use dark style for TWBS-HTML exporter
-#+HTML_HEAD: <link href="https://bootswatch.com/4/darkly/bootstrap.min.css" rel="stylesheet">
-#+HTML_HEAD: <script src="https://cdnjs.cloudflare.com/ajax/libs/jquery/1.11.2/jquery.min.js"></script>
-#+HTML_HEAD: <script src="https://cdnjs.cloudflare.com/ajax/libs/twitter-bootstrap/3.3.1/js/bootstrap.min.js"></script>"
-#+HTML_HEAD: <style type="text/css">
-#+HTML_HEAD: footer {background-color: #111 !important;}
-#+HTML_HEAD: pre {background-color: #111; color: #ccc;}
-#+HTML_HEAD: </style>
+- [[https://www2.svjatoslav.eu/gitweb/?p=sixth-data.git;a=summary][Browse Git repository online]]
+
+- Clone Git repository using command:
+ : git clone https://www2.svjatoslav.eu/git/sixth-data.git
+
+- See [[https://www3.svjatoslav.eu/projects/sixth-data/apidocs/][JavaDoc]].
* Vision / goal
:PROPERTIES:
:ID: f6764282-a6f6-44e6-8716-b428074dd093
:END:
-Provide versioned, clustered, flexible, distributed, multi-dimensional
-data storage engine for the [[http://www2.svjatoslav.eu/gitbrowse/sixth/doc/index.html][Sixth computation engine]].
-
-+ Speaking of traditional relational database and object oriented
- business applications:
-
- + I hate object-relational impedance mismatch.
-
- + I don't like to convert data between persistent database and
- runtime objects for every transaction. How about creating united
- database/computation engine instead to:
-
- + Eliminate constant moving and converting of data between 2 systems
- and make computing happen close to where the data is stored.
-
- + Abstract away difference between RAM VS persistent storage. Let
- the system decide at runtime which data to keep in what kind of
- memory.
+Provide hackable, versioned, optimized, distributed, geometrical,
+arbitrary dimensional ([[id:96116550-a6a1-4700-bef7-865d0deee7ea][hypercube based]]) data storage and computation
+engine ([[id:d2375acc-af14-4f18-8ad0-7949501178c5][as inspired by the brain]]) for general purpose visual computing
+environment called [[http://www2.svjatoslav.eu/gitbrowse/sixth/doc/index.html][Sixth]].
+Because [[http://www.paulgraham.com/rootsoflisp.html][Lisp is hackable self defined programmable programming
+language]] it would be used to provide [[https://en.wikipedia.org/wiki/Imperative_programming][imperative programming]] support.
* Inspiration
-+ Relational databases:
- + Transactional.
- + Indexable / Quickly searchable.
-
-+ Git (version control system)
- + Versionable
- + Branchable / mergeable.
- + Transparent cansistency, checksumming and deduplication.
- + (Git as a database:
- https://www.kenneth-truyers.net/2016/10/13/git-nosql-database/ )
-
+:PROPERTIES:
+:ID: 0fa6354b-18c9-4120-bbf5-c7239aebecab
+:END:
++ see also: [[https://en.wikipedia.org/wiki/OLAP_cube][OLAP cube]].
** Brain
:PROPERTIES:
:ID: d2375acc-af14-4f18-8ad0-7949501178c5
:END:
-+ Appears to have more than 3D dimensional design. Food for
- thought...)
- + https://singularityhub.com/2017/06/21/is-there-a-multidimensional-mathematical-world-hidden-in-the-brains-computation/
-
-+ It directly inspires following ideas
- + [[id:5d287158-53ea-44a2-a754-dd862366066a][Distributed comutation and data storage]]
- + [[id:a117c11e-97c1-4822-88b2-9fc10f96caec][Mapping of hyperspace to traditional object-oriented model]]
- + [[id:b6b15bd2-c78b-4c51-a343-72843a515c29][Handling of relations]]
-
-* Ideas
++ Brain appears to be natural geometrical/parallel data storage and
+ computational engine:
+ + https://www.quantamagazine.org/the-brain-maps-out-ideas-and-memories-like-spaces-20190114/
+
++ Even more awesome is that brain appears to operate and is wired as
+ arbitrary/variable dimensional structure:
+ https://singularityhub.com/2017/06/21/is-there-a-multidimensional-mathematical-world-hidden-in-the-brains-computation/
+
++ On top of this, this multidimensional space that brain represents
+ has dynamic/variable resolution/density:
+ + https://www.quantamagazine.org/goals-and-rewards-redraw-the-brains-map-of-the-world-20190328
+
++ Such properties allow parallel [[id:171fe375-c737-41e6-b429-a414f6abc5d8][Geometrical computation]] and
+ beautifully fits [[id:01aa65c1-3d44-44a8-9b90-58454bc6be80][CM-1 Connection Machine]] architecture (for extra
+ hardware accelerated solution).
+
+** CM-1 Connection Machine
+:PROPERTIES:
+:ID: 01aa65c1-3d44-44a8-9b90-58454bc6be80
+:END:
+https://en.wikipedia.org/wiki/Connection_Machine
+
+Massively parallel (thousands of CPUs) connected via
+machine's internal 12-dimensional hypercube network allows to
+efficiently simulate arbitrary dimensional hypercube and network
+topology between computational units. So that when we are
+solving/simulating for example 5 dimensional problem, we can arrange
+computational units into virtual 5D network. See:
+http://www.mission-base.com/tamiko/theory/cm_txts/di-ch2.html
+
+we can pre-distribute data across computation units and perform
+parallel [[id:171fe375-c737-41e6-b429-a414f6abc5d8][geometrical computation]].
+
+* Reasons for hypercube as a so called first class citizen
+:PROPERTIES:
+:ID: 96116550-a6a1-4700-bef7-865d0deee7ea
+:END:
++ Hypercube is quite general purpose data structure that naturally
+ encapsulates wide variety data and problems.
+
++ Nicely captures apparent [[id:d2375acc-af14-4f18-8ad0-7949501178c5][properties of the brain]].
+
++ Naturally supports distributed and parallel [[id:171fe375-c737-41e6-b429-a414f6abc5d8][geometrical data storage
+ and computation.]]
+
++ Dedicated hardware like [[id:01aa65c1-3d44-44a8-9b90-58454bc6be80][CM-1]] can be built around hypercube concept
+ that results in data, computation process and hardware, all
+ beautifully fitting together while complementing each other
+ strengths.
+
++ Hypercube stored data (and computation process) has geometry by its
+ nature and should fit nicely with "3D first" user interface ideology
+ of the parent [[http://www2.svjatoslav.eu/gitbrowse/sixth/doc/index.html][Sixth]] project.
+* Geometrical computation idea
+:PROPERTIES:
+:ID: 171fe375-c737-41e6-b429-a414f6abc5d8
+:END:
** Distributed computation and data storage
:PROPERTIES:
:ID: 5d287158-53ea-44a2-a754-dd862366066a
:END:
-Maybe every problem can be translated to geometry (use any shapes and
-as many dimensions as you need). Solution(s) to such problems would
-then appear as relatively simple search/comparison/lookup results. As
-a bonus, such geometrical *data storage* AND *computation* can be
-naturally made in *parallel* and *distributed*. That's what neurons in
-the brain appear to be doing ! :) . Learning means building/updating
-the model (the hard part). Question answering is making (relatively
-simple) lookups (geometrical queries) against the model.
-** Mapping of hyperspace to traditional object-oriented model
+Lots of problems can be translated to geometry (use any shapes and as
+many dimensions as you need). Solution(s) to such problems could be
+then found via geometrical search/comparison/lookup results. As a
+bonus, such geometrical *data storage* AND *computation* can be
+naturally made in *parallel* and *distributed*.
+
+Learning means building/updating/re-balancing the model (the hard
+part). Question answering is making (relatively simple) lookups
+(geometrical queries) against the model.
+** Mapping hypercube to object-oriented model and relational database
:PROPERTIES:
:ID: a117c11e-97c1-4822-88b2-9fc10f96caec
:END:
operates. It allows programmer to express ideas to computer in a more
human-like terms.
-It is possible to map object model to geometrical hyperspace:
-
-+ Object is a point in space (universe). Each object member variable
- translates to its own dimension. That is: if class declares 4
- variables for an object, then corresponding object can be stored as
- a single point inside 4 dimensional space. Variable values translate
- to point coordinates in space. That is: Integer, floating point
- number and even boolean and string can be translated to linear value
- that can be used as a coordinate along particular dimension.
-
-+ Each class declares its own space (universe). All class instances
- (objects) are points inside that particular universe. References
- between objects of different types are hyperlinks (portals) between
- different universes.
-** Handling of relations
+It is actually also possible to map object model and relational
+database to geometrical hyperspace:
+
++ Object or database table row is a point in hypercube arbitrary
+ dimensional space. Each object member variable or database table
+ column can be mapped to its own dimension in hypercube. That is: if
+ class declares 4 variables for an object, then corresponding object
+ can be stored as a single point inside 4 dimensional
+ hypercube. Variable values translate to point coordinates in that
+ hypercube. That is: numbers and string can be translated to linear
+ value that can be used as a coordinate along particular dimension.
+
++ Each object class or database table declares its own hypercube that
+ contain instances (objects) of that class or rows of a table.
+
+** Mapping entity relations in hypercube
:PROPERTIES:
:ID: b6b15bd2-c78b-4c51-a343-72843a515c29
:END:
-Consider we want to create database of books and authors. Book can
-have multiple authors, and single person can be author for multiple
-books. It is possible to store how many hours of work each author has
-contributed to every book, using hyperspace as follows:
-
-+ Every dimension corresponds to one particular book author. (10
- authors in the database, would require 10 dimensional space)
- + Point in space corresponds to one particular book.
- + Point location along particular (author) dimension corresponds
- to amount of work contributed by particular author for given
- book.
-
-Alternatively:
-
-+ Every dimension corresponds to one particular book.
- + Point in space corresponds to one particular author in the entire
- database.
- + Point location along particular (book) dimension corresponds to
- amount of work contributed for book by given author (point).
-
-** Layered architecture
-+ layer 1 :: disk / block storage / partition
-
-+ layer 2 :: key/value storage. Keys are unique and are dictated by
- storage engine. Value is arbitrary but limited size byte
- array. This layer is responsible for handling disk
- defragmentation and consistency in case of crash
- recovery.
-
-+ layer 3 :: key/value storage. Keys are content hashes. Values are
- arbitrary but limited size content byte arrays. This
- layer effectively implements content addressable
- storage. Content addressible storage enables GIT-like
- behavior (possibility for competing branches, retaining
- history, transparent deduplication)
-
-+ layer 4 :: Implements arbitrary dimensional multiverse.
-
-+ layer 5 :: Distributed computation engine.
-
+Consider we want to create database of:
++ Books.
++ Authors.
++ Effort: Amount of time contributed by every author to every book
+ that he/she wrote.
+
+Information above can be represented as 3D cube where dimensions are:
++ X: Book
++ Y: Author
++ Z: Effort
+
+Points in that cube would nicely capture many to many relations
+between authors and the books.
* Current status
- More or less defined [[id:f6764282-a6f6-44e6-8716-b428074dd093][Vision / goal]].
-- Collected some [[id:d2375acc-af14-4f18-8ad0-7949501178c5][ideas]].
+- Collected some [[id:0fa6354b-18c9-4120-bbf5-c7239aebecab][inspiring]] [[id:171fe375-c737-41e6-b429-a414f6abc5d8][ideas]].
- Implemented very simple persistent key-value map.
- Long term goal is to use it as a backing storage engine and
- implement more advanced features on top of this.
+ implement more advanced features on top of this via layered
+ architecture.
* See also
Interesting or competing projects with good ideas:
-+ GRAKN.AI: database in the form of a knowledge graph that uses
- machine reasoning to simplify data processing challenges for AI
- applications.
- + https://grakn.ai/
++ [[https://github.com/vygr/ChrysaLisp][ChrysaLisp]]
+ + Assembler/C-Script/Lisp 64 bit, MIMD, multi CPU, multi threaded,
+ multi core, multi user Parallel OS. With GUI, Terminal, OO
+ Assembler, Class libraries, C-Script compiler, Lisp interpreter,
+ Debugger, and more...
+
++ [[id:01aa65c1-3d44-44a8-9b90-58454bc6be80][CM-1 Connection Machine]]
+
++ [[http://esug.org/data/ESUG2015/3%20wednesday/1100-1130%20SQL%20Queries%20on%20Smalltalk%20Objects/SQL%20Queries%20in%20Smalltalk%20(James%20Foster).pdf][Gemstone/S]]
+ + Completely distributed smalltalk based computing
+ system.
+
++ [[https://gtoolkit.com/][Glamorous Toolkit]]
+ + Moldable development environment. It is a live notebook. It is a
+ flexible search interface. It is a fancy code editor. It is a
+ software analysis platform. It is a data visualization engine. All
+ in one.
+
++ GRAKN.AI
+ + database in the form of a knowledge graph that uses machine
+ reasoning to simplify data processing challenges for AI
+ applications. https://grakn.ai/
+
++ http://phantomos.org/
+ + Programs run forever. System crash or reboot does not destroy
+ state of running program.
+
++ [[http://wiki.squeak.org/squeak/2665][Magma]]
+ + Multi-user object database for Squeak
-+ Gemstone/S based on Smalltalk.
- + http://esug.org/data/ESUG2015/3%20wednesday/1100-1130%20SQL%20Queries%20on%20Smalltalk%20Objects/SQL%20Queries%20in%20Smalltalk%20(James%20Foster).pdf
++ Taichi: A Language for High-Performance Computation onSpatially
+ Sparse Data Structures
+ + http://taichi.graphics/wp-content/uploads/2019/09/taichi_lang.pdf
-+ Magma distributed database in Smalltalk.
- + http://wiki.squeak.org/squeak/2665
++ [[http://www.uruk.org/emu/Taos.html][TAOS]]
+ + Completely distributed operating system/virtual machine:
-+ ZetaVM
- + https://github.com/zetavm/zetavm
++ [[https://tiledb.com/][TileDB]]
+ + Analyze and share complex multi-dimensional data at scale
projects / sixth-data.git / blobdiff