From: Svjatoslav Agejenko Date: Sun, 2 Jul 2017 20:52:00 +0000 (+0200) Subject: Increased amount of dimensions from 4D to arbitrary. X-Git-Url: http://www2.svjatoslav.eu/gitweb/?p=sixth-data.git;a=commitdiff_plain;h=c2e5bc51afcb4776720ff274596bd68595f793d7 Increased amount of dimensions from 4D to arbitrary. --- diff --git a/doc/data model.png b/doc/data model.png deleted file mode 100644 index da3274d..0000000 Binary files a/doc/data model.png and /dev/null differ diff --git a/doc/index.html b/doc/index.html index 28f924f..0c9559e 100644 --- a/doc/index.html +++ b/doc/index.html @@ -2,7 +2,7 @@ Sixth - system for data storage, computation, exploration and interaction - + @@ -226,19 +226,22 @@ License or later as published by the Free Software Foundation.
-

1 Vision / goal

+

1 Vision / goal

-Provide versioned, clustered, flexible, object-relational database -functionality for the Sixth computation engine. +Provide versioned, clustered, flexible, distributed, multi-dimensional +data storage engine for the Sixth computation engine.

+
    +
  • Speaking of traditional relational database and object oriented +business applications:
    • I hate object-relational impedance mismatch.
      • -
    • I don't like to convert data between persistent database and runtime -objects for every transaction. How about creating united +
    • I don't like to convert data between persistent database and +runtime objects for every transaction. How about creating united database/computation engine instead to:
      • Eliminate constant moving and converting of data between 2 systems. @@ -250,6 +253,8 @@ memory.
    +
    • +
@@ -278,98 +283,61 @@ memory. - -
-
- -
-

1.2 Solution (the big idea)

-
-

-I see 4D data structure. -

- - -
-

data model.png -

-
-

-Dimensions: -

+
  • Brain (appears to have more than 3D dimensional design. Food for +thought…)
      -
    • List of all the objecs in the system (rows). -
      • -
    • List of all declared unique object fields (columns). -
      • -
    • List of all historical transactions/commits/versions (think of -sheets of paper). -
      • -
    • List of all concurrently running branches/threads. Branches can -appear and merge over time as needed. -
      • -
    • (Every cell is concrete field value within an object) -
      • -
    - -

    -Partitioning/clustering: -

    +
  • https://singularityhub.com/2017/06/21/is-there-a-multidimensional-mathematical-world-hidden-in-the-brains-computation/
      -
    • Why not to partition/(load balance) as required across networked -physical computers along arbitrary dimension(s) declared above ? +
    • From there comes following idea: Maybe every problem can be +translated to geometry (use any shapes and as many dimensions as +you need). Solution(s) to such problems would then appear as +relatively simple search/comparison/lookup results. As a bonus, +such geometrical *data storage* AND *computation* can be +naturally made in *parallel* and *distributed*. That's what +neurons in the brain appear to be doing ! :) . Learning means +building/updating the model (the hard part). Question answering +is making (relatively simple) lookups (geometrical queries) +against the model.
    - -

    -Indexing (for fast searching): -

    -
      -
    • Why not to index along arbitrary dimensions (as required) ?
    - -

    -Further optimizations: -

    -
      -
    • In current early stage, trying to focus on minimum possible set of -features that would provide maximum possible set of power/benefit :) -
      • -
    • Once featres are locked. Anything can be optimised. Optimization for -size (deduplication) can be solved using Git style content -addressible storage mechanism.
    • +

    2 Current status

      +
    • More or less defined Vision / goal. +
      • +
    • Implemented very simple persistent key-value map. +
        +
      • Long term goal is to use it as a backing storage engine and +implement more advanced features on top of this. +
        • +
    - -

    -Long term goal is to implement more advanced features on top of this. -

    3 TODO

    -
    -

    3.1 check out Magma

    -
    +
    + +
    diff --git a/doc/index.org b/doc/index.org index 42a9704..d58e2e3 100644 --- a/doc/index.org +++ b/doc/index.org @@ -28,18 +28,23 @@ #+HTML_HEAD: * Vision / goal -Provide versioned, clustered, flexible, object-relational database -functionality for the [[http://www2.svjatoslav.eu/gitbrowse/sixth/doc/index.html][Sixth computation engine]]. - -+ I hate object-relational impedance mismatch. - -+ I don't like to convert data between persistent database and runtime - objects for every transaction. How about creating united - database/computation engine instead to: - + Eliminate constant moving and converting of data between 2 systems. - + Abstract away difference between RAM VS persistent storage. Let - the system decide at runtime which data to keep in what kind of - memory. + :PROPERTIES: + :ID: f6764282-a6f6-44e6-8716-b428074dd093 + :END: +Provide versioned, clustered, flexible, distributed, multi-dimensional +data storage engine for the [[http://www2.svjatoslav.eu/gitbrowse/sixth/doc/index.html][Sixth computation engine]]. + ++ Speaking of traditional relational database and object oriented + business applications: + + I hate object-relational impedance mismatch. + + + I don't like to convert data between persistent database and + runtime objects for every transaction. How about creating united + database/computation engine instead to: + + Eliminate constant moving and converting of data between 2 systems. + + Abstract away difference between RAM VS persistent storage. Let + the system decide at runtime which data to keep in what kind of + memory. ** Inspiration + Relational databases: @@ -53,38 +58,27 @@ functionality for the [[http://www2.svjatoslav.eu/gitbrowse/sixth/doc/index.html + (Git as a database: https://www.kenneth-truyers.net/2016/10/13/git-nosql-database/ ) -** Solution (the big idea) -I see 4D data structure. - -[[file:data model.png]] - -Dimensions: -+ List of all the objecs in the system (rows). -+ List of all declared unique object fields (columns). -+ List of all historical transactions/commits/versions (think of - sheets of paper). -+ List of all concurrently running branches/threads. Branches can - appear and merge over time as needed. -+ (Every cell is concrete field value within an object) - -Partitioning/clustering: -+ Why not to partition/(load balance) as required across networked - physical computers along arbitrary dimension(s) declared above ? - -Indexing (for fast searching): -+ Why not to index along arbitrary dimensions (as required) ? - -Further optimizations: -+ In current early stage, trying to focus on minimum possible set of - features that would provide maximum possible set of power/benefit :) -+ Once featres are locked. Anything can be optimised. Optimization for - size (deduplication) can be solved using Git style content - addressible storage mechanism. ++ Brain (appears to have more than 3D dimensional design. Food for + thought...) + + https://singularityhub.com/2017/06/21/is-there-a-multidimensional-mathematical-world-hidden-in-the-brains-computation/ + + From there comes following idea: Maybe every problem can be + translated to geometry (use any shapes and as many dimensions as + you need). Solution(s) to such problems would then appear as + relatively simple search/comparison/lookup results. As a bonus, + such geometrical *data storage* AND *computation* can be + naturally made in *parallel* and *distributed*. That's what + neurons in the brain appear to be doing ! :) . Learning means + building/updating the model (the hard part). Question answering + is making (relatively simple) lookups (geometrical queries) + against the model. + * Current status -- Implemented very simple persistent key-value map. +- More or less defined [[id:f6764282-a6f6-44e6-8716-b428074dd093][Vision / goal]]. -Long term goal is to implement more advanced features on top of this. +- Implemented very simple persistent key-value map. + - Long term goal is to use it as a backing storage engine and + implement more advanced features on top of this. * TODO -** check out Magma ++ check out Magma + http://wiki.squeak.org/squeak/2665