flesh/Readme.org

397 lines
18 KiB
Org Mode
Raw Normal View History

#+Title: Relish: Rusty Expressive LIsp SHell
#+Author: Ava Hahn
2023-03-07 14:29:31 -08:00
Note: this document is best read using a dedicated ORG mode editor
* Purpose statement
2023-03-07 14:29:31 -08:00
The purpose of Relish is to create a highly portable, easy to integrate language that can be used in many environments.
* Goals
- Iterate on the ideas and designs that were tested with SHS
2023-03-08 12:09:20 -08:00
https://gitlab.com/whom/shs
2023-03-07 14:29:31 -08:00
- Create a usable POSIX shell
- Create usable applications/scripts
- To have quality code coverage
- No unsafe code
** Stretch Goals
- Create an interpreter that can be booted on one or more SOCs
- Create an interpreter that can be embedded in another application
- Create UI bindings
* Contact
- Matrix chat: #vomitorium:matrix.sunnypup.io
2023-03-08 12:09:20 -08:00
https://matrix.to/#/#vomitorium:matrix.sunnypup.io
* How to use
2023-03-08 11:41:00 -08:00
** Syntax
*** S-Expressions
Relish fits within the LISP family of languages alongside venerable languages like Scheme or Common Lisp.
Lisps are *HOMOICONIC* which means that the code is data, and that there is a direct correlation between the code as written and the program as stored in memory.
This is achieved through *S-EXPRESSIONS*. An S-Expression (or symbolic expression) is a tree of nested lists.
Programs in Relish (and most other lisps) are written with S-Expressions, and are then represented in memory as trees of nested linked lists.
An example:
#+BEGIN_SRC lisp
(top-level element1 "element2" 3 (nested 2 5 2) (peer-nested))
#+END_SRC
As in memory
#+BEGIN_SRC
top-level -> element1 -> "element2" -> 3 -> [] -> [] ->
| ^-> peer-nested ->
\-> nested -> 2 -> 5 -> 2 ->
#+END_SRC
Each node in memory has type information and potentially a cooresponding entry in a global symbol table.
**** Data types
2023-03-08 11:41:00 -08:00
Relish leverages the following data types:
- Strings: delimited by ~'~, ~"~, or ~`~
- Integers: up to 128 bit signed integers
- Floats: all floats are stored as 64 bit floats
- Booleans: ~true~ or ~false~
- Symbols: an un-delimited chunk of text containing alphanumerics, ~-~, ~_~, or ~?~
Symbols and Functions can contain data of any type. there is no restriction on what can be set/passed to what.....
However, internally Relish is typed, and many builtin functions will get very picky about what types are passed to them.
2023-03-08 11:41:00 -08:00
**** Calling a function
2023-03-08 11:41:00 -08:00
S-Expressions can represent function calls in addition to trees of data. A function call is a list of data starting with a symbol that is defined to be a function:
#+BEGIN_SRC lisp
(dothing arg1 arg2 arg3)
#+END_SRC
Function calls are executed as soon as the tree is evaluated. See the following example:
#+BEGIN_SRC lisp
(add 3 (add 5 2))
#+END_SRC
In this example, ~(add 5 2)~ is evaluated first, its result is then passed to ~(add 3 ...)~. In infix form: ~3 + (5 + 2)~.
*** Control flow
**** If
An *if form* is the most basic form of conditional evaluation offered by Relish.
It is a function that takes lazily evaluated arguments: a condition, a then clause, and an else clause.
If the condition evaluates to true, the then clause is evaluated and the result returned.
Otherwise the else clause is evaluated and the result is returned.
If the condition evaluates to neither true nor false (a non-boolean value) a type error is returned.
#+BEGIN_SRC lisp
;; simple condition
(if true
(echo "its true!")
(echo "its false!"))
;; more advanced condition, with hypothetical data
(if (get-my-flag global-state)
(echo "my flag is already on!")
(turn-on-my-flag global-state))
#+END_SRC
**** While
Another popular control flow structure is the *while loop*.
This is implemented as a condition followed by one or more bodies that are lazily evaluated only if the condition is true.
Like the *if form*, if the conditional returns a non-boolean value the *while loop* will return an error.
#+BEGIN_SRC lisp
(while (get-my-flag global-state) ;; if false, returns (nothing) immediately
(dothing) ;; this is evaluated
"simple token" ;; this is also evaluated
(toggle-my-flag global-state)) ;; this is also evaluated
#+END_SRC
**** Let
*Let* is one of the most powerful forms Relish offers. The first body in a call to let is a list of lists.
Specifically, a list of variable declarations that lookf like this: ~(name value)~.
Each successive variable definition can build off of the last one, like this: ~((step1 "hello") (step2 (concat step1 " ")) (step3 (concat step2 "world")))~.
In said example, the resulting value of step3 is "hello world". After the variable declaration list, the next for is one or more unevaluated trees of code to be evaluated.
Here is an example of a complete let statement using hypothetical data and methods:
#+BEGIN_SRC lisp
;; Example let statement accepts one incoming connection on a socket and sends one response
(let ((conn (accept-conn listen-socket)) ;; start the var decl list, decl first var
(hello-pfx "hello from ") ;; start the var decl list, declare second var
(hello-msg (concat hello-pfx (get-server-name))) ;; declare third var from the second var
(hello-response (make-http-response 200 hello-msg))) ;; declare fourth var from the third, end list
(log (concat "response to " (get-dst conn) ": " hello-msg)) ;; evaluates a function call using data from the first and third vars
(send-response conn hello-response)) ;; evaluates a function call using data from the first and fourth vars
#+END_SRC
Here you can see the usefulness of being able to declare multiple variables in quick succession.
Each variable is in scope for the duration of the let statement and then dropped when the statement has concluded.
Thus, it is little cost to break complex calculations down into reusable parts.
**** TODO Circuit
**** Not quite control flow
2023-03-08 12:09:20 -08:00
Several other functions use lazy evaluation of their arguments. The below list is non-exhaustive:
- toggle
- inc
- dec
These functions are mentioned here for their use with control flow.
- inc: increment a symbol by one
- dec: decrement a symbol by one
- toggle: flip a symbol from true to false, or vice versa
For more information on these functions consult the output of the help function:
#+BEGIN_SRC lisp
λ (help toggle)
NAME: toggle
ARGS: 1 args of any type
DOCUMENTATION:
switches a boolean symbol between true or false.
Takes a single argument (a symbol). Looks it up in the variable table.
Either sets the symbol to true if it is currently false, or vice versa.
CURRENT VALUE AND/OR BODY:
<builtin>
#+END_SRC
*** TODO Quote and Eval
*** Lambda
Another form of homoiconicity is the *anonymous function*.
This is a nameless function being passed around as data.
It can be bound to a variable, or called directly.
An *anonymous function* is created with the ~lambda~ function.
Here is an example of a lambda function:
#+BEGIN_SRC lisp
(lambda (x y) (add x y))
;; | ^ this is the function body
;; +-> this is the argument list
#+END_SRC
The result of the lambda call is returned as a piece of data.
It can later be called inline or bound to a variable.
Here is an example of an inline lambda call:
#+BEGIN_SRC lisp
((lambda (x y) (add x y)) 1 2)
#+END_SRC
This call returns ~3~.
Here is the lambda bound to a variable inside a let statement:
#+BEGIN_SRC lisp
(let ((adder (lambda (x y) (add x y)))) ;; let form contains one local var
(adder 1 2)) ;; local var (lambda) called here
#+END_SRC
*** TODO Defining variables and functions
**** TODO Anatomy
**** TODO Naming conventions
**** TODO Undefining variables and functions
2023-03-08 11:41:00 -08:00
** Easy patterns
This section can serve as a sort of cookbook for a user who is new to leveraging LISP languages or unsure of where to start with ~relish~.
More ideas may be explored in the file:snippets directory of this project.
The author encourages any users to contribute their own personal favorites not already in this section either by adding them to the file:snippets folder, or to extend the documentation here.
*** while-let combo
#+BEGIN_SRC lisp
;; myiter = (1 (2 3 4 5 6))
(def myiter 'iterator over a list' (head (1 2 3 4 5 6)))
;; iterate over each element in mylist
(while (gt? (len (cdr myiter)) 0) ;; while there are more elements to consume
(let ((elem (car myiter)) ;; elem = consumed element from myiter
(remaining (cdr myiter))) ;; remaining = rest of elements
(echo elem) ;; do a thing with the element, could be any operation
(def myiter (head remaining)))) ;; consume next element, loop
#+END_SRC
The while-let pattern can be used for many purposes. Above it is used to iterate over elements in a list. It can also be used to receive connections to a socket and write data to them.
*** TODO main loop application
- state switch (while-toggle)
- state calculation
2023-03-08 11:41:00 -08:00
*** TODO callback model via eval and passed-in functions
*** TODO quote/eval for pseudo-anonymous pseudo-functions
*** TODO short-circuit guard
- circuit example
- while-not-circuit-do-more-work
*** let destructuring
~let~ is very useful for destructuring complex return types. If you have a function that may return a whole list of values you can then call it from ~let~ to consume the result data.
In this example a let form is used to destructure a call to ~head~. ~head~ returns a list consisting of ~(first-element rest-of-list)~ (for more information see ~(help head)~).
The ~let~ form starts with the output of ~head~ stored in ~head-struct~ (short for head-structured). The next variables defined are ~first~ and ~rest~ which contain individual elements from the return of the call to ~head~.
Finally, the bodies evaluated in the ~let~ form are able to operate on the head and the rest.
#+BEGIN_SRC lisp
;; individually access the top of a list
(let ((head-struct (head (1 2 3))
(first (car head-struct))
(rest (cdr head-struct)))
(echo "this is 1: " first)
(echo "this is 2, 3: " rest))
#+END_SRC
*** if-set?
One common pattern seen in bash scripts and makefiles is the set-variable-if-not-set pattern.
#+BEGIN_SRC shell
MYVAR ?= MY_SPECIAL_VALUE
#+END_SRC
Translated, can be seen below
#+BEGIN_SRC lisp
(if (set? myvar)
() ;; no need to do anything... or add a call here
(def myvar "MY_SPECIAL_VALUE"))
#+END_SRC
Alternatively this combination can be used to process flags in a script or application:
#+BEGIN_SRC lisp
(if (set? myflag)
(process-flag myflag)
())
#+END_SRC
** TODO Builtin functions
*** TODO Env function
** TODO Documentation
*** TODO Tests
*** TODO Help function
*** TODO Snippets directory
** Configuration
By default Relish will read from ~/.relishrc for configuration, but the default shell will also accept a filename from the RELISH_CFG_FILE environment variable.
See file:snippets/basic_minimal_configuration.rls for an example of a basic minimal configuration file.
*** The configuration file
The configuration file is a script containing arbitrary Relish code.
On start, any shell which leverages the configuration code in the config module (file:src/config.rs) will create a clean seperate context, including default configuration values, within which the standard library will be initialized.
The configuration file is evaluated and run as a standalone script and may include arbitrary executable code. Afterwards, configuration values found in the variable map will be used to configure the standard library function mappings that the shell will use.
Errors during configuration are non-terminal. In such a case any defaults which have not been overwritten will remain present.
**** Important points to note
- When the configuration file is run, it will be run with default configuration values.
- The user/script interpreter will be run with the standard library configured to use the previously defined configuration variables.
- The standard library will then be re-processed and re-added to the symbol table with new configuration.
- Variables and functions defined during configuration will carry over to the user/script interpreter, allowing the user to load any number of custom functions and variables.
*** Configuration Values
- CFG_RELISH_POSIX (default 0): when on, enables POSIX style job control.
- CFG_RELISH_ENV (default 1): when on, interpreter's variable table and environment variable table are kept in sync.
- CFG_RELISH_PROMPT (default (echo "λ ")): A *function* definition which is called in order to output the prompt for each loop of the REPL.
This function will be reloaded each REPL loop and will be called by the interpreter with no arguments.
*** Further configuration
Further configuration can be done by loading scripts that contain more functions and data to evaluate.
Variables and functions defined in an external script loaded by your interpreter will persist in the symbol table.
#+BEGIN_SRC lisp
(load "my-extra-library-functions.rls")
#+END_SRC
** Compilation
#+BEGIN_SRC sh
cargo build
#+END_SRC
** Testing
#+BEGIN_SRC sh
cargo test
#+END_SRC
** Running (the main shell)
#+BEGIN_SRC sh
cargo run src/bin/main.rs
#+END_SRC
* Guide to codebase
** file:tests directory
Start here if you are new.
Most of these files have unimplemented tests commented out in them.
Contributions that help fill out all of these tests
*** Eval tests: file:tests/test_eval.rs
These are particularly easy to read and write tests.
*** Func tests: file:tests/test_func.rs
You can consider these to extend the eval tests to cover the co-recursive nature between eval and func calls.
*** Lex tests: file:tests/test_lex.rs
These tests verify the handling of syntax.
*** Lib tests: (tests/test_lib*)
These tests are unique per stdlib module.
** file:src directory
This directory contains all of the user facing code in relish.
Just a few entries of note:
*** segment: file:src/segment.rs
This file lays out the +spiritual+ +theological+ +ideological+ +theoretical+ mechanical underpinnings of the entire interpreter.
The entire LISP machine centers around a singlet or pairing of datum.
The ~Ctr~ datatype provides an abstraction for which any type of data, including a ~Seg~ can be a datum.
The ~Seg~ datatype provides a mechanism to hold a single datum or a pair of datum. It is implemented as two ~Ctr~s: ~car~ and ~cdr~.
A primitive type system is provided through the Rust Enum datatype. A number of utility functions follow.
*** lib: file:src/lib.rs
This defines a library that can be included to provide an interpreter interface within any Rust project.
It includes the core interpreter mechanisms, full stdlib, and the configuration system.
Your project can use or not use any number of these components. They can certainly be used to support language development for other LISP machines,
or even other languages.
*** config: file:src/config.rs
This file contains default configuration values as well as functions which load and run the configuration file script.
For more information see the configuraiton section above in this Readme.
*** stl: file:src/stl.rs
This defines the ~static_stdlib~ function and the ~dynamic_stdlib~ function.
The ~static_stdlib~ function loads all symbols in the standard library which do not need further configuration into the symbol table.
The ~dyanmic_stdlib~ function loads all symbols in the standard library which *do* need configuration into the symbol table.
The ~dynamic_stdlib~ function uses variables saved in the symbol table to configure the functions and variables it loads.
For more information see file:src/config.
Any new addition to the stdlib must make its way here to be included in the main shell (and any other shell using the included ~get_stdlib~ function).
You may choose to override these functions if you would like to include your own special functions in your own special interpreter, or if you would like to pare down the stdlib to a small minimal subet of what it is.
You can view the code for standard library functions in file:src/stl/.
*** bin: file:src/bin/
This contains any executable target of this project. Notably the main shell file:src/bin/main.rs.
* Current Status / TODO list
Note: this section will not show the status of each item unless you are viewing it with a proper orgmode viewer.
Note: this section only tracks the state of incomplete TODO items. Having everything on here would be cluttered.
*** TODO list contains via circuit
2023-03-10 16:54:22 -08:00
*** TODO Map function
- DOCUMENTATION + TEST:
apply a lambda to a list
*** TODO Reduce function
*** TODO Input function
2023-03-10 13:16:25 -08:00
*** TODO Lex function
*** TODO Read function (Input + Lex)
2023-03-10 16:54:22 -08:00
*** TODO Caps function (list functions in table)
*** TODO Default prompt says "<minimum shell> LAMBDA:"
*** TODO Load (load a script) function
Pull/Refactor the logic out of the configure functions.
Optionally return a list of new variables and/or functions?
Will need a concatenate function for tables
*** TODO Main shell calls Load function on arg and exits
2023-03-10 16:54:22 -08:00
*** TODO Ship a relish-based stdlib
*** TODO Map library written in relish ("slowmap")
*** TODO FINISH DOCUMENTATION
2023-03-10 16:54:22 -08:00
*** TODO Shell module-
2023-03-10 13:16:25 -08:00
**** TODO only loadable via POSIX config var
Overload Load function to call a binary too
**** TODO arg processor because these are control flow
**** TODO Process launching with environment variables
**** TODO Optional form of process which allows fd redirecting
2023-03-10 13:16:25 -08:00
**** TODO Background processes (bg function)
**** TODO Foreground process TTY (fg function)
**** TODO list jobs (j function)
**** TODO ESSENTIAL: DOCUMENT POSIX MODULE
*** TODO Can enter multiple lines of text, with formatting in repl
*** TODO Rename to Flesh
*** TODO Create a dedicated community channel on matrix.sunnypup.io
*** TODO Post to relevant channels
*** TODO Custom ast pretty print
*** TODO Implement Compose for lambdas
*** TODO file operations
**** TODO read-to-string
**** TODO write-to-file
*** TODO Pick through core/ast and make sure no unnessesary moves are happening
*** TODO Network library
**** TODO HTTP Client
**** TODO TCP Stream client
**** TODO UDP Client
**** TODO TCP Listener
**** TODO HTTP Listener
**** TODO UDP Listener