Ava Affine
1ef8ff042f
In order to provide persistence of runtime state across the application
the documentbuffer provides a simple cache interface that balances cached
internal state events between an in memory cache and an on disk storage.
From a feature development perspective the DocumentBuffer provides a simple
cache interface:
- Push() and Pop() individual items
- Remove() and Read() bulk items
- Peek() at the most recent item
- Flush() items in memory to the disk
as well as some control features:
- Close(), which calls flush and then returns
index of the last byte of useful data in the
backing store.
- Cached(), which returns the number of items
cached in memory.
Underneath the hood, documentbuffer balances the cache (memory) and store
(disk) by "promoting" the most recent documents in store to cache and by
"demoting" the least recent documents in cache to store. Thus, the cache
is always ordered by most recent, and so is the store.
Documentbuffer takes any implementation of readwriteseeker as an interface
for a backing store. Theoretically this means that documentbuffer can leverage
more than just a standard os.File. Possible implementations could include
transactions over the network or device drivers for long term cold storage devices.
In fact, documentbuffer comes with an in memory test implementation of the
readwriteseeker interface called ReadWriteSeekString. This emulates the functions
of Read(), Write(), and Seek() to operate on an internal string buffer.
This facility is only provided for testing and mock up purposes.
One note about Close(): Since the documentbuffer has no way of truncating the
underlying store an edge case can present itself that necessitates Close() and
specifically Close()'s return type. If the documentbuffer has Remove()ed or
promote()ed more bytes of data from store than it will subsequently Flush() or
demote() to disk one or more bytes of junk data may be left over from the
overwriting of the previously Remove()/promote()ed data. In this case, or
more specifically in all cases Close() wil return the last usable index
of the underlying store. It is up to the caller to then truncate it.
Regretably there is no reasonable truncate interface that applies polymorphicly
to any underlying data stream. Thus the quirk around Close() remains a design
challenge.
This commit provides comprehensive unit tests for both DocumentBuffer and
ReadWriteSeekString.
Not implemented in this commit is the whole design for runtime data persistence.
It is intended that internal modules for bingobot that provide functionality
directly to the user leverage the event pub/sub system as their sole authoritative
source for state management. As long as these modules provide the same output
for the same input sequence of events consistently than all parts of the application
will recover from error status or crashes by re-ingesting the on disk storage
of events provided by the documentbuffer. This way the application will always
come back up with minimal data loss and potentially the exact same state as
before it went down.
Signed-off-by: Ava Affine <ava@sunnypup.io>
100 lines
1.7 KiB
Go
100 lines
1.7 KiB
Go
package docbuf
|
|
|
|
import (
|
|
"errors"
|
|
"io"
|
|
"fmt"
|
|
)
|
|
|
|
/* WARNING:
|
|
* This code is meant to assist with testing and mock ups
|
|
* It is not only not designed to any rigorous standards,
|
|
* but additionally does not offer any benefit over a static
|
|
* in memory single layer cache.
|
|
*/
|
|
|
|
type ReadWriteSeekString struct {
|
|
inner string
|
|
cursor int
|
|
}
|
|
|
|
func NewReadWriteSeekString() ReadWriteSeekString{
|
|
return ReadWriteSeekString{
|
|
inner: "",
|
|
cursor: 0,
|
|
}
|
|
}
|
|
|
|
func (s *ReadWriteSeekString) Read(
|
|
buf []byte,
|
|
) (int, error) {
|
|
i := 0
|
|
for ; i < len(buf); i++ {
|
|
if len(s.inner) <= s.cursor {
|
|
return i, nil
|
|
}
|
|
buf[i] = s.inner[s.cursor]
|
|
s.cursor += 1
|
|
}
|
|
return i, nil
|
|
}
|
|
|
|
func (s *ReadWriteSeekString) Write(
|
|
buf []byte,
|
|
) (int, error) {
|
|
backfillDelta := s.cursor - (len(s.inner) - 1)
|
|
if backfillDelta > 0 {
|
|
for range backfillDelta {
|
|
s.inner += "\x00"
|
|
}
|
|
}
|
|
|
|
tmpBuf := ""
|
|
if s.cursor > 0 {
|
|
tmpBuf += s.inner[:s.cursor]
|
|
}
|
|
tmpBuf += string(buf)
|
|
if s.cursor + len(buf) < (len(s.inner) - 1) {
|
|
tmpBuf += s.inner[s.cursor + len(buf):]
|
|
}
|
|
|
|
s.inner = tmpBuf
|
|
s.cursor += len(buf)
|
|
return len(buf), nil
|
|
}
|
|
|
|
func (s *ReadWriteSeekString) Seek(
|
|
offset int64,
|
|
whence int,
|
|
) (int64, error) {
|
|
var tmpCur int64
|
|
tmpCur = 0
|
|
|
|
switch whence {
|
|
case io.SeekCurrent:
|
|
tmpCur = int64(s.cursor)
|
|
case io.SeekEnd:
|
|
tmpCur = int64(len(s.inner))
|
|
case io.SeekStart:
|
|
tmpCur = int64(0)
|
|
default:
|
|
return int64(s.cursor),
|
|
errors.New("invalid whence value")
|
|
}
|
|
|
|
tmpCur += offset
|
|
if tmpCur < 0 {
|
|
msg := fmt.Sprintf("seek index (%d) is negative", tmpCur)
|
|
return int64(s.cursor), errors.New(msg)
|
|
}
|
|
s.cursor = int(tmpCur)
|
|
return tmpCur, nil
|
|
}
|
|
|
|
func (s *ReadWriteSeekString) Contents() string {
|
|
return s.inner
|
|
}
|
|
|
|
func (s *ReadWriteSeekString) Cursor() int {
|
|
return s.cursor
|
|
}
|