This commit introduces the ability to reconfigure modules/event subscribers by
replaying previously stored events at startup. This way modules/subscribers can
always come up with the latest activity and data that was present at close at the
last run of the program. This process also includes the disposal of unneeded events
to minimize disk use over time. The changes include the following:
1. Event interface is extended with the Disposable() function
unimplemented stubs for existing interface implementations
2. state.Init() is split into Init() and Start()
Init() now initialized the file used by eventStream, and puts in place a 0 size
memory cache as a temporary measure on top of the file.
Start() reads Pop()s documents one by one from the temp nil-cache eventStream,
it attempts to dispose of each event and if the event is not disposable it is
pushed onto a second temporary no-memory-cache buffer which is then reverse ordered.
The underlying file is truncated and reopened.
Finally, the real eventStream is allocated, with in memory cache. All events in the
second buffer are published (sent to subscribers as well as added to the new eventStream).
3. Updates to main() to support Init() vs Start()
Signed-off-by: Ava Affine <ava@sunnypup.io>
This commit introduces a deferred call to a state teardown function
This is needed to properly flush data to the backing documentbuffer
as well as to truncate its underlying store. In doing so we make
sure data loss from process termination is minimal to nil.
when ever a panic happens or a signal is thrown the call to Teardown() is made
Signed-off-by: Ava Affine <ava@sunnypup.io>
This commit adds config values to the config package for state configuration.
This commit also adds a call to state.Init() to the main function.
Signed-off-by: Ava Affine <ava@sunnypup.io>
This commit refactors the state module to remove the eventCache and
introduce an eventStream instead. The eventStream is not a static array
but a DocumentBuffer, and thus many functions had to be altered. This
commit is best reviewed side by side with a before/after view instead of
just looking at the diff.
An Init() function is added to initialize the DocumentBuffer with config
values.
Facilities are added to both Event and EventType to allow for parsing
them to and from string documents.
a MakeEvent() function is added to create events of proper subtype based
on a general data map input.
ApplyToEvents() is added, which wraps around DocumentBuffer's Apply() method
to alter the filter input type with one that Marshals and Unmarshals events.
GetMatchingEvents() is now a proof of concept for DocumentBuffer's Apply()
PruneCache() is now no longer needed.
Signed-off-by: Ava Affine <ava@sunnypup.io>
The existing DocumentBuffer interface provided no facility
to iterate over events in order without caching them all in memory
and pushing/popping them. This was deemed significantly unoptimal and
so Apply() was born.
When Apply() is called, DocumentBuffer will recieve a filter callable
(functor, lambda, or closure) and will call it on all documents in order
throughout the entire cache and store, without caching stored events
or modifying the cache either.
The filter Apply() uses will recieve one document and return true or
false: true to continue iterating and false to stop iterating.
Signed-off-by: Ava Affine <ava@sunnypup.io>
In order to provide persistence of runtime state across the application
the documentbuffer provides a simple cache interface that balances cached
internal state events between an in memory cache and an on disk storage.
From a feature development perspective the DocumentBuffer provides a simple
cache interface:
- Push() and Pop() individual items
- Remove() and Read() bulk items
- Peek() at the most recent item
- Flush() items in memory to the disk
as well as some control features:
- Close(), which calls flush and then returns
index of the last byte of useful data in the
backing store.
- Cached(), which returns the number of items
cached in memory.
Underneath the hood, documentbuffer balances the cache (memory) and store
(disk) by "promoting" the most recent documents in store to cache and by
"demoting" the least recent documents in cache to store. Thus, the cache
is always ordered by most recent, and so is the store.
Documentbuffer takes any implementation of readwriteseeker as an interface
for a backing store. Theoretically this means that documentbuffer can leverage
more than just a standard os.File. Possible implementations could include
transactions over the network or device drivers for long term cold storage devices.
In fact, documentbuffer comes with an in memory test implementation of the
readwriteseeker interface called ReadWriteSeekString. This emulates the functions
of Read(), Write(), and Seek() to operate on an internal string buffer.
This facility is only provided for testing and mock up purposes.
One note about Close(): Since the documentbuffer has no way of truncating the
underlying store an edge case can present itself that necessitates Close() and
specifically Close()'s return type. If the documentbuffer has Remove()ed or
promote()ed more bytes of data from store than it will subsequently Flush() or
demote() to disk one or more bytes of junk data may be left over from the
overwriting of the previously Remove()/promote()ed data. In this case, or
more specifically in all cases Close() wil return the last usable index
of the underlying store. It is up to the caller to then truncate it.
Regretably there is no reasonable truncate interface that applies polymorphicly
to any underlying data stream. Thus the quirk around Close() remains a design
challenge.
This commit provides comprehensive unit tests for both DocumentBuffer and
ReadWriteSeekString.
Not implemented in this commit is the whole design for runtime data persistence.
It is intended that internal modules for bingobot that provide functionality
directly to the user leverage the event pub/sub system as their sole authoritative
source for state management. As long as these modules provide the same output
for the same input sequence of events consistently than all parts of the application
will recover from error status or crashes by re-ingesting the on disk storage
of events provided by the documentbuffer. This way the application will always
come back up with minimal data loss and potentially the exact same state as
before it went down.
Signed-off-by: Ava Affine <ava@sunnypup.io>
bingobot blindly uses the shell's working dir to write logs and read
configs. This change introduces the following changes:
1. start.sh now determines the project root's absolute path before
reading the token file and starting the app.
2. the config package now determine's the executable's working directory
(rather than assuming it's the same as the shell's) before loading
configs. It also now exposes this for use in other packages.
3. the logging package now uses config.GetWorkingDir() to determine
where to write logs.
This change introduces the UserActiveTimer, which tracks voice activity
and emits UserActive events.
UserActiveTimer is basically a fancy wrapper around a context with
a deadline and cancelFunc. When a user joins a voice channel, a
UserActiveTimer is started.
If the user stays in the voice channel for an amount of time defined in the configs,
the timer context's deadline trips and a UserActive event is emitted. A new timer is then started.
If instead the user leaves the voice channel, the timer's context is
cancelled.
This change introduces two config values to manage this process:
VoiceActivityThresholdSeconds defines the length of time a user is
required to stay in vc before a UserActive event is generated.
VoiceActivityTimerSleepInterval defines how long the timer sleeps at any
one time. After this interval, it wakes up to check if its context has
been cancelled.
This change also changes the state package's UserEvent validation to
remove an import loop. We will now assume that the discord package
has already validated any UIDs it passes along to the state package.
Logging needs to be initialized in order for the state unit tests
to function without issue.
See previous commit ("Fix config unit tests") for more information.
Signed-off-by: Ava Affine <ava@sunnypup.io>
Recent changes to config and logging modules have left the tests needing
to initialize both config and logging packages. This commit updates the
config module to automatically initialize into at least useful defaults
at module load time. This commit also fixes the config unit tests by
using the more up to date interface that the package provides.
Signed-off-by: Ava Affine <ava@sunnypup.io>
slog.HandlerOptions.AddSource is a cool feature, but since we moved the
logger to a separate package it no longer works. It now always shows
source info for the log package, rather than the code that called it.
This change removes it.
Before, we were exposing a Logger as logging.Log. This works, but
results in code calling Logger.Log.Info() which is a little wordy.
Instead, the logger is now kept private and we expose Info, Warn and
Error as public functions that wrap logger.Info, logger.Warn and
logger.Error.
Internal state tracking:
This commit introduces the concept of internal events. The thought here
is that bingobot will track its current state not by variables set from
incoming discord event data but rather from a cache of internal events that
represent digested and simplified observations made and actions taken by
internal code. Currently the events that are defined include the following:
- UserActive Events:
These events are intended to be generated when a user is noticed to have
been sitting in a voice chat for more than 10 minutes, however they can be
generated in response to anything that signifies that a user has been active
- Challenge Events:
These events signify that a user has been challenged. They are intended to
act as a primitive form of restraint from using bot features. It is intended
that these are made only as a response to a successful vote.
- Restore Events:
These events signify that a user has been restored to non-challenged status.
It is intended that this only happens in response to a successful vote.
- Vote Events:
These events serve as the primary means of consensus building, and represent the
state of a single active ongoing or finished vote.
Currently the events are cached for 10 days and then discarded, but future work will
add an on disk document storage for old events should any functionality desire it.
This internal event cache is intended to be the authoritative source of truth for
information about user permissions, vote results, and any additional future
functionality that relies on shared state. It is accessed through a GetMatchingEvents
function that accepts both an event type and a map of optional metadata filters.
See state_test.go for examples of this functionality. In addition to the cache
search function there is also a Publisher/Subscriber setting elaborated on below.
Pub/Sub system:
This commit also introduces a Pub/Sub system for other internal packages and
feature modules to recieve live events and continually process updates from our
code as opposed to just handling discord events.
- PublishEvent()
In this specific implementation anyone may become a Publisher by calling
PublishEvent() on any structure that implements the Event interface. This
provides for modules being able to define their own event types (so long as
they also extend the central enum of EventType in state.go).
- SubscribeWithHistory()
This returns a channel to the caller that returns copies of any event matching
the input event type. This channel will be prepopulated with up to 256 most
recent events already in the cache. Future work will make this number configurable.
Each event published will be sent down all relevant channels at publish time.
- Subscribe()
This returns a channel to the caller, as SubscribeWithHistory also does,
with the sole caveat that there are no historical "old" cached events
prepopulating the channel.
Finally, this commit also contains tests
Signed-off-by: Ava Affine <ava@sunnypup.io>
