[*/+/-] MEGA COMMIT. ~2 weeks compressed.

The intention is to quickly improve and add util apis, enhance functionality given current demands, go back to the build pipeline, finish that, publish runtime tests, and then use what we have to go back to to linux support with a more stable api.

[+] AuMakeSharedArray
[+] Technet ArgvQuote
[+] Grug subsystem (UNIX signal thread async safe ipc + telemetry flusher + log flusher.)
[+] auEndianness -> Endian swap utils
[+] AuGet<N>(...)
[*] AUE_DEFINE conversion for
        ECompresionType, EAnsiColor, EHashType, EStreamError, EHexDump
[+] ConsoleMessage ByteBuffer serialization
[+] CmdLine subsystem for parsing command line arguments and simple switch/flag checks
[*] Split logger from console subsystem
[+] StartupParameters -> A part of a clean up effort under Process
[*] Refactor SysErrors header + get caller hack
[+] Atomic APIs
[+] popcnt
[+] Ring Buffer sink
[+] Added more standard errors
        Catch,
        Submission,
        LockError,
        NoAccess,
        ResourceMissing,
        ResourceLocked,
        MalformedData,
        InSandboxContext,
        ParseError

[+] Added ErrorCategorySet, ErrorCategoryClear, GetStackTrace
[+] IExitSubscriber, ETriggerLevel
[*] Write bias the high performance RWLockImpl read-lock operation operation
[+] ExitHandlerAdd/ExitHandlerRemove (exit subsystem)
[*] Updated API style
        Digests
[+] CpuId::CpuBitCount
[+] GetUserProgramsFolder
[+] GetPackagePath
[*] Split IStreamReader with an inl file
[*] BlobWriter/BlobReader/BlobArbitraryReader can now take shared pointers to bytebuffers. default constructor allocates a new scalable bytebuffer
[+] ICharacterProvider
[+] ICharacterProviderEx
[+] IBufferedCharacterConsumer
[+] ProviderFromSharedString
[+] ProviderFromString
[+] BufferConsumerFromProvider
[*] Parse Subsystem uses character io bufferer
[*] Rewritten NT's high perf semaphore to use userland SRW/ConVars [like mutex, based on generic semaphore]
[+] ByteBuffer::ResetReadPointer
[*] Bug fix bytebuffer base not reset on free and some scaling issues
[+] ProcessMap -> Added kSectionNameStack, kSectionNameFile, kSectionNameHeap for Section
[*] ProcessMap -> Refactor Segment to Section. I was stupid for keeping a type conflict hack API facing
[+] Added 64 *byte* fast RNG seeds
[+] File Advisorys/File Lock Awareness
[+] Added extended IAuroraThread from OS identifier caches for debug purposes
[*] Tweaked how memory is reported on Windows. Better consistency of what values mean across functions.
[*] Broke AuroraUtils/Typedefs out into a separate library
[*] Update build script
[+] Put some more effort into adding detail to the readme before rewriting it, plus, added some media
[*] Improved public API documentation
[*] Bug fix `SetConsoleCtrlHandler`
[+] Locale TimeDateToFileNameISO8601
[+] Console config stdOutShortTime
[*] Begin using internal UTF8/16 decoders when platform support isnt available (instead of stl)
[*] Bug fixes in decoders
[*] Major bug fix, AuMax
[+] RateLimiter
[+] Binary file sink
[+] Log directory sink
[*] Data header usability (more operators)
[+] AuRemoveRange
[+] AuRemove
[+] AuTryRemove
[+] AuTryRemoveRange
[+] auCastUtils
[+] Finish NewLSWin32Source
[+] AuTryFindByTupleN, AuTryRemoveByTupleN
[+] Separated AuRead/Write types, now in auTypeUtils
[+] Added GetPosition/SetPosition to FileWriter
[*] Fix stupid AuMin in place of AuMax in SpawnThread.Unix.Cpp
[*] Refactored Arbitrary readers to SeekingReaders (as in, they could be atomic and/or parallelized, and accept an arbitrary position as a work parameter -> not Seekable, as in, you can simply set the position)
[*] Hack back in the sched deinit
[+] File AIO loop source interop
[+] Begin to prototype a LoopQueue object I had in mind for NT, untested btw
[+] Stub code for networking
[+] Compression BaseStream/IngestableStreamBase
[*] Major: read/write locks now support write-entrant read routines.
[*] Compression subsystem now uses the MemoryView concept
[*] Rewrite the base stream compressions, made them less broken
[*] Update hashing api
[*] WriterTryGoForward and ReaderTryGoForward now revert to the previous relative index instead of panicing
[+] Added new AuByteBuffer apis
    Trim, Pad, WriteFrom, WriteString, [TODO: ReadString]
[+] Added ByteBufferPushReadState
[+] Added ByteBufferPushWriteState
[*] Move from USC-16 to full UTF-16. Win32 can handle full UTF-16.
[*] ELogLevel is now an Aurora enum
[+] Raised arbitrary limit in header to 255, the max filter buffer
[+] Explicit GZip support
[+] Explicit Zip support
[+] Added [some] compressors

et al

Aurora.json

@@ -15,7 +15,7 @@
         {
             "filter": {"platforms": "win32"},
             "then": {
-                "links": ["Bcrypt.lib", "UxTheme.lib", "Aux_ulib.lib", "Dbghelp.lib", "ws2_32.lib", "Ntdll.lib"]
+                "links": ["Bcrypt.lib", "UxTheme.lib", "Aux_ulib.lib", "Dbghelp.lib", "ws2_32.lib", "Ntdll.lib", "Wer.lib", "wintrust.lib"]
             }
         }
     ]

Include/Aurora/Async/AsyncTypes.hpp

@@ -9,7 +9,7 @@
 
 namespace Aurora::Loop
 {
-    class ILoopSource;
+    struct ILoopSource;
 }
 
 namespace Aurora::Async

Include/Aurora/Async/IThreadPool.hpp

@@ -51,9 +51,10 @@ namespace Aurora::Async
         virtual WorkerId_t GetCurrentThread()                                           = 0;
 
         // Synchronization
-        // Note: syncing to yourself will nullify requireSignal to prevent deadlock
+        // Note: syncing to yourself will nullify requireSignal to prevent deadlock conditions
         virtual bool Sync(WorkerId_t workerId, AuUInt32 timeoutMs = 0, bool requireSignal = false) = 0;
         virtual void Signal(WorkerId_t workerId)                                        = 0;
+        virtual AuSPtr<Loop::ILoopSource> WorkerToLoopSource(WorkerId_t id)             = 0;
         virtual void SyncAllSafe()                                                      = 0;
 
         // Features

Include/Aurora/Async/IWorkItem.hpp

@@ -32,6 +32,7 @@ namespace Aurora::Async
         // ns = time relative to the time at which the work item would otherwise dispatch
         virtual AuSPtr<IWorkItem> AddDelayTimeNs(AuUInt64 ns)                   = 0;
 
+        // inverted WaitFor
         virtual AuSPtr<IWorkItem> Then(const AuSPtr<IWorkItem> &next)           = 0;
 
         virtual AuSPtr<IWorkItem> Dispatch()                                    = 0;

Include/Aurora/Async/JobFrom.hpp

@@ -37,7 +37,7 @@ namespace Aurora::Async
         FJob<AuTuple<Args...>, Out_t> ret;
         ret.onSuccess = [=](const AuTuple<Args...> &in, const Out_t &a)
         {
-            std::apply(onSuccess, std::tuple_cat(in, AuMakeTuple<const Out_t &>(a)));
+            AuTupleApply(onSuccess, AuTupleCat(in, AuMakeTuple<const Out_t &>(a)));
         };
         return ret;
     }
@@ -48,12 +48,12 @@ namespace Aurora::Async
         FJob<AuTuple<Args...>, Out_t> ret;
         ret.onSuccess = [=](const AuTuple<Args...> &in, const Out_t &a)
         {
-            std::apply(onSuccess, std::tuple_cat(in, AuMakeTuple<const Out_t &>(a)));
+            AuTupleApply(onSuccess, AuTupleCat(in, AuMakeTuple<const Out_t &>(a)));
         };
 
         ret.onFailure = [=](const AuTuple<Args...> &in)
         {
-            std::apply(onFailure, in);
+            AuTupleApply(onFailure, in);
         };
         return ret;
     }
@@ -115,7 +115,7 @@ namespace Aurora::Async
         FJob<AuTuple<Arg0_t, Args...>, ReturnValue_t> ret;
         ret.onSuccess = [=](const AuTuple<Arg0_t, Args...> &in, const ReturnValue_t &out)
         {
-            onSuccess(std::get<0>(in), out);
+            onSuccess(AuGet<0>(in), out);
         };
         return ret;
     }
@@ -126,12 +126,12 @@ namespace Aurora::Async
         FJob<AuTuple<Arg0_t, Args...>, ReturnValue_t> ret;
         ret.onSuccess = [=](const AuTuple<Arg0_t, Args...> &in, const ReturnValue_t &out)
         {
-            onSuccess(std::get<0>(in), out);
+            onSuccess(AuGet<0>(in), out);
         };
 
         ret.onFailure = [=](const AuTuple<Arg0_t, Args...> &in)
         {
-            onFailure(std::get<0>(in));
+            onFailure(AuGet<0>(in));
         };
         return ret;
     }
@@ -154,7 +154,7 @@ namespace Aurora::Async
         FJob<AuTuple<AuSPtr<Clazz_t>, Args...>, ReturnValue_t> ret;
         ret.onSuccess = [=](const AuTuple<AuSPtr<Clazz_t>, Args...> &in, const ReturnValue_t &out)
         {
-            std::apply(onSuccess, std::tuple_cat(AuTuplePopFront(in), AuMakeTuple<const ReturnValue_t &>(out)));
+            AuTupleApply(onSuccess, AuTupleCat(AuTuplePopFront(in), AuMakeTuple<const ReturnValue_t &>(out)));
         };
         return ret;
     }
@@ -177,7 +177,7 @@ namespace Aurora::Async
         FJob<AuTuple<Args...>, ReturnValue_t> ret;
         ret.onSuccess = [=](const AuTuple<Args...> &in, const ReturnValue_t &out)
         {
-            std::apply(onSuccess, std::tuple_cat(in, AuMakeTuple<const ReturnValue_t &>(out)));
+            AuTupleApply(onSuccess, AuTupleCat(in, AuMakeTuple<const ReturnValue_t &>(out)));
         };
         return ret;
     }

Include/Aurora/Async/TaskFrom.hpp

@@ -34,7 +34,7 @@ namespace Aurora::Async
         FTask<AuTuple<Args...>, Out_t> ret;
         ret.onFrame = [callable = func](const AuTuple<Args...> &in) -> Out_t
         {
-            return std::apply(callable, in);
+            return AuTupleApply(callable, in);
         };
         return ret;
     }
@@ -45,7 +45,7 @@ namespace Aurora::Async
         FTask<AuTuple<Owner_t, Args...>, Out_t> ret;
         ret.onFrame = [callable = func](const AuTuple<Args...> &in)  -> Out_t
         {
-            return std::apply(callable, AuTuple_cat(AuMakeTuple<Owner_t>(ownerToPin), in));
+            return AuTupleApply(callable, AuTupleCat(AuMakeTuple<Owner_t>(ownerToPin), in));
         };
         return ret;
     }
@@ -56,7 +56,7 @@ namespace Aurora::Async
         Task_t ret;
         ret.onFrame = [callable = func](const auto &in)  -> ReturnValue_t
         {
-            return std::apply(callable, AuTuplePopFront(in));
+            return AuTupleApply(callable, AuTuplePopFront(in));
         };
         return ret;
     }
@@ -67,7 +67,7 @@ namespace Aurora::Async
         Task_t ret;
         ret.onFrame = [callable = func](const auto &in)  -> ReturnValue_t
         {
-            return std::apply(callable, AuTuplePopFront(in));
+            return AuTupleApply(callable, AuTuplePopFront(in));
         };
         return ret;
     }

Include/Aurora/CmdLine/CmdLine.hpp

@@ -0,0 +1,59 @@
+/***
+    Copyright (C) 2022 J Reece Wilson (a/k/a "Reece"). All rights reserved.
+
+    File: CmdLine.hpp
+    Date: 2022-1-31
+    Author: Reece
+    Note: Even kernels have a commandline so why not?
+***/
+#pragma once
+
+namespace Aurora::CmdLine
+{
+    /**
+     * @brief Returns a UTF-8 string array of argv[1, ...]
+     * @return 
+    */
+    AUKN_SYM const AuList<AuString> &GetCommandLineArguments();
+
+    /**
+     * @brief Performs a check on whether the exact key matches an argument
+     * @param key 
+     * @return 
+    */
+    AUKN_SYM bool HasFlag(const AuString &key);
+
+    /**
+     * @brief Performs a check on whether such string came before an equals sign
+     * @param key 
+     * @return 
+    */
+    AUKN_SYM bool HasValue(const AuString &key);
+
+    /**
+     * @brief Returns part after key= or defaultDefault
+     * @param key 
+     * @param defaultValue 
+     * @return 
+    */
+    AUKN_SYM const AuString &GetValue(const AuString &key, const AuString &defaultValue);
+    
+    /**
+     * @brief Returns part after key= or an empty string
+     * @param key 
+     * @return 
+    */
+    AUKN_SYM const AuString &GetValue(const AuString &key);
+
+    /**
+     * @brief Returns a constant array of flag keys
+     * @return 
+    */
+    AUKN_SYM const AuList<AuString> &GetFlags();
+
+    /**
+     * @brief Returns a constant array of value keys
+     * @return 
+    */
+    AUKN_SYM const AuList<AuString> &GetValues();
+}

Include/Aurora/Compression/BasicCompression.hpp

@@ -12,20 +12,10 @@ namespace Aurora::Compression
     /**
         Compresses an in memory blob with zstandard
     */
-    AUKN_SYM bool Compress(const void *buffer, AuUInt32 length, Memory::ByteBuffer &out, int compressionLevel = 3);
+    AUKN_SYM bool Compress(const Memory::MemoryViewRead &source, Memory::ByteBuffer &out, int compressionLevel = 3);
-
-    /**
-        Compresses an in memory blob with zstandard
-    */
-    AUKN_SYM bool Compress(const Memory::ByteBuffer &in, Memory::ByteBuffer &out, int compressionLevel = 3);
 
     /**
         Decompresses an in memory blob with zstandard
     */
-    AUKN_SYM bool Decompress(const void *buffer, AuUInt32 length, Memory::ByteBuffer &out);
+    AUKN_SYM bool Decompress(const Memory::MemoryViewRead &source, Memory::ByteBuffer &out);
-
-    /**
-        Decompresses an in memory blob with zstandard
-    */
-    AUKN_SYM bool Decompress(const Memory::ByteBuffer &in, Memory::ByteBuffer &out);
 }

Include/Aurora/Compression/Compression.hpp

@@ -7,6 +7,44 @@
 ***/
 #pragma once
 
+
+// Types
+#include "ECompresionType.hpp"
+#include "CompressionInfo.hpp"
+
+// Legacy API
+#include "StreamPipeProcessor.hpp"
+
+// Recommended API
+#include "ICompressionStream.hpp"
+#include "StreamProcessor.hpp"
+
+// Utility
+#include "BasicCompression.hpp"
+
+
+
+// TODO: neat comment + real world data
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
 /*
     Ballpark figures of real world performance:
         https://www.gaia-gis.it/fossil/librasterlite2/wiki?name=benchmarks+(2019+update) 
@@ -24,13 +62,3 @@
     ZSTD -> Standard use (~1.5GB/s to 2.5GB/s, respective compression ratios 2.4 and 2.1. Can be pushed to ~10 at around 750MB/s. Great general use. )
     ZIP  -> Zlib has a disgusting decompression upper limit of around 450MB/s for 2.7
 */
-
-#include "BasicCompression.hpp"
-
-#include "ECompresionType.hpp"
-#include "CompressionInfo.hpp"
-
-#include "StreamPipeProcessor.hpp"
-
-#include "ICompressionStream.hpp"
-#include "StreamProcessor.hpp"

Include/Aurora/Compression/CompressionInfo.hpp

@@ -13,38 +13,93 @@ namespace Aurora::Compression
     {
         ECompresionType type;
         
-        /// ZSTD: -5 <= level <= 22
+        /**
-        ///      recommended: ZSTD_CLEVEL_DEFAULT 
+         * @brief 
-        /// LZMA: 0 <= level <= 9
+         * 
-        /// LZ4 : N/A
+         * ZSTD: -5 <= level <= 22
-        /// ZLIB: 0 <= x >= 9
+         *      recommended: ZSTD_CLEVEL_DEFAULT 
-        ///      recommended: 6
+         * LZMA: 0 <= level <= 9
-        /// BZIP: 0 <= x >= 9
+         * LZ4 : N/A
-        AuInt8 compressionLevel{};
+         * Deflate: 0 <= x >= 9,
+         *      recommended: 6
+         * Zip: 0 <= x >= 9, 
+         *      recommended: 6
+         * GZip: 0 <= x >= 9, 
+         *      recommended: 6
+         * BZIP: 0 <= x >= 9
+         * 
+        */
+        AuInt8 compressionLevel {6};
 
-        /// LZMA: 5 <= fb <= 273, default = 32
+        /**
-        AuUInt32 fbWordSize{};
+         * @brief LZMA: 5 <= fb <= 273, default = 32
+        */
+        AuUInt32 fbWordSize {32};
 
-        /// LZMA only
+        /**
-        /// (1 << 12) <= dictSize <= (1 << 27) for 32-bit version
+         *  LZMA only
-        /// (1 << 12) <= dictSize <= (3 << 29) for 64-bit version
+         *  (1 << 12) <= dictSize <= (1 << 27) for 32-bit version
-        /// default = (1 << 24) 
+         *  (1 << 12) <= dictSize <= (3 << 29) for 64-bit version
+         *  default = (1 << 24) 
+        */
         AuUInt32 dictSize{};
 
-        // 64KiB is a recommended "small" block size
+        /**
+         * @brief 64KiB is a recommended "small" block size
+        */
         AuUInt16 lz4BlockSize {};
-    };
+
+        /**
+         * @brief DEFLATE related variabl
+         * 
+         * Deflate: 0 <= x >= 15,
+         *      recommended: 15
+         * Zip: 0 <= x >= 15,
+         *      recommended: 15
+         * GZip: 0 <= x >= 15,
+         *      recommended: 15
+        */
+        AuUInt8 windowBits {15};
+
+        /**
+         * @brief Internal output buffer size. 
+         *        Internal swap page is undefined.
+        */
+        AuUInt32 internalStreamSize {4096};
+
+        AuUInt8 threads {1};
+;    };
     
     struct DecompressInfo
     {
+        /**
+         * @brief algorithm
+        */
         ECompresionType alg {ECompresionType::eDeflate};
 
+        /**
+         * @brief Internal output buffer size. Internal swap page is undefined.
+        */
+        AuUInt32 internalStreamSize {4096};
+        
+        /**
+         * @brief Flag for headerless decompression streams
+        */
+        bool hasWindowbits {true};
+
+        /**
+         * @brief Flag for headerless decompression streams
+        */
+        AuInt8 windowBits {15};
+
         DecompressInfo(ECompresionType alg) : alg(alg)
         {
 
         }
 
-        AuUInt32 internalStreamSize {};
+        DecompressInfo(ECompresionType alg, AuUInt32 bufferSize) : alg(alg), internalStreamSize(bufferSize)
-        bool permitResize {};
+        {
+
+        }
     };
 }

Include/Aurora/Compression/ECompresionType.hpp

@@ -9,12 +9,14 @@
 
 namespace Aurora::Compression
 {
-    enum class ECompresionType
+    AUE_DEFINE(ECompresionType,
-    {
+    (
         eLZMA,
         eZSTD,
         eDeflate,
+        eZip,
+        eGZip,
         eLZ4,
         eBZIP2
-    };
+    ));
 }

Include/Aurora/Compression/ICompressionStream.hpp

@@ -9,25 +9,66 @@
 
 namespace Aurora::Compression
 {
-    class ICompressionStream
+    struct ICompressionStream
     {
-    public:
+        /**
-        /// Ingest n bytes from the input stream assigned to the compression object
+            * @brief Ingest n bytes from the input stream assigned to the compression object.
+            *        On error, returns {0, 0} or {bytesRead, 0} 
+            *        If the stream buffer runs out of memory, {bytesRead, 0} is expected, and although
+            *        `GetAvailableProcessedBytes()` will still return some data, the decompressed data 
+            *        or uncompressed stream will be dropped in part, and you should destroy from the stream object
+            * @param bytesFromUnprocessedInputSource 
+            * @return 
+        */
         virtual AuStreamReadWrittenPair_t Ingest(AuUInt32 bytesFromUnprocessedInputSource) = 0;
 
-        /// Limited stream API
+        /**
-        virtual bool ReadByProcessedN     (void * /*opt*/, AuUInt32 minimumProcessed, AuStreamReadWrittenPair_t &pair, bool ingestUntilEOS = true) = 0;
+         * @brief Returns the available bytes for immediate release
+         * @return
+        */
+        virtual AuUInt32 GetAvailableProcessedBytes() = 0;
 
-        /// Limited stream API
+        /**
-        virtual bool ReadByProcessedN     (void * /*opt*/, AuUInt32 minimumProcessed) = 0;
+         * @brief Returns the internal overhead to store the seekable stream buffer
+         * @return
+        */
+        virtual AuUInt32 GetInternalBufferSize() = 0;
 
-        /// Limited stream API
+        /**
+         * @brief Reads 'minimumProcessed', optionally into the first buffer, until EOS or destination length. 
+         *        If the destination is null and the length is a nonzero value, the stream seeks ahead
+         *        If the destination is null and the length is a zero, {0, GetAvailableProcessedBytes} is returned 
+         * @param destination
+         * @param ingestUntilEOS should continue to poll Ingest with an arbitrary page size to fulfill destination.length
+         * @return Bytes read / written
+        */
+        virtual AuStreamReadWrittenPair_t ReadEx(const Memory::MemoryViewWrite & /*opt*/ destination, bool ingestUntilEOS = true) = 0;
+        
+        /**
+         * @brief Reads 'minimumProcessed', optionally into the first buffer, from the internal stream buffer
+         * @param destination
+         * @return Bytes written
+        */
+        virtual AuUInt32 Read(const Memory::MemoryViewWrite & /*opt*/ destination) = 0;
+        
+        /**
+         * @brief Seek processed read functions backwards
+         * @param offset 
+         * @return 
+        */
         virtual bool GoBackByProcessedN   (AuUInt32 offset) = 0;
         
-        /// Limited stream API
+        /**
+         * @brief Seek read processed forward
+         * @param offset 
+         * @return 
+        */
         virtual bool GoForwardByProcessedN(AuUInt32 offset) = 0;
 
         /// Compression only
-        virtual void Flush() = 0;
+        virtual bool Flush() = 0;
+
+        /// Compression only
+        virtual bool Finish() = 0;
     };
 }

Include/Aurora/Compression/StreamPipeProcessor.hpp

@@ -14,7 +14,7 @@ namespace Aurora::Compression
         /// algorithm 
 
         /// LZMA decompression + compression, and ZSTD compression only
-        AuUInt32 threads;
+        AuUInt32 threads {1};
 
         /// consume from stream callback 
         AuFunction<AuUInt(void *, AuUInt)>     inPipe;

Include/Aurora/Compression/StreamProcessor.hpp

@@ -11,6 +11,13 @@
 
 namespace Aurora::Compression
 {
+    /**
+     * @brief 
+    */
     AUKN_SHARED_API(Decompressor, ICompressionStream, const AuSPtr<IO::IStreamReader> &reader, const DecompressInfo &info);
+
+    /**
+     * @brief
+    */
     AUKN_SHARED_API(Compressor, ICompressionStream, const AuSPtr<IO::IStreamReader> &reader, const CompressionInfo &info);
 }

Include/Aurora/Console/Commands/Commands.hpp

@@ -8,6 +8,7 @@
 #pragma once
 
 #include <Aurora/Parse/Parse.hpp>
+#include "ICommandSubscriber.hpp"
 
 namespace Aurora::Async
 {
@@ -15,8 +16,6 @@ namespace Aurora::Async
     struct WorkerPId_t;
 }
 
-#include "ICommandSubscriber.hpp"
-
 namespace Aurora::Console::Commands
 {
     AUKN_SYM void AddCommand(const AuString &tag, const Parse::ParseObject &commandStructure, const AuSPtr<ICommandSubscriber> &subscriber);

Include/Aurora/Console/Console.hpp

@@ -10,23 +10,13 @@
 #include "EAnsiColor.hpp"
 #include "ConsoleMessage.hpp"
 
-namespace Aurora::Console
+namespace Aurora::Logging
-{
-    namespace Logging
 {
     struct ILogger;
 }
 
-    /// Writes a log message to the console subscribers and telemetry outputs
+namespace Aurora::Console
-    AUKN_SYM void WriteLine(AuUInt8 level, const ConsoleMessage &msg);
+{
-
-    /**
-     * @brief Overloads the ILogger backend of the AuLogXX functions
-     * @param defaultGlobalLogger 
-     * @return 
-    */
-    AUKN_SYM void SetGlobalLogger(const AuSPtr<Logging::ILogger> &defaultGlobalLogger);
-
     /**
      * @brief Returns the untouched ILogger interface of the AuLogXX functions as configured by the Aurora::RuntimeStartInfo structure
      * @return 
@@ -63,4 +53,3 @@ namespace Aurora::Console
 
 #include "Commands/Commands.hpp"
 #include "Hooks/Hooks.hpp"
-#include "Logging/Logging.hpp"

Include/Aurora/Console/ConsoleMessage.hpp

@@ -47,6 +47,9 @@ namespace Aurora::Console
             tid   = Threading::Threads::GetThreadId();
         }
 
+        AUKN_SYM void     Read(Memory::ByteBuffer &deserialize);
+        AUKN_SYM void     Write(Memory::ByteBuffer &serialize) const;
+
         AUKN_SYM AuString StringifyTime(bool simple = false) const;
         AUKN_SYM AuString StringifyTimeUTC() const;
         AUKN_SYM AuString GetWrappedTag() const;

Include/Aurora/Console/EAnsiColor.hpp

@@ -9,8 +9,8 @@
 
 namespace Aurora::Console
 {
-    enum class EAnsiColor
+    AUE_DEFINE(EAnsiColor,
-    {
+    (
         eRed,
         eBoldRed,
         eGreen,
@@ -23,7 +23,6 @@ namespace Aurora::Console
         eBoldMagenta,
         eCyan,
         eBoldCyan,
-        eReset,
+        eReset
-        eCount
+    ));
-    };
 }

Include/Aurora/Console/Logging/IBasicSink.hpp

@@ -1,17 +0,0 @@
-/***
-    Copyright (C) 2021 J Reece Wilson (a/k/a "Reece"). All rights reserved.
-
-    File: IBasicSink.hpp
-    Date: 2021-11-1
-    Author: Reece
-***/
-#pragma once
-
-namespace Aurora::Console::Logging
-{
-    AUKN_INTERFACE(IBasicSink,
-        AUI_METHOD(void, OnMessageBlocking, (AuUInt8, level, const ConsoleMessage &, msg)),
-        AUI_METHOD(void, OnMessageNonblocking, (AuUInt8, level, const ConsoleMessage &, msg)),
-        AUI_METHOD(void, OnFlush, ())
-    )
-}

Include/Aurora/Console/Logging/IBasicSinkRB.hpp

@@ -1,17 +0,0 @@
-/***
-    Copyright (C) 2021 J Reece Wilson (a/k/a "Reece"). All rights reserved.
-
-    File: IBasicSinkRB.hpp
-    Date: 2021-11-2
-    Author: Reece
-***/
-#pragma once
-
-namespace Aurora::Console::Logging
-{
-    struct IBasicSinkRB : IBasicSink
-    {
-        virtual void SaveToPath(const AuString &path, bool plainText = false) = 0;
-        virtual AuList<ConsoleMessage> Export() = 0;
-    };
-}

Include/Aurora/Console/Logging/Sinks.hpp

@@ -1,23 +0,0 @@
-/***
-    Copyright (C) 2021 J Reece Wilson (a/k/a "Reece"). All rights reserved.
-
-    File: Sinks.hpp
-    Date: 2021-11-2
-    Author: Reece
-***/
-#pragma once 
-
-namespace Aurora
-{
-    struct SocketConsole;
-}
-
-namespace Aurora::Console::Logging
-{
-    AUKN_SHARED_API(NewStdSink, IBasicSink);
-    AUKN_SHARED_API(NewOSEventDirectorySink, IBasicSink);
-    AUKN_SHARED_API(NewFileSink, IBasicSink, const AuString &path, bool binary = false);
-    AUKN_SHARED_API(NewIPCSink,  IBasicSink, const SocketConsole &console);
-    AUKN_SHARED_API(NewRingLogger, IBasicSinkRB, AuUInt32 approxMaxBytes);
-    AUKN_SHARED_API(NewLogger, ILogger, const AuList<AuSPtr<IBasicSink>> &sinks);
-}

Include/Aurora/Crypto/ECC/EECCCurve.hpp

@@ -9,12 +9,12 @@
 
 namespace Aurora::Crypto::ECC
 {
-    enum EECCCurve
+    AUE_DEFINE(EECCCurve,
-    {
+    (
         eCurve256,
         eCurve384,
         eCurve521,
         eCurveX25519,
         eCurveEd25519
-    };
+    ));
 }

Include/Aurora/Data/TypedValue.hpp

@@ -13,5 +13,98 @@ namespace Aurora::Data
     {
         EDataType type;
         Value value;
+
+        TypedValue()
+        {
+            type = EDataType::kTypeEND;
+        }
+
+    #define TYPEDVALUE_CONSTRUCTOR(typen, typeval, datatype) inline TypedValue(typen typeval) : value(typeval), type(datatype) {}
+
+        TYPEDVALUE_CONSTRUCTOR(const AuVec3 &, vec3, EDataType::kTypeVec3);
+        TYPEDVALUE_CONSTRUCTOR(const AuVec4 &, vec4, EDataType::kTypeVec4);
+        TYPEDVALUE_CONSTRUCTOR(bool, boolean, EDataType::kTypeBoolean);
+        TYPEDVALUE_CONSTRUCTOR(double, number, EDataType::kTypeNumber);
+        TYPEDVALUE_CONSTRUCTOR(AuInt64, sint, EDataType::kTypeSInt);
+        TYPEDVALUE_CONSTRUCTOR(AuUInt64, uint, EDataType::kTypeUInt);
+        TYPEDVALUE_CONSTRUCTOR(uuids::uuid, uuid, EDataType::kTypeUUID);
+        TYPEDVALUE_CONSTRUCTOR(const AuString &, str, EDataType::kTypeString);
+
+    #undef TYPEDVALUE_CONSTRUCTOR
+
+
+        inline AuString ToString() const
+        {
+            if (!IsValid()) return "INVALID DATATYPE";
+            switch (type)
+            {
+            case EDataType::kTypeUInt:
+                return AuToString(value.primitive.uint);
+            case EDataType::kTypeSInt:
+                return AuToString(value.primitive.sint);
+            case EDataType::kTypeNumber:
+                return AuToString(value.primitive.number);
+            case EDataType::kTypeString:
+                return value.string;
+            case EDataType::kTypeBoolean:
+                return value.primitive.boolean ? "true" : "false";
+            case EDataType::kTypeUUID:
+                return uuids::to_string(value.UUID);
+            case EDataType::kTypeVec3:
+                return AuToString(value.primitive.vec3[0]) + "." +
+                       AuToString(value.primitive.vec3[1]) + "." +
+                       AuToString(value.primitive.vec3[2]);
+            case EDataType::kTypeVec4:
+                return AuToString(value.primitive.vec4[0]) + "." +
+                       AuToString(value.primitive.vec4[1]) + "." +
+                       AuToString(value.primitive.vec4[2]) + "." +
+                       AuToString(value.primitive.vec4[3]);
+            default:
+                return {};
+            }
+            return {};
+        }
+
+        inline bool operator==(const TypedValue &cmp) const
+        {
+            if (cmp.type != type)
+            {
+                return false;
+            }
+
+            switch (type)
+            {
+            case EDataType::kTypeUInt:
+                return cmp.value.primitive.uint == value.primitive.uint;
+            case EDataType::kTypeSInt:
+                return cmp.value.primitive.sint == value.primitive.sint;
+            case EDataType::kTypeNumber:
+                return cmp.value.primitive.number == value.primitive.number;
+            case EDataType::kTypeString:
+                return cmp.value.string == value.string;
+            case EDataType::kTypeBoolean:
+                return cmp.value.primitive.boolean == value.primitive.boolean;
+            case EDataType::kTypeUUID:
+                return cmp.value.UUID == value.UUID;
+            case EDataType::kTypeVec3:
+                return cmp.value.primitive.vec3 == value.primitive.vec3;
+            case EDataType::kTypeVec4:
+                return cmp.value.primitive.vec4 == value.primitive.vec4;
+            default:
+                return false;
+            }
+
+            return false;
+        }
+
+        inline bool IsValid() const
+        {
+            return static_cast<int>(type) < static_cast<int>(EDataType::kTypeGenericMax);
+        }
+
+        inline operator bool() const
+        {
+            return IsValid();
+        }
     };
 }

Include/Aurora/Debug/Debug.hpp

@@ -48,6 +48,24 @@ namespace Aurora::Debug
     
     AUKN_SYM void CheckErrors();
     
+    /**
+     * @brief Specifies an EFailureCategory value for the current thread
+     * @param category 
+     * @return 
+    */
+    AUKN_SYM void ErrorCategorySet(EFailureCategory category);
+
+    /**
+     * @brief Guarantees a refresh of ErrorCategoryGet's current value
+     * @return 
+    */
+    AUKN_SYM void ErrorCategoryClear();
+
+    /**
+     * @brief Returns the last EFailureCategory as specified by ErrorCategorySet or SysPushErrors
+     * @return 
+    */
+    AUKN_SYM EFailureCategory ErrorCategoryGet();
 
     AUKN_SYM StackTrace GetStackTrace();
     

Include/Aurora/Debug/EFailureCategory.hpp

@@ -34,7 +34,17 @@ namespace Aurora::Debug
         kFailureDisconnected,
         kFailureUninitialized,
         kFailureUnimplemented,
+        kFailureCatch,
+        kFailureSubmission,
+        kFailureLockError,
+        kFailureNoAccess,
+        kFailureResourceMissing,
+        kFailureResourceLocked,
+        kFailureMalformedData,
+        kFailureInSandboxContext,
+        kFailureParseError,
 
+        kFailureNone = 255,
         kFailureUserBegin = 256
     };
 }

Include/Aurora/Debug/SysAssertions.hpp

@@ -21,8 +21,8 @@
             return;
         }
 
-        Aurora::Console::Logging::WriteLinef(
+        Aurora::Logging::WriteLinef(
-            static_cast<AuUInt8>(Aurora::Console::Logging::ELogLevel::eError),
+            static_cast<AuUInt8>(Aurora::Logging::ELogLevel::eError),
             Aurora::Console::EAnsiColor::eBoldRed,
             "Fatal",
             "Expression address: {} {}:{}", func, file, fileno);
@@ -107,14 +107,14 @@
             return;
         }
 
-        Aurora::Console::Logging::WriteLinef(
+        Aurora::Logging::WriteLinef(
-            static_cast<AuUInt8>(Aurora::Console::Logging::ELogLevel::eError),
+            static_cast<AuUInt8>(Aurora::Logging::ELogLevel::eError),
             Aurora::Console::EAnsiColor::eBoldRed,
             "Fatal",
             "Expression address: {} {}:{}", func, file, fileno);
 
-        Aurora::Console::Logging::WriteLinef(
+        Aurora::Logging::WriteLinef(
-            static_cast<AuUInt8>(Aurora::Console::Logging::ELogLevel::eError),
+            static_cast<AuUInt8>(Aurora::Logging::ELogLevel::eError),
             Aurora::Console::EAnsiColor::eBoldRed,
             "Fatal",
             "Expression failed: {}", exp);
@@ -158,8 +158,8 @@
             return;
         }
 
-        Aurora::Console::Logging::WriteLinef(
+        Aurora::Logging::WriteLinef(
-            static_cast<AuUInt8>(Aurora::Console::Logging::ELogLevel::eError),
+            static_cast<AuUInt8>(Aurora::Logging::ELogLevel::eError),
             Aurora::Console::EAnsiColor::eBoldRed,
             "Fatal",
             "Expression failed: {}", exp);

Include/Aurora/Debug/SysErrors.hpp

@@ -28,23 +28,55 @@ namespace Aurora::Debug
         #define _FREECOMPILER_OPTIMIZE_OFF __attribute__((optimize("0")))
     #endif
     
-    static AU_NOINLINE void ErrorMakeNested() _FREECOMPILER_OPTIMIZE_OFF
+    // TODO: bring the xenus thing into here
+
+    static AU_NOINLINE AuUInt GetIPNoBackend() _FREECOMPILER_OPTIMIZE_OFF
+    {
+        return _DBG_RET_ADDR;
+    }
+
+    #if defined(AURORA_COMPILER_MSVC)
+    #pragma optimize("", on)
+    #endif
+    #undef _FREECOMPILER_OPTIMIZE_OFF
+
+    static auline void ErrorMakeNested()
     {
         _PushError(_DBG_RET_ADDR, EFailureCategory::kFailureNested, nullptr);
     }
     
     template<typename ... T>
-    static AU_NOINLINE void ErrorMakeNested(const AuString &msg, T&& ... args) _FREECOMPILER_OPTIMIZE_OFF
+    auline void ErrorMakeNested(const AuString &msg, T&& ... args)
+    {
+        if constexpr (sizeof...(T) == 0)
+        {
+            _PushError(_DBG_RET_ADDR, EFailureCategory::kFailureNested, msg.c_str());
+        }
+        else
         {
         #if defined(_AUHAS_FMT)
-        _PushError(_DBG_RET_ADDR, EFailureCategory::kFailureNested, fmt::format(msg, AuForward<T>(args)...).c_str());
+            try
+            {
+                auto tempString = fmt::format(msg, AuForward<T>(args)...);
+                _PushError(_DBG_RET_ADDR, EFailureCategory::kFailureNested, tempString.c_str());
+            }
+            catch (...)
+            {
+                _PushError(_DBG_RET_ADDR, EFailureCategory::kFailureNested, msg.c_str());
+            }
         #else
-        _PushError(_DBG_RET_ADDR, EFailureCategory::kFailureNested, nullptr);
+            _PushError(_DBG_RET_ADDR, EFailureCategory::kFailureNested, msg.c_str());
         #endif
         }
+    }
+
+    static auline void ErrorMakeNested(const char *msg)
+    {
+        _PushError(_DBG_RET_ADDR, EFailureCategory::kFailureNested, msg);
+    }
 
     template<typename ... T>
-    static AU_NOINLINE void SysPushError(EFailureCategory category, const AuString &msg, T&& ... args) _FREECOMPILER_OPTIMIZE_OFF
+    auline void SysPushError(EFailureCategory category, const AuString &msg, T&& ... args)
     {
         if constexpr (sizeof...(T) == 0)
         {
@@ -53,22 +85,30 @@ namespace Aurora::Debug
         else
         {
         #if defined(_AUHAS_FMT)
-            _PushError(_DBG_RET_ADDR, category, fmt::format(msg, AuForward<T>(args)...).c_str());
+            try
+            {
+                auto tempString = fmt::format(msg, AuForward<T>(args)...);
+                _PushError(_DBG_RET_ADDR, category, tempString.c_str());
+            }
+            catch (...)
+            {
+                _PushError(_DBG_RET_ADDR, category, msg.c_str());
+            }
         #else
-            _PushError(_DBG_RET_ADDR, category, "Missing dependency");
+            _PushError(_DBG_RET_ADDR, category, msg.c_str());
         #endif
         }
     }
 
-    static AU_NOINLINE void SysPushError(EFailureCategory category) _FREECOMPILER_OPTIMIZE_OFF
+    static auline void SysPushError(EFailureCategory category, const char *msg)
+    {
+        _PushError(_DBG_RET_ADDR, category, msg);
+    }
+
+    static auline void SysPushError(EFailureCategory category)
     {
         _PushError(_DBG_RET_ADDR, category, nullptr);
     }
-    
-    #if defined(AURORA_COMPILER_MSVC)
-        #pragma optimize("", on)
-    #endif
-    #undef _FREECOMPILER_OPTIMIZE_OFF
 }
 
 #define SysCheckReturn(x, ...)            if (!(static_cast<bool>(x))) { Aurora::Debug::ErrorMakeNested(); return __VA_ARGS__; }
@@ -88,6 +128,7 @@ namespace Aurora::Debug
 
 // enums
 #define SysPushErrorGeneric(...)          SysPushErrorError(kFailureGeneric,          ## __VA_ARGS__)
+#define SysPushErrorCatch(...)            SysPushErrorError(kFailureCatch,            ## __VA_ARGS__)
 #define SysPushErrorMemory(...)           SysPushErrorError(kFailureMemory,           ## __VA_ARGS__)
 #define SysPushErrorIO(...)               SysPushErrorError(kFailureIO,               ## __VA_ARGS__)
 #define SysPushErrorFIO(...)              SysPushErrorError(kFailureFIO,              ## __VA_ARGS__)
@@ -109,6 +150,15 @@ namespace Aurora::Debug
 #define SysPushErrorDisconnected(...)     SysPushErrorError(kFailureDisconnected,     ## __VA_ARGS__)
 #define SysPushErrorUninitialized(...)    SysPushErrorError(kFailureUninitialized,    ## __VA_ARGS__)
 #define SysPushErrorUnimplemented(...)    SysPushErrorError(kFailureUnimplemented,    ## __VA_ARGS__)
+#define SysPushErrorSubmission(...)       SysPushErrorError(kFailureSubmission,       ## __VA_ARGS__)
+#define SysPushErrorLockError(...)        SysPushErrorError(kFailureLockError,        ## __VA_ARGS__)
+#define SysPushErrorSyntax(...)           SysPushErrorError(kFailureSyntax,         ## __VA_ARGS__)
+#define SysPushErrorNoAccess(...)         SysPushErrorError(kFailureNoAccess,         ## __VA_ARGS__)
+#define SysPushErrorResourceMissing(...)  SysPushErrorError(kFailureResourceMissing,  ## __VA_ARGS__)
+#define SysPushErrorResourceLocked(...)   SysPushErrorError(kFailureResourceLocked,   ## __VA_ARGS__)
+#define SysPushErrorMalformedData(...)    SysPushErrorError(kFailureMalformedData,    ## __VA_ARGS__)
+#define SysPushErrorInSandboxContext(...) SysPushErrorError(kFailureInSandboxContext, ## __VA_ARGS__)
+#define SysPushErrorParseError(...)       SysPushErrorError(kFailureParseError,       ## __VA_ARGS__)
 
 
 #if defined(DEBUG) || defined(STAGING)
@@ -144,6 +194,16 @@ namespace Aurora::Debug
     #define SysPushErrorDisconnectedDbg          SysPushErrorDisconnected
     #define SysPushErrorUninitializedDbg         SysPushErrorUninitialized
     #define SysPushErrorUnimplementedDbg         SysPushErrorUnimplemented
+    #define SysPushErrorCatchDbg                 SysPushErrorCatch
+    #define SysPushErrorSubmissionDbg            SysPushErrorSubmission
+    #define SysPushErrorLockErrorDbg             SysPushErrorLockError
+    #define SysPushErrorSyntaxDbg                SysPushErrorSyntax
+    #define SysPushErrorNoAccessDbg              SysPushErrorNoAccess         
+    #define SysPushErrorResourceMissingDbg       SysPushErrorResourceMissing
+    #define SysPushErrorResourceLockedDbg        SysPushErrorResourceLocked
+    #define SysPushErrorMalformedDataDbg         SysPushErrorMalformedData
+    #define SysPushErrorInSandboxContextDbg      SysPushErrorInSandboxContext
+    #define SysPushErrorParseErrorDbg            SysPushErrorParseError
 
 #else
     
@@ -177,5 +237,16 @@ namespace Aurora::Debug
     #define SysPushErrorDisconnectedDbg(...)
     #define SysPushErrorUninitializedDbg(...)
     #define SysPushErrorUnimplementedDbg(...)
+    #define SysPushErrorCatchDbg(...)
+    #define SysPushErrorSubmissionDbg(...)
+    #define SysPushErrorLockErrorDbg(...)
+    #define SysPushErrorSyntaxDbg(...)
+    #define SysPushErrorNoAccessDbg(...)
+    #define SysPushErrorResourceMissingDbg(...)
+    #define SysPushErrorResourceLockedDbg(...)
+    #define SysPushErrorMalformedDataDbg(...)
+    #define SysPushErrorInSandboxContextDbg(...)
+    #define SysPushErrorParseErrorDbg(...)
+
 
 #endif

Include/Aurora/Debug/SysPanic.hpp

@@ -9,9 +9,9 @@
 
 #if defined(_AUHAS_FMT)
 template<typename ... T>
-static inline void __declspec(noreturn) SysPanic(T... args)
+static inline void AU_NORETURN SysPanic(T... args)
 {
-    Aurora::Console::Logging::WriteLinef(static_cast<AuUInt8>(Aurora::Console::Logging::ELogLevel::eError), Aurora::Console::EAnsiColor::eBoldRed, "Fatal", AuForward<T>(args)...);
+    Aurora::Logging::WriteLinef(static_cast<AuUInt8>(Aurora::Logging::ELogLevel::eError), Aurora::Console::EAnsiColor::eBoldRed, "Fatal", AuForward<T>(args)...);
     Aurora::Debug::Panic();
 }
 #endif

Include/Aurora/Exit/ETriggerLevel.hpp

@@ -0,0 +1,26 @@
+/***
+    Copyright (C) 2022 J Reece Wilson (a/k/a "Reece"). All rights reserved.
+
+    File: ETriggerLevel.hpp
+    Date: 2022-1-30
+    Author: Reece
+***/
+#pragma once
+
+namespace Aurora::Exit
+{
+    AUE_DEFINE(ETriggerLevel, (
+    
+        // Runtime deinitialize
+        eSafeTermination,
+        
+        // A fatal exception was caught by the watchdog. The process is on its way out
+        eFatalException,
+        
+        // Control+C was sent, termination must follow
+        eSigTerminate,
+        
+        // Something went wrong in the process. These could be frequent depending on the process
+        eProblematicEvent
+    ));
+}

Include/Aurora/Exit/Exit.hpp

@@ -0,0 +1,36 @@
+/***
+    Copyright (C) 2022 J Reece Wilson (a/k/a "Reece"). All rights reserved.
+
+    File: Exit.hpp
+    Date: 2022-1-30
+    Author: Reece
+***/
+#pragma once
+
+#include "ETriggerLevel.hpp"
+#include "IExitSubscriber.hpp"
+
+namespace Aurora::Exit
+{
+    /**
+     * @brief Registers @param callback to the registry to subscribe to @param level events
+     * @param level 
+     * @param callback 
+     * @return 
+    */
+    AUKN_SYM bool ExitHandlerAdd(ETriggerLevel level, const AuSPtr<IExitSubscriber> &callback);
+
+    /**
+     * @brief Removes all exit handlers by pointer
+     * @param callback 
+     * @return 
+    */
+    AUKN_SYM void ExitHandlerRemove(const AuSPtr<IExitSubscriber> &callback);
+
+
+    /**
+     * @brief Used from within callbacks to determine if the application is shutting down. 
+     * @return 
+    */
+    AUKN_SYM bool IsAppRunning();
+}

Include/Aurora/Exit/IExitSubscriber.hpp

@@ -0,0 +1,23 @@
+/***
+    Copyright (C) 2022 J Reece Wilson (a/k/a "Reece"). All rights reserved.
+
+    File: IExitSubscriber.hpp
+    Date: 2022-1-30
+    Author: Reece
+***/
+#pragma once
+
+namespace Aurora::Exit
+{
+    struct ExitInvoker
+    {
+        Threading::Threads::IAuroraThread *pCaller;
+    };
+    
+    /**
+     * @brief User definedable callback
+    */
+    AUKN_INTERFACE(IExitSubscriber,
+        AUI_METHOD(void, OnTrigger, (ETriggerLevel, level, const ExitInvoker *, pInvoker))
+    );
+}

Include/Aurora/HWInfo/CpuBitId.hpp

@@ -32,6 +32,7 @@ namespace Aurora::HWInfo
         inline CpuBitId Or(const CpuBitId &id) const;
         
         inline bool CpuBitScanForward(AuUInt8 &index, AuUInt8 offset) const;
+        inline AuUInt8 CpuBitCount() const;
 
         inline bool TestCpuIdx(AuUInt8 idx) const;
         inline void Clear();

Include/Aurora/HWInfo/CpuBitId.inl

@@ -33,6 +33,17 @@ namespace Aurora::HWInfo
    #endif
    }
 
+   inline AuUInt8 CpuBitId::CpuBitCount() const
+   {
+       return AuPopCnt(lower) +
+              AuPopCnt(upper)
+       #if defined(_AU_MASSIVE_CPUID)
+              + AuPopCnt(upper2)
+              + AuPopCnt(upper3)
+       #endif
+        ;
+   }
+
    bool CpuBitId::CpuBitScanForward(AuUInt8 &index, AuUInt8 offset) const
    {
    #if defined(_AU_MASSIVE_CPUID)

Include/Aurora/Hashing/Digests.hpp

@@ -9,53 +9,9 @@
 
 namespace Aurora::Hashing
 {
-    AUKN_SYM void MD5(const void *buffer, AuMach length, AuArray<AuUInt8, 16> &md5);
+    AUKN_SYM void MD5(const Memory::MemoryViewRead &span, AuArray<AuUInt8, 16> &md5);
-    static void MD5(const Memory::ByteBuffer &bytebuffer, AuArray<AuUInt8, 16> &md5)
+    AUKN_SYM void SHA1(const Memory::MemoryViewRead &span, AuArray<AuUInt8, 20> &sha1);
-    {
+    AUKN_SYM void Tiger(const Memory::MemoryViewRead &span, AuArray<AuUInt8, 24> &tiger);
-        return MD5(bytebuffer.data(), bytebuffer.size(), md5);
+    AUKN_SYM void SHA2(const Memory::MemoryViewRead &span, AuArray<AuUInt8, 32> &sha2);
-    }
+    AUKN_SYM void SHA2_64(const Memory::MemoryViewRead &span, AuArray<AuUInt8, 64> &sha2);
-    static void MD5(const AuString &bytebuffer, AuArray<AuUInt8, 16> &md5)
-    {
-        return MD5(bytebuffer.data(), bytebuffer.size(), md5);
-    }
-
-    AUKN_SYM void SHA1(const void *buffer, AuMach length, AuArray<AuUInt8, 20> &sha1);
-    static void SHA1(const Memory::ByteBuffer &bytebuffer, AuArray<AuUInt8, 20> &sha1)
-    {
-        return SHA1(bytebuffer.data(), bytebuffer.size(), sha1);
-    }
-    static void SHA1(const AuString &bytebuffer, AuArray<AuUInt8, 20> &sha1)
-    {
-        return SHA1(bytebuffer.data(), bytebuffer.size(), sha1);
-    }
-
-    AUKN_SYM void Tiger(const void *buffer, AuMach length, AuArray<AuUInt8, 24> &tiger);
-    static void Tiger(const Memory::ByteBuffer &bytebuffer, AuArray<AuUInt8, 24> &tiger)
-    {
-        return Tiger(bytebuffer.data(), bytebuffer.size(), tiger);
-    }
-    static void Tiger(const AuString &bytebuffer, AuArray<AuUInt8, 24> &tiger)
-    {
-        return Tiger(bytebuffer.data(), bytebuffer.size(), tiger);
-    }
-
-    AUKN_SYM void SHA2(const void *buffer, AuMach length, AuArray<AuUInt8, 32> &sha2);
-    static void SHA2(const Memory::ByteBuffer &bytebuffer, AuArray<AuUInt8, 32> &sha2)
-    {
-        return SHA2(bytebuffer.data(), bytebuffer.size(), sha2);
-    }
-    static void SHA2(const AuString &bytebuffer, AuArray<AuUInt8, 32> &sha2)
-    {
-        return SHA2(bytebuffer.data(), bytebuffer.size(), sha2);
-    }
-
-    AUKN_SYM void SHA2_64(const void *buffer, AuMach length, AuArray<AuUInt8, 64> &sha2);
-    static void SHA2_64(const Memory::ByteBuffer &bytebuffer, AuArray<AuUInt8, 64> &sha2)
-    {
-        return SHA2_64(bytebuffer.data(), bytebuffer.size(), sha2);
-    }
-    static void SHA2_64(const AuString &bytebuffer, AuArray<AuUInt8, 64> &sha2)
-    {
-        return SHA2_64(bytebuffer.data(), bytebuffer.size(), sha2);
-    }
 }

Include/Aurora/Hashing/EHashType.hpp

@@ -9,12 +9,12 @@
 
 namespace Aurora::Hashing
 {
-    enum class EHashType
+    AUE_DEFINE(EHashType,
-    {
+    (
         eMD5,
         eSHA1,
         eSHA2_32,
         eSHA2_64,
         eTiger
-    };
+    ));
 }

Include/Aurora/Hashing/HashStream.hpp

@@ -15,7 +15,7 @@ namespace Aurora::Hashing
         /**
          * Digest length of pBuf bytes 
          */
-        virtual void Ingest(const void *pBuf, AuUInt32 length) = 0;
+        virtual void Ingest(const Memory::MemoryViewRead &input) = 0;
 
         /**
          * Locks and returns the internal buffer

Include/Aurora/IO/Buffered/BlobArbitraryReader.hpp

@@ -1,50 +0,0 @@
-/***
-    Copyright (C) 2021 J Reece Wilson (a/k/a "Reece"). All rights reserved.
-
-    File: BlobArbitraryReader.hpp
-    Date: 2021-6-10
-    Author: Reece
-***/
-#pragma once
-
-namespace Aurora::IO::Buffered
-{
-    class BlobArbitraryReader : public IArbitraryStreamer
-    {
-    public:
-        AU_NO_COPY_NO_MOVE(BlobArbitraryReader)
-
-        BlobArbitraryReader(const Memory::ByteBuffer &buffer) : buffer_(buffer) {}
-        BlobArbitraryReader() {}
-        ~BlobArbitraryReader(){}
-
-        virtual EStreamError Open() override
-        {
-            return EStreamError::eErrorNone;
-        }
-
-        virtual EStreamError ArbitraryRead(AuUInt32 offset, const Memory::MemoryViewStreamWrite &paramters) override
-        {
-            if (buffer_.empty()) return EStreamError::eErrorEndOfStream;
-
-            auto endOffset     = offset + paramters.length;
-            auto realEndOffset = AuMin(buffer_.size(), endOffset);
-            auto actualLength  = realEndOffset - offset;
-
-            if (actualLength < 0) return EStreamError::eErrorEndOfStream;
-
-            paramters.outVariable = actualLength;
-            AuMemcpy(paramters.ptr, buffer_.data() + offset, paramters.outVariable);
-
-            return EStreamError::eErrorNone;
-        }
-
-        virtual void Close() override
-        {
-            buffer_.clear();
-        }
-
-    private:
-        Memory::ByteBuffer buffer_;
-    };
-}

Include/Aurora/IO/Buffered/BlobReader.hpp

@@ -9,39 +9,33 @@
 
 namespace Aurora::IO::Buffered
 {
-    class BlobReader : public IStreamReader
+    struct BlobReader : public IStreamReader
     {
-    public:
         AU_NO_COPY(BlobReader)
         
-        BlobReader(const AuSPtr<Memory::ByteBuffer> &buffer) : buffer_(buffer) {}
+        inline BlobReader(const AuSPtr<Memory::ByteBuffer> &buffer) : buffer_(buffer) {}
-        BlobReader(Memory::ByteBuffer &&buffer) : buffer_(AuMakeShared<Memory::ByteBuffer>(buffer)) {}
+        inline BlobReader(Memory::ByteBuffer &&buffer) : buffer_(AuMakeShared<Memory::ByteBuffer>(buffer)) {}
-        BlobReader(const Memory::ByteBuffer &buffer) : buffer_(AuMakeShared<Memory::ByteBuffer>(buffer)) {}
+        inline BlobReader(const Memory::ByteBuffer &buffer) : buffer_(AuMakeShared<Memory::ByteBuffer>(buffer)) {}
 
-        BlobReader()  {}
+        inline BlobReader()  {}
-        ~BlobReader() {}
+        inline ~BlobReader() {}
 
-        virtual EStreamError Open() override
+        inline virtual EStreamError Open() override
         {
             if (!buffer_) return EStreamError::eErrorStreamNotOpen;
-            if (!buffer_->operator bool()) return EStreamError::eErrorStreamNotOpen;
             return EStreamError::eErrorNone;
         }
 
-        virtual EStreamError Read(const Memory::MemoryViewStreamWrite &paramters) override
+        inline virtual EStreamError Read(const Memory::MemoryViewStreamWrite &parameters) override
         {
-            auto realEndOffset = AuMin(buffer_->size() - offset_, paramters.length);
+            if (!buffer_) return EStreamError::eErrorStreamNotOpen;
-            if (realEndOffset == 0) return EStreamError::eErrorEndOfStream;
+            parameters.outVariable = buffer_->Read(parameters.ptr, parameters.length);
-
+            return parameters.outVariable == 0 ? EStreamError::eErrorEndOfStream : EStreamError::eErrorNone;
-            paramters.outVariable = realEndOffset;
-            AuMemcpy(paramters.ptr, buffer_->data() + offset_, realEndOffset);
-            offset_ += realEndOffset;
-
-            return EStreamError::eErrorNone;
         }
 
-        virtual void Close() override
+        inline virtual void Close() override
         {
+            if (!buffer_) return;
             buffer_->clear();
         }
 

Include/Aurora/IO/Buffered/BlobSeekableReader.hpp

@@ -0,0 +1,46 @@
+/***
+    Copyright (C) 2021 J Reece Wilson (a/k/a "Reece"). All rights reserved.
+
+    File: BlobSeekableReader.hpp
+    Date: 2021-6-10
+    Author: Reece
+***/
+#pragma once
+
+namespace Aurora::IO::Buffered
+{
+    struct BlobSeekableReader : public ISeekingReader
+    {
+        AU_NO_COPY_NO_MOVE(BlobSeekableReader)
+
+        inline BlobSeekableReader(const AuSPtr<Memory::ByteBuffer> &buffer) : buffer_(buffer) {}
+        inline BlobSeekableReader(Memory::ByteBuffer &&buffer) : buffer_(AuMakeShared<Memory::ByteBuffer>(buffer)) {}
+        inline BlobSeekableReader(const Memory::ByteBuffer &buffer) : buffer_(AuMakeShared<Memory::ByteBuffer>(buffer)) {}
+
+        inline BlobSeekableReader() {}
+        inline ~BlobSeekableReader(){}
+
+        inline virtual EStreamError Open() override
+        {
+            if (!buffer_) return EStreamError::eErrorStreamNotOpen;
+            return EStreamError::eErrorNone;
+        }
+
+        inline virtual EStreamError ArbitraryRead(AuUInt offset, const Memory::MemoryViewStreamWrite &parameters) override
+        {
+            if (!buffer_) return EStreamError::eErrorStreamNotOpen;
+            if (buffer_->empty()) return EStreamError::eErrorEndOfStream;
+            buffer_->readPtr = buffer_->base + offset;
+            parameters.outVariable = buffer_->Read(parameters.ptr, parameters.length);
+            return parameters.outVariable == 0 ? EStreamError::eErrorEndOfStream : EStreamError::eErrorNone;
+        }
+
+        inline virtual void Close() override
+        {
+            if (buffer_) buffer_->clear();
+        }
+
+    private:
+        AuSPtr<Memory::ByteBuffer> buffer_;
+    };
+}

Include/Aurora/IO/Buffered/BlobWriter.hpp

@@ -9,41 +9,40 @@
 
 namespace Aurora::IO::Buffered
 {
-    class BlobWriter : public IStreamWriter
+    struct BlobWriter : public IStreamWriter
     {
-    public:
         AU_NO_COPY_NO_MOVE(BlobWriter)
         
-        BlobWriter()  {}
+        inline BlobWriter(const AuSPtr<Memory::ByteBuffer> &buffer) : buffer_(buffer) {}
-        ~BlobWriter() {}
+        inline BlobWriter() : buffer_(AuMakeShared<Memory::ByteBuffer>()) {}
+        inline ~BlobWriter() {}
 
-        virtual EStreamError Open() override
+        inline virtual EStreamError Open() override
         {
+            return buffer_ ? EStreamError::eErrorNone : EStreamError::eErrorStreamNotOpen;
+        }
+
+        inline virtual EStreamError Write(const Memory::MemoryViewStreamRead &parameters) override
+        {
+            if (!buffer_) return EStreamError::eErrorStreamNotOpen;
+            parameters.outVariable = buffer_->Write(parameters.ptr, parameters.length);
             return EStreamError::eErrorNone;
         }
 
-        virtual EStreamError Write(const Memory::MemoryViewStreamRead & parameters) override
+        inline virtual void Flush() override
-        {
-            auto idx = buffer_.size();
-            buffer_.resize(idx + parameters.length);
-            AuMemcpy(buffer_.data() + idx, parameters.ptr, buffer_.size() - idx);
-            return EStreamError::eErrorNone;
-        }
-
-        virtual void Flush() override
         {
         }
 
-        virtual void Close() override
+        inline virtual void Close() override
         {
         }
 
-        const Memory::ByteBuffer &GetBuffer()
+        inline AuSPtr<Memory::ByteBuffer> GetBuffer()
         {
             return this->buffer_;
         }
 
     private:
-        Memory::ByteBuffer buffer_;
+        AuSPtr<Memory::ByteBuffer> buffer_;
     };
 }

Include/Aurora/IO/Buffered/Buffered.hpp

@@ -9,6 +9,6 @@
 
 // Most internal transactions will be buffered. 
 // For sake of being able to mix network, file, serial, and other code, its nice to use the same reader interface for buffered and streams where read/write until EOS is required
-#include "BlobArbitraryReader.hpp"
+#include "BlobSeekableReader.hpp"
 #include "BlobReader.hpp"
 #include "BlobWriter.hpp"

Include/Aurora/IO/Character/Character.hpp

@@ -8,5 +8,7 @@
 #pragma once
 
 #include "ICharacterProvider.hpp"
+#include "ICharacterProviderEx.hpp"
 #include "IBufferedCharacterConsumer.hpp"
-#include "ILineBufferer.hpp"
+#include "IBufferedLineReader.hpp"
+#include "Providers.hpp"

Include/Aurora/IO/Character/IBufferedCharacterConsumer.hpp

@@ -9,5 +9,15 @@
 
 namespace Aurora::IO::Character
 {
+    struct IBufferedCharacterConsumer
+    {
+        virtual bool HasBufferedNext() const = 0;
+        virtual AuUInt8 NextBufferedByte() const = 0;
         
+        virtual bool PeekNext(AuUInt8 &out) = 0;
+        virtual AuUInt8 GetCurrent() = 0;
+        virtual bool Next(AuUInt8 &out) = 0;
+    };
+
+    AUKN_SHARED_API(BufferConsumerFromProvider, IBufferedCharacterConsumer, const AuSPtr<ICharacterProvider> &provider);
 }

Include/Aurora/IO/Character/IBufferedLineReader.hpp

@@ -0,0 +1,20 @@
+/***
+    Copyright (C) 2022 J Reece Wilson (a/k/a "Reece"). All rights reserved.
+
+    File: IBufferedLineReader.hpp
+    Date: 2022-1-29
+    Author: Reece
+***/
+#pragma once
+
+namespace Aurora::IO::Character
+{
+    struct IBufferedLineReader
+    {
+        virtual bool ReadBytes(AuUInt32 length) = 0;
+        virtual void Flush() = 0;
+        virtual AuList<AuString> ReadLines() = 0;
+    };
+    
+    AUKN_SHARED_API(NewLineReader, IBufferedLineReader, const AuSPtr<IStreamReader> &input);
+}

Include/Aurora/IO/Character/ICharacterProvider.hpp

@@ -10,6 +10,6 @@
 namespace Aurora::IO::Character
 {
     AUKN_INTERFACE(ICharacterProvider, 
-        AUI_METHOD(AuUInt8, GetByte, ())
+        AUI_METHOD(bool, GetByte, (AuUInt8 &, val))
     );
 }

Include/Aurora/IO/Character/ICharacterProviderEx.hpp

@@ -0,0 +1,17 @@
+/***
+    Copyright (C) 2022 J Reece Wilson (a/k/a "Reece"). All rights reserved.
+
+    File: ICharacterProviderEx.hpp
+    Date: 2022-1-29
+    Author: Reece
+***/
+#pragma once
+
+namespace Aurora::IO::Character
+{
+    struct ICharacterProviderEx : ICharacterProvider
+    {
+        virtual AuUInt GetPosition() = 0;
+        virtual bool SetPosition(AuUInt offset) = 0;
+    };
+}

Include/Aurora/IO/Character/Providers.hpp

@@ -0,0 +1,16 @@
+/***
+    Copyright (C) 2022 J Reece Wilson (a/k/a "Reece"). All rights reserved.
+
+    File: Providers.hpp
+    Date: 2022-1-29
+    Author: Reece
+***/
+#pragma once
+
+namespace Aurora::IO::Character
+{
+    // you are responsible for maintaining a lock over shared string writes and provider use instances
+    // (string may change so long as a read/write lock is protecting ICharacterProviderEx reads)
+    AUKN_SHARED_API(ProviderFromSharedString, ICharacterProviderEx, const AuSPtr<AuString> &str, AuUInt index = 0);
+    AUKN_SHARED_API(ProviderFromString, ICharacterProviderEx, const AuString &str, AuUInt index = 0);
+}

Include/Aurora/IO/EStreamError.hpp

@@ -9,12 +9,11 @@
 
 namespace Aurora::IO
 {
-    enum class EStreamError
+    AUE_DEFINE_VA(EStreamError,
-    {
         eErrorNone,
         eErrorEndOfStream,
         eErrorStreamNotOpen,
         eErrorStreamInterrupted,
         eErrorHandleClosed
-    };
+    );
 }

Include/Aurora/IO/FS/Async.hpp

@@ -9,36 +9,7 @@
 
 namespace Aurora::IO::FS
 {
-    class IAsyncTransaction;
+    AUKN_SHARED_API(OpenAsync, IAsyncFileStream, const AuString &path, EFileOpenMode openMode, bool directIO = false, EFileAdvisoryLockLevel lock = EFileAdvisoryLockLevel::eNoSafety);
-    
-    class IAsyncFileStream
-    {
-    public:
-        virtual AuSPtr<IAsyncTransaction> NewTransaction() = 0;
-    };
-    
-    class IAsyncFinishedSubscriber
-    {
-    public:
-        virtual void OnAsyncFileOpFinished(AuUInt64 offset, AuUInt32 length) = 0;
-    };
-    
-    class IAsyncTransaction
-    {
-    public:
-        // Do not switch to Aurora::Memory::MemoryView, you must use a raw pointer to the parent object that effectively owns the IAsyncTransaction
-        virtual bool StartRead(AuUInt64 offset, void *, AuUInt32 length) = 0;
-        virtual bool StartWrite(AuUInt64 offset, const void *, AuUInt32 length) = 0;
-        
-        virtual bool Complete() = 0;
-        virtual AuUInt32 GetLastPacketLength() = 0;
-        
-        virtual void SetCallback(const AuSPtr<IAsyncFinishedSubscriber> &sub) = 0;
-        
-        virtual bool Wait(AuUInt32 timeout) = 0;
-    };
-    
-    AUKN_SHARED_API(OpenAsync, IAsyncFileStream, const AuString &path, bool readOnly = true, bool directIO = false);
     
     /// \param transactions Array of FIO transactions
     /// \param timeout Aurora Timeout

Include/Aurora/IO/FS/EFileAdvisoryLockLevel.hpp

@@ -0,0 +1,29 @@
+/***
+    Copyright (C) 2022 J Reece Wilson (a/k/a "Reece"). All rights reserved.
+
+    File: EFileAdvisoryLockLevel.hpp
+    Date: 2022-1-29
+    Author: Reece
+***/
+#pragma once
+
+namespace Aurora::IO::FS
+{
+    // I know NT can do much better, network drives and unix can not. 
+    // Do not open an issue over this
+    // There is no portable solution for file locks
+    
+    AUE_DEFINE(EFileAdvisoryLockLevel, 
+    (
+        // Nothing will invalidate the file stream, potentially causing corruption on lock race
+        // No lock will be placed on the file nor created to be polled
+        // File locks just aren't portable enough yet
+        eNoSafety,
+
+        // Prevent software that is willing to play ball from writing to the file path
+        eBlockWrite,
+
+        // Prevent software that is willing to play ball from reading the file, write blocking inherited 
+        eBlockReadWrite
+    ));
+}

Include/Aurora/IO/FS/EFileOpenMode.hpp

@@ -0,0 +1,18 @@
+/***
+    Copyright (C) 2022 J Reece Wilson (a/k/a "Reece"). All rights reserved.
+
+    File: EOpenMode.hpp
+    Date: 2022-1-29
+    Author: Reece
+***/
+#pragma once
+
+namespace Aurora::IO::FS
+{
+    AUE_DEFINE(EFileOpenMode,
+    (
+        eRead,
+        eWrite,
+        eReadWrite
+    ));
+}

Include/Aurora/IO/FS/FS.hpp

@@ -44,11 +44,16 @@ namespace Aurora::IO::FS
     AUKN_SYM bool GoUpToSeparator(AuString &out, const AuString &path);
 }
 
+#include "EFileAdvisoryLockLevel.hpp"
+#include "EFileOpenMode.hpp"
 #include "IFileStream.hpp"
 #include "FileStream.hpp"
-#include "FileArbitraryReader.hpp"
+#include "FileSeekableReader.hpp"
 #include "FileReader.hpp"
 #include "FileWriter.hpp"
 #include "Resources.hpp"
 #include "Stat.hpp"
+#include "IAsyncFileStream.hpp"
+#include "IAsyncFinishedSubscriber.hpp"
+#include "IAsyncTransaction.hpp"
 #include "Async.hpp"

Include/Aurora/IO/FS/FileReader.hpp

@@ -9,27 +9,26 @@
 
 namespace Aurora::IO::FS
 {
-    class FileReader : public IStreamReader
+    struct FileReader : public IStreamReader
     {
-    public:
         AU_NO_COPY_NO_MOVE(FileReader)
         
-        FileReader() {}
+        inline FileReader() {}
-        ~FileReader() {}
+        inline ~FileReader() {}
 
         template<typename... T>
-        bool OpenFile(T... args)
+        inline bool OpenFile(T... args)
         {
             stream_ = OpenReadUnique(args...);
             return stream_ != nullptr;
         }
 
-        virtual EStreamError Open() override
+        inline virtual EStreamError Open() override
         {
             return stream_ ? EStreamError::eErrorNone : EStreamError::eErrorStreamNotOpen;
         }
 
-        virtual EStreamError Read(const Memory::MemoryViewStreamWrite &paramters) override
+        inline virtual EStreamError Read(const Memory::MemoryViewStreamWrite &paramters) override
         {
             if (!stream_) return EStreamError::eErrorStreamNotOpen;
             if (!stream_->Read(paramters)) return EStreamError::eErrorStreamInterrupted;
@@ -37,7 +36,7 @@ namespace Aurora::IO::FS
             return EStreamError::eErrorNone;
         }
 
-        virtual void Close() override
+        inline virtual void Close() override
         {
             stream_.reset();
         }

Include/Aurora/IO/FS/FileSeekableReader.hpp

@@ -1,7 +1,7 @@
 /***
     Copyright (C) 2021 J Reece Wilson (a/k/a "Reece"). All rights reserved.
 
-    File: FileArbitraryReader.hpp
+    File: FileSeekableReader.hpp
     Date: 2021-6-10
     Author: Reece
 ***/
@@ -9,13 +9,12 @@
 
 namespace Aurora::IO::FS
 {
-    class FileArbitraryReader : public IArbitraryStreamer
+    struct FileSeekableReader : public ISeekingReader
     {
-    public:
+        AU_NO_COPY_NO_MOVE(FileSeekableReader)
-        AU_NO_COPY_NO_MOVE(FileArbitraryReader)
         
-        FileArbitraryReader() {}
+        FileSeekableReader() {}
-        ~FileArbitraryReader() {}
+        ~FileSeekableReader() {}
         
         template<typename... T>
         bool OpenFile(T... args)
@@ -29,7 +28,7 @@ namespace Aurora::IO::FS
             return stream_ ? EStreamError::eErrorNone : EStreamError::eErrorStreamInterrupted;
         }
 
-        virtual EStreamError ArbitraryRead(AuUInt32 offset, const Memory::MemoryViewStreamWrite &paramters) override
+        virtual EStreamError ArbitraryRead(AuUInt offset, const Memory::MemoryViewStreamWrite &paramters) override
         {
             if (!stream_) return EStreamError::eErrorStreamNotOpen;
             if (!stream_->SetOffset(offset)) return EStreamError::eErrorEndOfStream;

Include/Aurora/IO/FS/FileStream.hpp

@@ -9,6 +9,8 @@
 
 namespace Aurora::IO::FS
 {
-    AUKN_SHARED_API(OpenRead, IFileStream, const AuString &path);
+    AUKN_SHARED_API(OpenRead, IFileStream, const AuString &path, EFileAdvisoryLockLevel successPostAdvisoryLevel = EFileAdvisoryLockLevel::eBlockWrite);
-    AUKN_SHARED_API(OpenWrite, IFileStream, const AuString &path);
+    AUKN_SHARED_API(OpenWrite, IFileStream, const AuString &path, EFileAdvisoryLockLevel successPostAdvisoryLevel = EFileAdvisoryLockLevel::eBlockReadWrite);
+
+    AUKN_SHARED_API(Open, IFileStream, const AuString &path, EFileOpenMode mode = EFileOpenMode::eRead, EFileAdvisoryLockLevel successPostAdvisoryLevel = EFileAdvisoryLockLevel::eBlockReadWrite);
 }

Include/Aurora/IO/FS/FileWriter.hpp

@@ -9,27 +9,26 @@
 
 namespace Aurora::IO::FS
 {
-    class FileWriter : public IStreamWriter
+    struct FileWriter : public IStreamWriter
     {
-    public:
         AU_NO_COPY_NO_MOVE(FileWriter)
         
-        FileWriter() {}
+        inline FileWriter() {}
-        ~FileWriter()  {}
+        inline ~FileWriter()  {}
 
         template<typename... T>
-        bool OpenFile(T... args)
+        inline bool OpenFile(T... args)
         {
             stream_ = OpenWriteUnique(args...);
             return stream_ != nullptr;
         }
 
-        virtual EStreamError Open() override
+        inline virtual EStreamError Open() override
         {
             return stream_ ? EStreamError::eErrorNone : EStreamError::eErrorStreamNotOpen;
         }
 
-        virtual EStreamError Write(const Memory::MemoryViewStreamRead & parameters) override
+        inline virtual EStreamError Write(const Memory::MemoryViewStreamRead &parameters) override
         {
             if (!stream_) return EStreamError::eErrorStreamNotOpen;
             if (!stream_->Write(parameters)) return EStreamError::eErrorStreamInterrupted;
@@ -38,16 +37,26 @@ namespace Aurora::IO::FS
             return EStreamError::eErrorNone;
         }
 
-        virtual void Flush() override
+        inline virtual void Flush() override
         {
             if (stream_) stream_->Flush();
         }
 
-        virtual void Close() override
+        inline virtual void Close() override
         {
             stream_.reset();
         }
 
+        inline virtual AuUInt64 GetPosition()
+        {
+            return stream_ ? stream_->GetOffset() : 0;
+        }
+
+        inline virtual bool SetPosition(AuUInt64 pos)
+        {
+            return stream_ ? stream_->SetOffset(pos) : false;
+        }
+
     private:
         OpenWriteUnique_t stream_{};
     };

Include/Aurora/IO/FS/IAsyncFileStream.hpp

@@ -0,0 +1,18 @@
+/***
+    Copyright (C) 2022 J Reece Wilson (a/k/a "Reece"). All rights reserved.
+
+    File: IAsyncFileStream.hpp
+    Date: 2022-2-14
+    Author: Reece
+***/
+#pragma once
+
+namespace Aurora::IO::FS
+{
+    struct IAsyncTransaction;
+    
+    struct IAsyncFileStream
+    {
+        virtual AuSPtr<IAsyncTransaction> NewTransaction() = 0;
+    };
+}

Include/Aurora/IO/FS/IAsyncFinishedSubscriber.hpp

@@ -0,0 +1,15 @@
+/***
+    Copyright (C) 2022 J Reece Wilson (a/k/a "Reece"). All rights reserved.
+
+    File: IAsyncFinishedSubscriber.hpp
+    Date: 2022-2-14
+    Author: Reece
+***/
+#pragma once
+
+namespace Aurora::IO::FS
+{
+    AUKN_INTERFACE(IAsyncFinishedSubscriber, 
+        AUI_METHOD(void, OnAsyncFileOpFinished, (AuUInt64, offset, AuUInt32, length))
+    );
+}

Include/Aurora/IO/FS/IAsyncTransaction.hpp

@@ -0,0 +1,30 @@
+/***
+    Copyright (C) 2022 J Reece Wilson (a/k/a "Reece"). All rights reserved.
+
+    File: IAsyncTransaction.hpp
+    Date: 2022-2-14
+    Author: Reece
+***/
+#pragma once
+
+namespace Aurora::Loop
+{
+    struct ILoopSource;
+}
+
+namespace Aurora::IO::FS
+{
+    struct IAsyncTransaction
+    {
+        virtual bool StartRead(AuUInt64 offset, const AuSPtr<Memory::MemoryViewWrite> &memoryView) = 0;
+        virtual bool StartWrite(AuUInt64 offset, const AuSPtr<Memory::MemoryViewRead> &memoryView) = 0;
+        
+        virtual bool     Complete() = 0;
+        virtual AuUInt32 GetLastPacketLength() = 0;
+        
+        virtual void SetCallback(const AuSPtr<IAsyncFinishedSubscriber> &sub) = 0;
+
+        virtual bool Wait(AuUInt32 timeout) = 0;
+        virtual AuSPtr<Loop::ILoopSource> NewLoopSource() = 0;
+    };
+}

Include/Aurora/IO/FS/IFileStream.hpp

@@ -1,17 +1,64 @@
+/***
+    Copyright (C) 2021 J Reece Wilson (a/k/a "Reece"). All rights reserved.
+
+    File: IFileStream.hpp
+    Date: 2022-01-17 (wrong)
+    Author: Reece
+***/
 #pragma once
 
 namespace Aurora::IO::FS
 {
-    class IFileStream
+    struct IFileStream
     {
-    public:
+        /**
-        virtual AuUInt64 GetOffset() = 0;
+         * @brief Read memoryview from the filestream at an auto-incrementing offset
-        virtual bool     SetOffset(AuUInt64 offset) = 0;
+         * @param parameters 
-        virtual AuUInt64 GetLength() = 0;
+         * @return 
+        */
         virtual bool     Read(const Memory::MemoryViewStreamWrite &parameters) = 0;
+
+        /**
+         * @brief Writes memoryview to the filestream at an auto-incrementing offset
+         * @param parameters
+         * @return 
+        */
         virtual bool     Write(const Memory::MemoryViewStreamRead &parameters) = 0;
-        virtual void     Close() = 0;
+
+        /**
+         * @brief Returns the length of the file
+         * @return
+        */
+        virtual AuUInt64 GetLength() = 0;
+
+        /**
+         * @brief Returns the offset of the internal streams
+         * @return 
+        */
+        virtual AuUInt64 GetOffset() = 0;
+
+        /**
+         * @brief Updates the stream pointer
+         * @param offset 
+         * @return 
+        */
+        virtual bool     SetOffset(AuUInt64 offset) = 0;
+
+        /**
+         * @brief Flush read/write streams
+        */
         virtual void     Flush() = 0;
+
+        /**
+         * @brief Termiantes/truncates the file at the current offset
+        */
         virtual void     WriteEoS() = 0;
+
+
+        /**
+         * @brief Close IFileStream resource
+        */
+        virtual void     Close() = 0;
+
     };
 }

Include/Aurora/IO/FS/Resources.hpp

@@ -15,15 +15,29 @@ namespace Aurora::IO::FS
     AUKN_SYM bool GetSystemDomain(AuString &path);
 
     /**
-     * @brief Provides an application specific storage path for local application data, isolated for your Aurora application brand info (defer to the init structure)
+     * @brief Provides an application specific storage path for user-local application data, isolated with respect to your Aurora application brand info (defer to the init structure)
     */
     AUKN_SYM bool GetProfileDomain(AuString &path);
 
 
     AUKN_SYM bool GetSystemResourcePath(const AuString &fileName, AuString &path);
     
+    /**
+     * @brief Get package path
+     *        This could be a read-only path if relevant to the platform
+     *        This may be `/storage/emulated/0/Android/data/` (`Context#getExternalFilesDir(java.lang.String)`)
+     *                    `/Applications/<mac bundle>` 
+     *                    `\\some uwp path`
+     *        ...on platforms with read-only application specific data directories
+     * @param path 
+     * @return 
+    */
+    AUKN_SYM bool GetPackagePath(AuString &path);
+
     /**
      * @brief Pulls the application directory as defined by the operating system standard file system hierarchy
+     *        Otherwise, XDG_CONFIG_CONFIG or %appdata%.
+     *        This directory might be sandboxed to the user or application by the operating system.
     */
     AUKN_SYM bool GetAppData(AuString &path);
 
@@ -33,7 +47,9 @@ namespace Aurora::IO::FS
     AUKN_SYM bool GetUserHome(AuString &path);
 
     /**
-     * @brief Global application data that requires no special permissions to access
+     * @brief Global application data that requires no special permissions to access.
+     *        This could be an SD-Card or user profile.
+     *        This could be the root of the user managable aplication directory.
     */
     AUKN_SYM bool GetWritableAppdata(AuString &path);
 
@@ -41,4 +57,9 @@ namespace Aurora::IO::FS
      * @brief Global application data that requires special permissions to access, usually configured by a system account during installation
     */
     AUKN_SYM bool GetRootAppdata(AuString &path);
+
+    /**
+     * @brief Get user installable application directory
+    */
+    AUKN_SYM bool GetUserProgramsFolder(AuString &path);
 }

Include/Aurora/IO/IO.hpp

@@ -11,9 +11,10 @@
 
 #include "IStreamReader.hpp"
 #include "IStreamWriter.hpp"
-#include "IArbitraryStreamer.hpp" // arbitrary read stream, dunno what you would want to call thousands_sep
+#include "ISeekingReader.hpp" // arbitrary read stream, dunno what you would want to call thousands_sep
 
 #include "Buffered/Buffered.hpp"
 
 #include "FS/FS.hpp"
 #include "Net/Net.hpp"
+#include "Character/Character.hpp"

Include/Aurora/IO/ISeekingReader.hpp

@@ -1,7 +1,7 @@
 /***
     Copyright (C) 2021 J Reece Wilson (a/k/a "Reece"). All rights reserved.
 
-    File: IArbitraryStreamer.hpp
+    File: ISeekingReader.hpp
     Date: 2021-6-10
     Author: Reece
 ***/
@@ -9,11 +9,10 @@
 
 namespace Aurora::IO
 {
-    class IArbitraryStreamer
+    struct ISeekingReader
     {
-    public:
         virtual EStreamError Open() = 0;
-        virtual EStreamError ArbitraryRead(AuUInt32 offset, const Memory::MemoryViewStreamWrite &paramters) = 0;
+        virtual EStreamError ArbitraryRead(AuUInt offset, const Memory::MemoryViewStreamWrite &paramters) = 0;
         virtual void         Close() = 0;
     };
 }

Include/Aurora/IO/IStreamReader.hpp

@@ -9,47 +9,17 @@
 
 namespace Aurora::IO
 {
-    class IStreamReader
+    // TODO (Reece): Consider publishing the multiple inverted ext hack for Aurora Interfaces for binding this. 
+    // It would be nice to have cs-style extensions on this interface. Maybe I shouldn't double down
+    // over a double blocking API. Maybe I should focus on the stream pumpers.
+    struct IStreamReader
     {
-    public:
         virtual EStreamError Open() = 0;
         virtual EStreamError Read(const Memory::MemoryViewStreamWrite &paramters) = 0;
         virtual void         Close() = 0;
 
-        EStreamError ReadAll(Memory::ByteBuffer &buffer)
+        inline EStreamError  ReadAll(Memory::ByteBuffer &buffer);
-        {
-            static const int kBufferSize = 2048;
-            
-            AuUInt len;
-            EStreamError ret;
-            AuUInt8 temp[kBufferSize];
-
-            len = kBufferSize;
-            ret = EStreamError::eErrorEndOfStream;
-
-            buffer = Memory::NewResizableBuffer();
-
-            while ((ret = Read(Memory::MemoryViewStreamWrite(temp, len))) == EStreamError::eErrorNone)
-            {
-                if (len == 0)
-                {
-                    break;
-                }
-
-                buffer.Write(temp, len);
-
-                if (len != kBufferSize)
-                {
-                    break;
-                }
-            }
-
-            if (ret == EStreamError::eErrorEndOfStream)
-            {
-                return EStreamError::eErrorNone;
-            }
-
-            return ret;
-        }
     };
 }
+
+#include "IStreamReader.inl" // ReadAll utility

Include/Aurora/IO/IStreamReader.inl

@@ -0,0 +1,47 @@
+/***
+    Copyright (C) 2022 J Reece Wilson (a/k/a "Reece"). All rights reserved.
+
+    File: IStreamReader.inl
+    Date: 2022-2-1
+    Author: Reece
+***/
+#pragma once
+
+namespace Aurora::IO
+{
+    EStreamError IStreamReader::ReadAll(Memory::ByteBuffer &buffer)
+    {
+        static const int kBufferSize = 2048;
+        
+        AuUInt len;
+        EStreamError ret;
+        AuUInt8 temp[kBufferSize];
+
+        len = kBufferSize;
+        ret = EStreamError::eErrorEndOfStream;
+
+        buffer = Memory::NewResizableBuffer();
+
+        while ((ret = Read(Memory::MemoryViewStreamWrite(temp, len))) == EStreamError::eErrorNone)
+        {
+            if (len == 0)
+            {
+                break;
+            }
+
+            buffer.Write(temp, len);
+
+            if (len != kBufferSize)
+            {
+                break;
+            }
+        }
+
+        if (ret == EStreamError::eErrorEndOfStream)
+        {
+            return EStreamError::eErrorNone;
+        }
+
+        return ret;
+    }
+}

Include/Aurora/IO/IStreamWriter.hpp

@@ -9,9 +9,8 @@
 
 namespace Aurora::IO
 {
-    class IStreamWriter
+    struct IStreamWriter
     {
-    public:
         virtual EStreamError Open() = 0;
         virtual EStreamError Write(const Memory::MemoryViewStreamRead &parameters) = 0;
         virtual void         Flush() = 0;

Include/Aurora/IO/Net/Net.hpp

@@ -14,11 +14,11 @@ namespace Aurora::Async
 
 namespace Aurora::IO::Net
 {
-    static const AuUInt16 kMagicPortAny = 65535;
+    static const AuUInt16 kMagicPortAny = 0;
 
     struct INetworkStream;
     struct IBasicSocket;
-    struct IClientSocket;
+    struct ISocket;
     struct IServer;
 
     AUE_DEFINE(ETransportProtocol, (
@@ -60,11 +60,11 @@ namespace Aurora::IO::Net
 
             if (cmp.ip == EIPProtocol::eIPProtocolV4)
             {
-                return memcmp(cmp.v4, this->v4, sizeof(this->v4)) == 0;
+                return AuMemcmp(cmp.v4, this->v4, sizeof(this->v4)) == 0;
             }
             else
             {
-                return memcmp(cmp.v6, this->v6, sizeof(this->v6)) == 0;
+                return AuMemcmp(cmp.v6, this->v6, sizeof(this->v6)) == 0;
             }
         }
     };
@@ -104,11 +104,13 @@ namespace Aurora::IO::Net
     {
         IPAddress ip;
         AuUInt16 port;
+        AuUInt8 hint[32] {0};
     };
 
     struct ConnectionEndpoint
     {
         ETransportProtocol protocol;
+        IPEndpoint ip;
         bool tls {};
         bool compressed {};
         // 0 - destination is a stateless datagram server
@@ -138,18 +140,18 @@ namespace Aurora::IO::Net
 
     AUKN_INTERFACE(IClientSubscriber,
                    // 
-                   AUI_METHOD(void, OnServerConnectSuccess, (const AuSPtr<IClientSocket> &, socket)),
+                   AUI_METHOD(void, OnServerConnectSuccess, (const AuSPtr<ISocket> &, socket)),
-                   AUI_METHOD(void, OnServerConnectFailed,  (const AuSPtr<IClientSocket> &, socket)),
+                   AUI_METHOD(void, OnServerConnectFailed,  (const AuSPtr<ISocket> &, socket)),
                    
                    // DTLS/UDP/TCP/TLS -> TRUE  = expects another datagram or read pump
                    //                     FALSE = end of socket life
-                   AUI_METHOD(bool, OnSockeData,            (const AuSPtr<IClientSocket> &, socket)),
+                   AUI_METHOD(bool, OnSocketData,           (const AuSPtr<ISocket> &, socket)),
 
                    // 
-                   AUI_METHOD(void, OnSocketError,          (const AuSPtr<IClientSocket> &, socket)),
+                   AUI_METHOD(void, OnSocketError,          (const AuSPtr<ISocket> &, socket)),
 
                    // 
-                   AUI_METHOD(void, OnSocketShutdown,       (const AuSPtr<IClientSocket> &, socket))
+                   AUI_METHOD(void, OnSocketShutdown,       (const AuSPtr<ISocket> &, socket))
     );
 
     AUKN_INTERFACE(IClientSubscriberTls,
@@ -158,16 +160,16 @@ namespace Aurora::IO::Net
     );
 
     AUKN_INTERFACE(IServerSubscriber,
-                   AUI_METHOD(bool, OnClientAccept,    (const AuSPtr<IServer> &, server, const AuSPtr<IClientSocket> &, socket)),
+                   AUI_METHOD(bool, OnClientAccept,    (const AuSPtr<IServer> &, server, const AuSPtr<ISocket> &, socket)),
-                   AUI_METHOD(bool, OnClientDoS,       (const AuSPtr<IServer> &, server, const AuSPtr<IClientSocket> &, socket)),
+                   AUI_METHOD(bool, OnClientDoS,       (const AuSPtr<IServer> &, server, const AuSPtr<ISocket> &, socket)),
-                   AUI_METHOD(void, OnClientError,     (const AuSPtr<IServer> &, server, const AuSPtr<IClientSocket> &, socket)),
+                   AUI_METHOD(void, OnClientError,     (const AuSPtr<IServer> &, server, const AuSPtr<ISocket> &, socket)),
-                   AUI_METHOD(void, OnClientShutdown,  (const AuSPtr<IServer> &, server, const AuSPtr<IClientSocket> &, socket)),
+                   AUI_METHOD(void, OnClientShutdown,  (const AuSPtr<IServer> &, server, const AuSPtr<ISocket> &, socket)),
-                   AUI_METHOD(bool, OnReadFrame,       (const AuSPtr<IServer> &, server, const AuList<AuSPtr<IClientSocket>> &, sockets)),
+                   AUI_METHOD(bool, OnReadFrame,       (const AuSPtr<IServer> &, server, const AuList<AuSPtr<ISocket>> &, sockets)),
                    AUI_METHOD(void, OnShutdown,        (const AuSPtr<IServer> &, server))
     );
     
     AUKN_INTERFACE(IServerSubscriberTls,
-                   AUI_METHOD(bool, OnClientTLSReport, (const AuSPtr<IServer> &, server, const AuSPtr<IClientSocket> &, socket, const TLSHandshakeError &, error))
+                   AUI_METHOD(bool, OnClientTLSReport, (const AuSPtr<IServer> &, server, const AuSPtr<ISocket> &, socket, const TLSHandshakeError &, error))
     );
 
     // TODO: We should introduce another std:: customer overloadable type reproducing hardcoded ascii and an int, basically std::error_code 
@@ -235,19 +237,8 @@ namespace Aurora::IO::Net
         AuUInt32 bufferedWriteSize {4096 * 10}; // see: enableBufferedOutput
     };
 
-
+    struct ISocketChannel
-    struct IClientSocket : public IBasicSocket
     {
-        virtual bool     GetRemoteEndpoint(ConnectionEndpoint &out) = 0;
-        virtual bool     GetLocalEndpoint(ConnectionEndpoint &out) = 0;
-
-        virtual bool     PumpRead() = 0;
-        virtual bool     PumpWrite() = 0;
-        virtual bool     Pump() = 0;
-
-        virtual void     Run(int idx, AuUInt32 timeout) = 0;
-
-        
         // If memory.ptr is a nullptr, this method immediately returns with the expected write length in memory.out
         // 
         // If all is true and the internal buffer is not saturated enough yet, no data is read and
@@ -333,7 +324,21 @@ namespace Aurora::IO::Net
         virtual void     ReconfigureStreams(const StreamConfig &config) = 0;
     };
 
-    struct ILocalClientSocket : public IClientSocket
+    struct IBasicSocketThreaded : public IBasicSocket
+    {
+        virtual bool     PumpRead() = 0;
+        virtual bool     PumpWrite() = 0;
+        virtual bool     Pump() = 0;
+
+        virtual void     Run(int idx, AuUInt32 timeout) = 0;
+    };
+
+    struct ISocket : public IBasicSocketThreaded, public ISocketChannel
+    {
+        virtual bool     GetRemoteEndpoint(ConnectionEndpoint &out) = 0;
+    };
+
+    struct ILocalClientSocket : public ISocket
     {
         // Connects to the endpoint defined in the ClientConfig 
         // Completion will be notified by the following callbacks;
@@ -389,9 +394,10 @@ namespace Aurora::IO::Net
         AuSPtr<IClientSubscriberTls> clientSubscriber;
     };
 
-    struct IServer : public IBasicSocket
+    struct IServer : public IBasicSocketThreaded
     {
-        virtual void GetClients(AuList<AuSPtr<IClientSocket>> &clients) = 0;
+        virtual bool GetLocalEndpoint(ConnectionEndpoint &out) = 0;
+        virtual void GetClients(AuList<AuSPtr<IBasicSocketThreaded>> &clients) = 0;
         virtual bool Listen() = 0;
         virtual void ReconfigureDefaultStream(const StreamConfig &config) = 0;
     };
@@ -414,10 +420,11 @@ namespace Aurora::IO::Net
 
     struct INetworkInterface
     {
-        virtual IPEndpoint ResolveSocketSync(const SocketHostName &hostname, AuUInt16 port);
+        virtual AuList<IPEndpoint> ResolveSocketSync(const SocketHostName &hostname, AuUInt16 port) = 0;
-        virtual IPEndpoint ResolveServiceSync(const ServiceEndpoint &service);
+        virtual AuList<IPEndpoint> ResolveServiceSync(const ServiceEndpoint &service) = 0;
 
-        virtual bool SendDatagramAsync(const ConnectionEndpoint &endpoint, const Memory::MemoryViewRead &memory);
+        virtual bool SendDatagramAsync(const ConnectionEndpoint &endpoint, const Memory::MemoryViewRead &memory) = 0;
+        virtual bool SendDatagramAsync(const AuSPtr<IServer> &datagramServer, const ConnectionEndpoint &endpoint, const Memory::MemoryViewRead &memory) = 0;
 
         virtual AuSPtr<ISocketFactory> GetSocketFactory() = 0;
     };
@@ -453,12 +460,14 @@ namespace Aurora::IO::Net
         virtual AuUInt32 PollWorker(AuUInt8 workerId) = 0;
         virtual AuUInt32 RunWorker(AuUInt8 workerId, AuUInt32 timeout) = 0;
 
-
+        // D: 
-        // D: class APIs
         virtual bool BeginReadPollingOnWorkQueues(const WorkPoolGroup &workGroup) = 0;
         virtual bool BeginSubmissionsOnOnWorkQueues(const WorkPoolGroup &workGroup) = 0;
         virtual void StopPollingOnWorkQueues() = 0;
 
+        // E:
+        // .. 
+
         virtual AuUInt8 GetWorkers() = 0;
 
         virtual AuSPtr<INetworkInterface> GetNetworkInterface() = 0;
@@ -466,7 +475,6 @@ namespace Aurora::IO::Net
         virtual void Shutdown() = 0;
     };
 
-
     struct NetworkPool
     {
         AuUInt8 workers {1};

Include/Aurora/Locale/ECodePage.hpp

@@ -9,21 +9,35 @@
 
 namespace Aurora::Locale
 {
-    enum class ECodePage
+    AUE_DEFINE(ECodePage, (
-    {
-        eUnsupported,
         eUTF32,
         eUTF32BE,
+
+        
         eUTF16,
         eUTF16BE,
+
+        // Extended 6 byte UTF8, capable of encoding disallowed characters in recent specs
         eUTF8,
+
+        // Barely supported
         eUTF7,
+
+        // GB2312 -> (Microsofts non-standard) GBK <-> GB 18030
+        // GBK is the relevant subset of GB 18030, implemented by MS and iconv
+        // GB2312 is not so relevant
+        // GB 18030 is a backwards compatible modern spec 
         e18030,
         e2312,
         eGBK,
+
+        // SHIFT_JIS
         eSJIS,
+
+        // Latin-x
         eLatin1,
-        eSysUnk,
+
-        eMax = eUnsupported
+        // System Codepage
-    };
+        eSysUnk
+    ));
 }

Include/Aurora/Locale/Encoding/Encoding.hpp

@@ -19,11 +19,12 @@ namespace Aurora::Locale::Encoding
     AUKN_SYM BOM DecodeBOM(const Memory::MemoryViewRead &binary);
 
     // General purpose arbitrary page to UTF8 (AuStrings are UTF-8 - not 16 or 32; bite me)
-    AUKN_SYM AuStreamReadWrittenPair_t EncodeUTF8(const Memory::MemoryViewRead &utf8, const Memory::MemoryViewWrite &binary, ECodePage page = ECodePage::eUnsupported);
+    AUKN_SYM AuStreamReadWrittenPair_t EncodeUTF8(const Memory::MemoryViewRead &utf8, const Memory::MemoryViewWrite &binary, ECodePage page = ECodePage::eEnumInvalid);
-    AUKN_SYM AuStreamReadWrittenPair_t DecodeUTF8(const Memory::MemoryViewRead &binary, const Memory::MemoryViewWrite &utf8, ECodePage page = ECodePage::eUnsupported);
+    AUKN_SYM AuStreamReadWrittenPair_t DecodeUTF8(const Memory::MemoryViewRead &binary, const Memory::MemoryViewWrite &utf8, ECodePage page = ECodePage::eEnumInvalid);
-    AUKN_SYM AuStreamReadWrittenPair_t DecodeUTF8(const Memory::MemoryViewRead &binary, AuString &out, ECodePage page = ECodePage::eUnsupported);
+    AUKN_SYM AuStreamReadWrittenPair_t DecodeUTF8(const Memory::MemoryViewRead &binary, AuString &out, ECodePage page = ECodePage::eEnumInvalid);
     
     // Optimized UTF translation functions
+    // Note: these functions support full 6-byte UTF8 encoding. 5&6 are reserved.
     AUKN_SYM AuStreamReadWrittenPair_t ReadUTF32IntoUTF8ByteString(const Memory::MemoryViewRead &utf32, const Memory::MemoryViewWrite &utf8);
     AUKN_SYM AuStreamReadWrittenPair_t ReadUTF8IntoUTF32ByteString(const Memory::MemoryViewRead &utf8, const Memory::MemoryViewWrite &utf32);
 
@@ -37,4 +38,7 @@ namespace Aurora::Locale::Encoding
     AUKN_SYM AuUInt32 CountUTF8Length (const Memory::MemoryViewRead &utf8, bool bytes = false); // codepoint = U32 encoded; at most: 6 bytes per codepoint
     AUKN_SYM AuUInt32 CountSJISLength (const Memory::MemoryViewRead &sjis, bool bytes = false); // codepoint = one character
     AUKN_SYM AuUInt32 CountGBK16Length(const Memory::MemoryViewRead &gbk, bool bytes = false); // codepoint = at most; one GBK byte pair
+
+
+    AUKN_SYM AuUInt32 CountEncodedStringLength(ECodePage page, const Memory::MemoryViewRead &view, bool bytes = false);
 }

Include/Aurora/Locale/Locale.hpp

@@ -20,7 +20,7 @@ namespace Aurora::Locale
         const AuString &language; /// ISO 639
         const AuString &country;  /// ISO 3166
         const AuString &codeset;  /// 
-        const ECodePage codepage; /// Potentially eSysUnk; noting that eSysUnk is valid and handlable by the internal backend
+        const ECodePage codepage; /// Potentially eSysUnk. Note that eSysUnk is valid and handlable by iconv and nsl. On windows and unix, this is your codepage.
     };
 
     /*

Include/Aurora/Locale/LocaleStrings.hpp

@@ -1,3 +1,10 @@
+/***
+    Copyright (C) 2022 J Reece Wilson (a/k/a "Reece"). All rights reserved.
+
+    File: LocaleStrings.hpp
+    Date: 2022-1-24
+    Author: Reece
+***/
 #pragma once
 
 #include <Aurora/Time/ETimezoneShift.hpp>
@@ -10,10 +17,11 @@ namespace Aurora::Locale
     AUKN_SYM const AuString &NewLine();
 
     AUKN_SYM const AuString &TimeLocaleGetMSChar();
-    AUKN_SYM const AuString &TimeLocaleS();
+    AUKN_SYM const AuString &TimeLocaleGetSChar();
     AUKN_SYM const AuString &TimeLocaleGetDayChar();
 
     AUKN_SYM AuString TimeDateToString(const Time::tm &time);
+    AUKN_SYM AuString TimeDateToFileNameISO8601(const Time::tm &time, Time::ETimezoneShift shift = Time::ETimezoneShift::eUTC);
     AUKN_SYM AuString TimeDateToISO8601(const Time::tm &time, Time::ETimezoneShift shift = Time::ETimezoneShift::eUTC);
 
     AUKN_SYM AuString ConvertMSToTimescale(AuUInt32 ms);

Include/Aurora/Logging/IBasicSink.hpp

@@ -0,0 +1,25 @@
+/***
+    Copyright (C) 2021 J Reece Wilson (a/k/a "Reece"). All rights reserved.
+
+    File: IBasicSink.hpp
+    Date: 2021-11-1
+    Author: Reece
+***/
+#pragma once
+
+namespace Aurora::Logging
+{
+    AUKN_INTERFACE(IBasicSink,
+
+        // async callback
+        AUI_METHOD(void, OnMessageBlocking, (AuUInt8, level, const ConsoleMessage &, msg)),
+
+        // sync callback
+        // returns true if async aware, that is, OnMessageBlocking will be scheduled
+        AUI_METHOD(bool, OnMessageNonblocking, (AuUInt8, level, const ConsoleMessage &, msg)),
+
+        // called every configurable MS or during a problematic event warrenting flush
+        // expect onflush to be called once after a series of OnMessageNonblocking and every N milliseconds (^1)
+        AUI_METHOD(void, OnFlush, ())
+    )
+}

Include/Aurora/Logging/IBasicSinkRB.hpp

@@ -0,0 +1,25 @@
+/***
+    Copyright (C) 2021 J Reece Wilson (a/k/a "Reece"). All rights reserved.
+
+    File: IBasicSinkRB.hpp
+    Date: 2021-11-2
+    Author: Reece
+***/
+#pragma once
+
+namespace Aurora::Logging
+{
+    struct IBasicSinkRB : IBasicSink
+    {
+        /**
+         * @brief Writes the log buffer to a file as though it were an FIO sink
+        */
+        virtual void SaveToPath(const AuString &path, bool binary = false) = 0;
+
+        /**
+         * @brief Dumps the ring buffer to an ordered list 
+         *        This method does not clear the ring buffer
+        */
+        virtual AuList<ConsoleMessage> Export() = 0;
+    };
+}

Include/Aurora/Logging/ILogger.hpp

@@ -7,21 +7,24 @@
 ***/
 #pragma once 
 
-namespace Aurora::Console::Logging
+namespace Aurora::Logging
 {
-    enum class ELogLevel
+    using ConsoleMessage = Console::ConsoleMessage;
-    {
+    using EAnsiColor = Console::EAnsiColor;
-        eZero = 0,
+
-        eInfo = 0,
+    AUE_DEFINE(ELogLevel,
+    (
+        eInfo,
         eVerbose,
         eError,
         eDebug,
         eWarn
-    };
+    ));
+    
+    static auto const kLogLevelDefault = static_cast<AuUInt8>(kELogLevelMinLegal);
+    static auto const kLogLevelUsr     = static_cast<AuUInt8>(kELogLevelMaxLegal);
+    static auto const kLogLevelMax     = AuUInt8(255);
 
-    static auto const kLogLevelDefault = static_cast<AuUInt8>(ELogLevel::eZero);
-    static auto const kLogLevelUsr     = static_cast<AuUInt8>(ELogLevel::eWarn);
-    static auto const kLogLevelMax     = 16;
     struct ILogger
     {
         virtual void WriteMessage(AuUInt8 level, const ConsoleMessage &msg) = 0;

Include/Aurora/Logging/Logging.hpp

@@ -12,8 +12,19 @@
 #include "IBasicSinkRB.hpp"
 #include "Sinks.hpp"
 
-namespace Aurora::Console::Logging
+namespace Aurora::Logging
 {
+    /// Writes a log message to the console subscribers and telemetry sinks
+    AUKN_SYM void WriteLine(AuUInt8 level, const Console::ConsoleMessage &msg);
+
+    /**
+     * @brief Overloads the ILogger backend of the AuLogXX functions
+     * @param defaultGlobalLogger 
+     * @return 
+    */
+    AUKN_SYM void SetGlobalLogger(const AuSPtr<Logging::ILogger> &defaultGlobalLogger);
+
+
 #if defined(_AUHAS_FMT)
 
     template<typename ... T>
@@ -134,7 +145,7 @@ namespace Aurora::Console::Logging
     template<typename ... T> \
     static void AuLog ## level(T&& ... args) \
     { \
-        Aurora::Console::Logging::Log ## level(AuForward<T>(args)...); \
+        Aurora::Logging::Log ## level(AuForward<T>(args)...); \
     }
 
 ADD_AU_GLOBAL_ALIAS(Info)
@@ -149,3 +160,5 @@ ADD_AU_GLOBAL_ALIAS(Verbose)
 #else
     #define AuLogVerboseNoShip(...)
 #endif
+
+#undef ADD_AU_GLOBAL_ALIAS

Include/Aurora/Logging/Sinks.hpp

@@ -0,0 +1,61 @@
+/***
+    Copyright (C) 2021 J Reece Wilson (a/k/a "Reece"). All rights reserved.
+
+    File: Sinks.hpp
+    Date: 2021-11-2
+    Author: Reece
+***/
+#pragma once 
+
+namespace Aurora
+{
+    struct SocketConsole;
+}
+
+namespace Aurora::Logging
+{
+    struct DirectoryLogger
+    {
+        AuUInt32 maxLogsOrZero {};
+        AuUInt32 maxFileSizeOrZero {}; // MB
+    };
+
+    /**
+     * @brief Constructs a UTF8 output sink. 
+     *        Backed by stdconsole; supports posix fd stdin/out localized, visual studio debugger, and conhost
+    */
+    AUKN_SHARED_API(NewStdSink, IBasicSink);
+
+    /**
+     * @brief Unimplemented systemd or service (?) backend
+    */
+    AUKN_SHARED_API(NewOSEventDirectorySink, IBasicSink);
+
+    /**
+     * @brief Constructs a new sink backed by, syslog or eventlog, by name or event source respectively
+    */
+    AUKN_SHARED_API(NewOSNamedEventDirectorySink, IBasicSink, const AuString &name);
+    
+    /**
+     * @brief Constructs a text of binary log file sink
+    */
+    AUKN_SHARED_API(NewFileSink, IBasicSink, const AuString &path, bool binary = false);
+
+    /**
+     * @brief Constructs a dedicated log directory subject to erasure as defined by defined DirectoryLogger 
+    */
+    AUKN_SHARED_API(NewDirectorySink, IBasicSink, const AuString &path, DirectoryLogger dirInfo, bool binary = false);
+
+    // TODO:
+    AUKN_SHARED_API(NewIPCSink,  IBasicSink, const SocketConsole &console);
+
+    /**
+     * @brief Constructs an in-memory ring buffer sink
+    */
+    AUKN_SHARED_API(NewRingLogger, IBasicSinkRB, AuUInt32 maxLogEntries);
+
+    /**
+     * @brief Constructs a logger object from an array of sinks
+    */
+    AUKN_SHARED_API(NewLogger, ILogger, const AuList<AuSPtr<IBasicSink>> &sinks);
+}

Include/Aurora/Loop/ELoopSource.hpp

@@ -0,0 +1,41 @@
+/***
+    Copyright (C) 2022 J Reece Wilson (a/k/a "Reece"). All rights reserved.
+
+    File: ELoopSource.hpp
+    Date: 2022-2-9
+    Author: Reece
+***/
+#pragma once
+
+namespace Aurora::Loop
+{
+    enum class ELoopSource
+    {
+        eSourceInternalReserved1,
+        eSourceInternalReserved2,
+        eSourceInternalReserved3,
+        eSourceInternalReserved4,
+        
+        // generic
+        eSourceSemaphore,
+        eSourceCV,
+        eSourceEvent,
+        eSourceMutex,
+        eSourceSRW,
+        eSourceSocketGroup,
+        eSourceTimer,
+        eSourceAIO,
+        eSourceHandle,
+        
+        // Specific to the runtime subsystem
+        eSourceAsync,
+        
+        // glib oses only
+        eSourceGlib,
+        
+        // window messge loops
+        eSourceApple,
+        eSourceX11,
+        eSourceWin32
+    };
+}

Include/Aurora/Loop/ILoopQueue.hpp

@@ -0,0 +1,186 @@
+/***
+    Copyright (C) 2022 J Reece Wilson (a/k/a "Reece"). All rights reserved.
+
+    File: ILoopQueue.hpp
+    Date: 2022-2-12
+    Author: Reece
+***/
+#pragma once
+
+namespace Aurora::Loop
+{   
+    // This is like a CF RunLoop with the input parameters of NT's WaitMultipleObjects
+    // This shouldn't be too much heavier than CF's libevent/NT abstraction in the style of a kevent interface with objects
+
+    struct ILoopQueue
+    {
+        /**
+         * @brief 
+         *        [*] Source add/remove operations should be followed by a Commit() to commit the changes
+         * 
+         *        [*] Calling Source[Add/Remove] during a Wait[Any/All[Ex]] will cause the wait function to
+         *             return false, allowing for the caller to rewait on the updated queue, or to allow
+         *             the thread to account for the object-was-invalidated condition
+         * 
+         * @param source 
+         * @return 
+        */
+        virtual bool SourceAdd(const AuSPtr<ILoopSource> &source) = 0;
+
+        /**
+         * @brief Same behaviour as SourceAdd
+         * @param source 
+         * @param timeoutMS 
+         * @return 
+        */
+        virtual bool SourceAddWithTimeout(const AuSPtr<ILoopSource> &source, AuUInt32 timeoutMS) = 0;
+
+        /**
+         * @brief Remove operations include commit. You do not need to follow a remove with a commit.
+         *        Sources are defacto poped unless subscriber returns false indicating repeated lock attempts are wanted.
+         *        Should no subscriber be registered, the loop source will not be automatically removed  
+         * @param source 
+        */
+        virtual bool SourceRemove(const AuSPtr<ILoopSource> &source) = 0;
+
+        /**
+         * @brief Updates the OS watchdog list cache concept after Source[Remove/Add[WithTimeout]]
+         * @return
+        */
+        virtual bool Commit() = 0;
+
+        /**
+         * @brief 
+         * @return the amount of loop sources added to the queue 
+        */
+        virtual AuUInt32 GetSourceCount() = 0;
+
+        /**
+         * @brief Appends a callback handler to be paired with a loop source
+         * 
+         *        Once a loop source become signaled, and a Wait[Any/All] function call is made, loop source subscribers are called
+         *         to filter the final should-remove operation. The final popped objects are returned from Wait[Any[Ex]] and forgotten.
+         * 
+         *        If no ILoopSourceSubscribers are registered, the edge case behaviour is that loop sources will not be automatically
+         *         dequeued. No user code explicitly told us to do anything other than wait, so the default behaviour of Wait[Any[Ex]]
+         *         shall be to not evict loop sources, until at least one callback is registered, or until SourceRemove is called.
+         * 
+         *        Returning true from a ILoopSourceSubscriber will allow the ILoopQueue to evict the loop source
+         *        Returning false from a ILoopSourceSubscriber will prevent eviction, resulting in continued ILoopSourceSubscriber callbacks to all subscribers
+         *        
+         * @param source 
+         * @param subscriber 
+         * @return 
+        */
+        virtual bool AddCallback(const AuSPtr<ILoopSource> &source, const AuSPtr<ILoopSourceSubscriber> &subscriber) = 0;
+
+        /**
+         * @brief Identical behaviour to that of the basic method. In addition, ILoopSourceSubscriberEx::OnTimeout is called
+         *        on violation of the timeout provided to SourceAddWithTimeout 
+         * @param source 
+         * @param subscriber 
+         * @return 
+        */
+        virtual bool AddCallbackEx(const AuSPtr<ILoopSource> &source, const AuSPtr<ILoopSourceSubscriberEx> &subscriber) = 0;
+
+        /**
+         * @brief Near identical behaviour of the extended AddCallback method.
+         *        Registers a callback to handle all loop source signaled events.
+         * @param subscriber 
+         * @return 
+        */
+        virtual bool AddCallback(const AuSPtr<ILoopSourceSubscriber> &subscriber) = 0;
+
+        /**
+         * @brief Indicates whether or not all the loop sources are in a signaled state and all 
+         *        relevant interfaces, such as internal IPC handlers and ILoopSourceSubscribers,
+         *        have received the update. Merely waiting for all loop sources to finish is not
+         *        sufficient. One must process the signaled states by calling a Wait function
+         *        when the HasAnyPending() condition is valid.
+         * @return 
+        */
+        virtual bool HasFinished() = 0;
+
+        /**
+         * @brief Nonblocking wait-any for all objects in the loop queue
+         * @return 
+        */
+        virtual bool IsSignaled() = 0;
+        
+        /**
+         * @brief Waits on all the submitted loop sources until they are all complete or until the timeout has finished.
+         *        Note: the completion of another Wait[All/Any[Ex]] call may result in a 
+         * @param timeout 
+         * @return 
+        */
+        virtual bool                        WaitAll  (AuUInt32 timeout = 0) = 0;
+
+        /**
+         * @brief Waits on all the loop sources until at least one is signaled
+         * @param timeout 
+         * @return 
+        */
+        virtual AuUInt32                    WaitAny  (AuUInt32 timeout = 0) = 0;
+
+        /**
+         * @brief 
+         * @param timeout 
+         * @return 
+        */
+        virtual AuList<AuSPtr<ILoopSource>> WaitAnyEx(AuUInt32 timeout = 0) = 0;
+        
+        /**
+         * @brief 
+         *        Mostly relevant on Windows where the limit is 64 versus horrible BSD-y socket
+         *        optimized interfaces that can handle thousands of fds
+         * 
+         *        In large loop queues, a maximum of
+         *          `sectionTickTime * (sectionDequeCount + clamp(platformSanity, sectionDequeCount))`
+         *          kernel objects can be checked within a given time frame.
+         * 
+         *         What this effectively does is iterate over a subdivided work queue, aligned
+         *          to platformSanity, [i, i+platform sanity] objects can be safely waitany on.
+         * 
+         *         On timeout of sectionTickTime,
+         *          [i, i+sectionDequeCount] of the tick could be checked, however, we know
+         *          [i, i+platform sanity] timeouted so, in the event of a timeout, we can
+         *          asynchronously check [i+platform sanity, ...] to be nicer faster. Since
+         *          we've already spent the maximum allocated time to poll on a mere subset,
+         *          we may as well desperately check sectionDequeCount worth of objects, and
+         *          write-off the time spent blocking. That way, we get upto platform sanity
+         *          of is signaled checks for free in the worst case scenario. The perspective
+         *          that the overshoot is expensive doesn't make sense, when the is signaled
+         *          check is basically free and the timeout on a subset of the req was the
+         *          bottleneck.
+         * 
+         *         Otherwise on success, we know to check in the range of [i, platform sanity
+         *          for at least one alerted object.
+         * 
+         *         Looking anywhere else given any help from a waitany interface would be stupid.
+         * 
+         *         Internal logic will dequeue platform coefficient aligned handles, and the timeout
+         *         optimization shall check [i+platform sanity, i+platform sanity+sectionDequeCount].
+         *         1 handle =/= 1 loop source, tho usually 1 loop source ~= 1 handle
+         * 
+         *  tl'dr: 
+         *         [*] dont use if you cant stall for sectionTickTime if sectionDequeCount of your
+         *             loop sources are unsignaled
+         *         [*] chug paths are intended for observer threads and not for maintaining high 
+         *             performance. no sane traditional application would ever have 64+ signalable
+         *             objects stuck in a loop. though if you do, and you dont configure anything,
+         *             the worst case scenario is, WaitAny can take upto a millisecond to discover
+         *             a signaled loop source.
+         *       
+         * @param sectionTickTime 
+         * @param sectionDequeCount
+        */
+        virtual void ChugPathConfigure(AuUInt32 sectionTickTime, AuSInt sectionDequeCount) = 0;
+
+        /**
+         * @brief Hints that the calling program understands the kernel shouldnt schedule tne entire source list, and instead, we should long poll
+        */
+        virtual void ChugHint(bool value) = 0;
+    };
+
+    AUKN_SYM AuSPtr<ILoopQueue> NewLoopQueue();
+}

Include/Aurora/Loop/ILoopSource.hpp

@@ -0,0 +1,26 @@
+/***
+    Copyright (C) 2022 J Reece Wilson (a/k/a "Reece"). All rights reserved.
+
+    File: ILoopSource.hpp
+    Date: 2022-2-12
+    Author: Reece
+***/
+#pragma once
+
+namespace Aurora::Loop
+{
+    struct ILoopSource
+    {
+        /**
+         * @brief Atomic is-signaled-and-latch
+         * @return 
+        */
+        virtual bool IsSignaled() = 0;
+        
+        /**
+         * @brief Returns a generic description about the loop source handle
+         * @return 
+        */
+        virtual ELoopSource GetType() = 0;
+    };
+}

Include/Aurora/Loop/ILoopSourceSubscriber.hpp

@@ -0,0 +1,36 @@
+/***
+    Copyright (C) 2022 J Reece Wilson (a/k/a "Reece"). All rights reserved.
+
+    File: ILoopSourceSubscriber.hpp
+    Date: 2022-2-12
+    Author: Reece
+***/
+#pragma once
+
+namespace Aurora::Loop
+{
+    AUKN_INTERFACE(ILoopSourceSubscriber,
+        /**
+         * @brief called under Wait[All/Any[Ex]] once an object is signaled
+         * @return should pop from ILoopQueue 
+        */
+        AUI_METHOD(bool, OnFinished, (const AuSPtr<ILoopSource> &, source))
+    );
+
+    
+    AUKN_INTERFACE(ILoopSourceSubscriberEx,
+        /**
+         * @brief called under Wait[All/Any[Ex]] once an object is signaled
+         * @return should pop from ILoopQueue 
+        */
+        AUI_METHOD(bool, OnFinished, (const AuSPtr<ILoopSource> &, source)),
+
+        /**
+         * @brief called under Wait[All/Any[Ex]] once an object timesout before being evicted
+         * @param  
+         * @param AUI_METHOD 
+         * @param AUI_METHOD 
+        */
+        AUI_METHOD(void, OnTimeout, (const AuSPtr<ILoopSource> &, source))
+    );
+}

Include/Aurora/Loop/Loop.hpp

@@ -4,91 +4,63 @@
     File: Loop.hpp
     Date: 2021-8-21
     Author: Reece
+    Notes: This API class is specifically for kernel objects or similar userland schedular handles, this is not comparable to the async subsystem.
+           While you may drive low perf user facing apps and small services from this, other services should divide and conquer
+           * Allow the network subsystem to load balance sockets across a predefined amount of workers,
+           * Use semaphores instead of massive lengthy arrays of mutexes
+           * Use grouped long-polling LoopSources to convert kernel or waitable objects to a single time-polled loop source
 ***/
 #pragma once
 
+#include "ELoopSource.hpp"
+#include "ILoopSource.hpp"
+#include "ILoopSourceSubscriber.hpp"
+#include "ILoopQueue.hpp"
+
 namespace Aurora::Loop
 {    
-    enum class ELoopSource
+    /// @deprecated
-    {
-        eSourceInternalReserved1,
-        eSourceInternalReserved2,
-        eSourceInternalReserved3,
-        eSourceInternalReserved4,
-        
-        // generic
-        eSourceSemaphore,
-        eSourceCV,
-        eSourceEvent,
-        eSourceMutex,
-        eSourceSRW,
-        eSourceTimer,
-        eSourceAIO,
-        eSourceHandle,
-        
-        // Specific to the runtime subsystem
-        eSourceAsync,
-        
-        // glib oses only
-        eSourceGlib,
-        
-        // window messge loops
-        eSourceApple,
-        eSourceX11,
-        eSourceWin32
-    };
-
-    class ILoopSource
-    {
-    public:
-        virtual bool IsSignaled() = 0;
-        virtual ELoopSource GetType() = 0;
-    };
-    
     AUKN_SYM AuList<AuSPtr<ILoopSource>> WaitMultipleOrObjects(const AuList<AuSPtr<ILoopSource>> &objects, AuUInt32 timeout);
     
-    class ILSSemaphore : public ILoopSource
+    struct ILSSemaphore : public ILoopSource
     {
-    public:
         virtual bool AddOne() = 0;
     };
     
-    class ILSEvent : public ILoopSource
+    struct ILSEvent : public ILoopSource
     {
-    public:
         virtual bool Set() = 0;
         virtual bool Reset() = 0;
     };
     
-    class ILSMutex : public ILoopSource
+    struct ILSMutex : public ILoopSource
     {
-    public:
         virtual bool Unlock() = 0;
     };
     
-    class IConditionVar : public ILoopSource
+    struct IConditionVar : public ILoopSource
     {
-    public:
         virtual bool Signal() = 0;
         virtual bool Broadcast() = 0;
     };
     
-    class ITimer : public ILoopSource
+    struct ITimer : public ILoopSource
     {
-    public:
         virtual void UpdateTime(AuUInt64 absTimeMs) = 0;
+        virtual void Stop() = 0;
     };
     
     AUKN_SYM AuSPtr<IConditionVar>  NewLSCondVar(const AuSPtr<Threading::IWaitable> &primitive); 
     AUKN_SYM AuSPtr<IConditionVar>  NewLSCondVar(const AuSPtr<ILSMutex> &source); 
     
-    AUKN_SYM AuSPtr<ITimer>         NewLSTimer(AuUInt64 absTimeMs); 
+    AUKN_SYM AuSPtr<ITimer>         NewLSTimer(AuUInt64 absTimeMs, AuUInt32 reschedMs = 0); 
     AUKN_SYM AuSPtr<ILSMutex>       NewLSMutex();
     AUKN_SYM AuSPtr<ILSEvent>       NewLSEvent(bool triggerd = false, bool atomicRelease = true, bool permitMultipleTriggers = false);
     AUKN_SYM AuSPtr<ILSSemaphore>   NewLSSemaphore(AuUInt32 initialCount = 0);
     AUKN_SYM AuSPtr<ILoopSource>    NewLSOSHandle(AuUInt);
-    AUKN_SYM AuSPtr<ILoopSource>    NewLSAsync();
+    AUKN_SYM AuSPtr<ILoopSource>    NewLSAsync(Async::WorkerPId_t workerPid);
-    AUKN_SYM AuSPtr<ILoopSource>    NewLSWin32Source();
+    AUKN_SYM AuSPtr<ILoopSource>    NewLSFile(const AuSPtr<IO::FS::IAsyncTransaction> &fileTransaction);
+    AUKN_SYM AuSPtr<ILoopSource>    NewLSWin32Source(bool dispatchMessages);
     AUKN_SYM AuSPtr<ILoopSource>    NewLSAppleSource();
     
     #if defined(X_PROTOCOL)
@@ -114,3 +86,4 @@ namespace Aurora::Loop
     }
     #endif
 }
+

Include/Aurora/Memory/ByteBuffer.hpp

@@ -7,33 +7,45 @@
 ***/
 #pragma once
 
+#include <Aurora/Locale/ECodePage.hpp>
+
 namespace Aurora::Memory
 {
+    enum class EStringType
+    {
+        eStringTerminated,
+        eStringByte,
+        eStringWord,
+        eStringDword, 
+        eStringQword
+    };
+
     static const auto kBufferPageSize  = 512;
     //static const auto kBufferBasePower = 8;
     static const auto kBufferInitialPower = 9;// -kBufferBasePower; // 4-bit integer
 
     /***
-     * A bytebuffer object represents a linear, partially-linear resizable, buffer **or** a ring buffer. 
+     * A bytebuffer object represents a exponentially page-scale resizable buffer **or** a ring buffer. 
+     
+     * Trivial, linear, use-once serialization use cases will likely follow the linear fast paths, not that of a
+     *  ring buffer.
      *
      * Use cases for a ring buffer include wrapping streams for a use case in which the consumer may
      * expect arbitrary stream seeks of an otherwise limited consume-once stream
      * 
-     *  IE;
+     *  EG:
      *  -> Peeking a header in a datagram, or tcp stream; where instead of freeing the datagram or double
      *       buffering the network stack when required, a ring buffer is used to prevent reallocation on each frame
-     *  -> Peeking, or seeking back after, compression read. A compression api could be fed on-Sdemand or ad hoc,
+     *  -> Peeking, or seeking back after, compression read. A compression api could be fed on-known-input-available 
-     *       writing to its write head pointer, while never running out of space so long as the decompressed ring 
+     *       or on-demand, writing to its write head pointer, while never running out of space so long as the 
-    *        read head continues moving
+     *       decompressed ring read head continues moving
-     *
-     * Small, linear, write/read-once [de]serialization use cases may elect to allocate a buffer and 
-     *  follow the linear fast paths; perhaps even enabling flagExpandable for hopefully-smarter-than-stdvec-scaling 
      *
+     * Writing:
      *     Ring buffers scale from the write head, to the read head, potentially going-around in the process
      *     
-     * Linear flagExpandable buffers scale from [0, length]; reallocating at end of buffer if flagExpandable is enabled
+     *     Linear flagExpandable buffers scale from [0, length]
      *     if expanding is enabled, 
-     *     realloc(max(size + offset, (offset / kBufferPageSize + 1) * kBufferPageSize))
+     *        ~~realloc(max(size + offset, (offset / kBufferPageSize + [1 or 2]) * kBufferPageSize))~~  wrong, this was too slow for large streams, high write count cases
      *
      * Deprecates INetworkStream, fixes allocation issues around compression backends
      * Superseeds abuse of AuList<AuUInt8> for binary blobs, alongside Memory::Array
@@ -72,6 +84,10 @@ namespace Aurora::Memory
         AuUInt8 scaleSize;// : 4; screw it.... we should just take 6 * (4/8) up to 32/64, we wont go up a slab allocation bucket, whatever you want to call it
         ///////////////////////////////////////////////////////////////////////
         
+        /**
+         * @brief Move constructor
+         * @param buffer 
+        */
         ByteBuffer(ByteBuffer &&buffer)
         {
             this->base            = buffer.base;
@@ -92,6 +108,11 @@ namespace Aurora::Memory
             buffer.scaleSize      = {};
         }
         
+        /**
+         * @brief Copy with possible preserve pointers
+         * @param buffer 
+         * @param preservePointers 
+        */
         ByteBuffer(const ByteBuffer &buffer, bool preservePointers = true)
         {
             this->base      = FAlloc<AuUInt8 *>(buffer.length);
@@ -118,6 +139,13 @@ namespace Aurora::Memory
             this->scaleSize = buffer.scaleSize;
         }
 
+        /**
+         * @brief Copy pointer range into a new ring or byte buffer
+         * @param in 
+         * @param length 
+         * @param circular 
+         * @param expandable 
+        */
         ByteBuffer(const void *in, AuUInt length, bool circular = false, bool expandable = false) : flagCircular(circular), flagExpandable(expandable), flagReadError(0), flagWriteError(0)
         {
             this->base      = FAlloc<AuUInt8 *>(length);
@@ -183,11 +211,14 @@ namespace Aurora::Memory
             this->length    = length;
             this->allocSize = length;
             this->readPtr   = this->base;
-            this->writePtr = this->base;
+            this->writePtr  = this->base + length;
             this->scaleSize = kBufferInitialPower;
             AuMemcpy(this->base, base, length);
         }
         
+        /**
+         * @brief Default constructor, allocates an auto-expanding linear bytebuffer
+        */
         ByteBuffer() : flagCircular(0), flagExpandable(true), flagReadError(0), flagWriteError(0)
         {
             this->base      = {};
@@ -214,7 +245,7 @@ namespace Aurora::Memory
             this->writePtr       = base;
         }
         
-        // Iterator
+        // utils: Iterator
         inline auline AuUInt8 * data()  const;
         inline auline AuUInt    size()  const;
         inline auline AuUInt8 * begin() const;
@@ -224,29 +255,30 @@ namespace Aurora::Memory
         inline void resize(AuUInt size);
         inline void reserve(AuUInt size);
         
-        // Utils To alternative types 
+        // utils: Utils to alternative types 
         inline auline AuList<AuUInt8> ToVector() const;
 
         inline AuUInt32 GetAllocationPower() const;
         inline operator AuList<AuUInt8>() const;
         inline operator MemoryViewRead()  const;
 
-        // Internal buffer comparison
+        // utils: Internal buffer comparison
         inline bool operator ==(const AuList<AuUInt8> &) const;
         inline bool operator ==(const MemoryViewRead &) const;
         inline bool operator ==(const ByteBuffer &) const;
 
-        // Move assignment
+        // utils: Move assignment
         inline ByteBuffer &operator =(ByteBuffer &&);
 
-        // &byteArray[n]
+        // utils: &byteArray[n]
         inline AuUInt8 &operator [](AuUInt idx) const;
         
-        // if (byteArray) -> if (byteArray->IsValid())
+        // utils: if (byteArray) -> if (byteArray->IsValid())
         inline operator bool() const;
 
         inline AuList<AuUInt8> RemainingBytesToVector(bool endAtWrite = true) const;
-
+        // ... utils are mostly const functions that provide language intrinsics, access, and container-like compatibility
+        // ByteBuffer specific utils can be found under Utilities
         
         // Seek / Position
         inline auline bool ReaderTryGoForward(AuUInt32 offset);
@@ -261,6 +293,7 @@ namespace Aurora::Memory
         inline auline AuUInt GetReadOffset() const;
         inline auline AuUInt GetWriteOffset() const;
 
+        inline auline void ResetReadPointer();
         
         inline AuOptional<AuUInt8 *> WriterTryGetWriteHeadFor(AuUInt32 nBytes);
 
@@ -270,13 +303,53 @@ namespace Aurora::Memory
         inline auline bool SetBuffer(const void *in, AuUInt length);
         inline auline bool SetBuffer(const AuList<AuUInt8> &buffer);
 
+        /**
+         * @brief Releases excess memory (like, shrink to fit in c++)
+         * @return 
+        */
         inline auline void GC();
+
+        /**
+         * @brief Releases all resources and resets the bytebuffer without an allocation
+        */
         inline void Reset();
+
+        /**
+         * @brief Expands the underlying buffer allocation to at least length.
+         *        Does nothing on failure. Programs can try to allocate in real time
+         *        and handle the write error flag condition from there. To pull the
+         *        real buffer size, see member field allocSize.
+         * @param length 
+        */
         inline void Reserve(AuUInt length);
+
+        /**
+         * @brief Is allocated or dummy object?
+         * @return 
+        */
         inline auline bool IsEmpty() const;
+
+        /**
+         * @brief Is an error flag set?
+         * @return 
+        */
         inline auline bool HasStreamError() const;
+        
+        /**
+         * @brief Returns true so long as
+         *        1) an error has not occured or,
+         *        1) the relevant error flag was reset; and
+         *        2) there is a valid underlying buffer
+         * 
+         * @return 
+        */
         inline auline bool IsValid() const;
 
+        /**
+         * @brief Allocate at least length bytes, without adjusting the relative read/write head offsets
+         * @param length 
+         * @return 
+        */
         inline auline bool Resize(AuUInt length);
 
         // Basic Read Write
@@ -284,6 +357,19 @@ namespace Aurora::Memory
         inline auline AuUInt Write(const void *buffer, AuUInt requestLength);
         inline auline AuUInt Read(void *out, AuUInt requestedLength, bool peek = false);
 
+        // String API
+
+        inline bool WriteString(const AuString &string, EStringType type, Locale::ECodePage codepage = Locale::ECodePage::eUTF8);
+        inline bool ReadString(AuString &string, EStringType type, Locale::ECodePage codepage = Locale::ECodePage::eUTF8);
+
+        // Copy, concat, etc 
+        inline bool WriteFrom(ByteBuffer &buffer, AuUInt length);
+
+        // Utilities
+        inline bool Trim(AuUInt tail);
+        inline bool Pad(AuUInt16 aPowOf2, AuUInt8 magicCharacter = '\x00');
+        inline bool Fill(AuUInt length, AuUInt8 magicCharacter = '\x00');
+
         // Typed read/write
         template<typename T>
         T Read();

Include/Aurora/Memory/ByteBufferPushReadState.hpp

@@ -0,0 +1,39 @@
+/***
+    Copyright (C) 2022 J Reece Wilson (a/k/a "Reece"). All rights reserved.
+
+    File: ByteBufferPushReadState.hpp
+    Date: 2022-2-15
+    Author: Reece
+***/
+#pragma once
+
+namespace Aurora::Memory
+{
+    struct ByteBufferPushReadState
+    {
+        ByteBuffer &bytebuffer;
+
+        ByteBufferPushReadState(ByteBuffer &bytebuffer, bool pushError = false) : bytebuffer(bytebuffer), shouldPopError_(pushError)
+        {
+            this->readPos_ = this->bytebuffer.readPtr - this->bytebuffer.base;
+            this->readErrorFlag_ = bytebuffer.flagReadError;
+        }
+
+        ~ByteBufferPushReadState()
+        {
+            if (this->readErrorFlag_ != bytebuffer.flagReadError)
+            {
+                if (this->shouldPopError_)
+                {
+                    this->bytebuffer.flagReadError = this->readErrorFlag_;
+                }
+                this->bytebuffer.readPtr = this->bytebuffer.base + this->readPos_;
+            }
+        }
+
+    private:
+        bool readErrorFlag_;
+        AuUInt readPos_;
+        const bool shouldPopError_;
+    };
+}

Include/Aurora/Memory/ByteBufferPushWriteState.hpp

@@ -0,0 +1,39 @@
+/***
+    Copyright (C) 2022 J Reece Wilson (a/k/a "Reece"). All rights reserved.
+
+    File: ByteBufferPushWriteState.hpp
+    Date: 2022-2-15
+    Author: Reece
+***/
+#pragma once
+
+namespace Aurora::Memory
+{
+    struct ByteBufferPushWriteState
+    {
+        ByteBuffer &bytebuffer;
+
+        ByteBufferPushWriteState(ByteBuffer &bytebuffer, bool pushError = false) : bytebuffer(bytebuffer), shouldPopError_(pushError)
+        {
+            this->writePos_ = this->bytebuffer.writePtr - this->bytebuffer.base;
+            this->writeErrorFlag_ = bytebuffer.flagWriteError;
+        }
+
+        ~ByteBufferPushWriteState()
+        {
+            if (this->writeErrorFlag_ != bytebuffer.flagWriteError)
+            {
+                if (this->shouldPopError_)
+                {
+                    this->bytebuffer.flagWriteError = this->writeErrorFlag_;
+                }
+                this->bytebuffer.writePtr = this->bytebuffer.base + this->writePos_;
+            }
+        }
+
+    private:
+        bool writeErrorFlag_;
+        AuUInt writePos_;
+        const bool shouldPopError_;
+    };
+}

Include/Aurora/Memory/ByteBuffer_Memory.inl

@@ -49,6 +49,7 @@ namespace Aurora::Memory
         if (this->base)
         {
             Free(this->base);
+            this->base = {};
         }
 
         this->base = {};
@@ -62,16 +63,32 @@ namespace Aurora::Memory
     void ByteBuffer::Reserve(AuUInt length)
     {
         auto oldLength = this->length;
+        if (length > this->allocSize)
+        {
             this->Resize(length);
-        this->length = AuMin(oldLength, length);
+        }
+        this->length = AuMin(oldLength, this->length);
     }
 
     void ByteBuffer::GC()
     {
-        if (this->length)
+        if (!this->base)
-        if (this->allocSize == this->length) return;
+        {
-        auto temp = Memory::FRealloc(this->base, this->length);
+            return;
-        if (!temp) return;
+        }
+
+        if (this->allocSize == this->length)
+        {
+            return;
+        }
+
+        auto temp = Memory::ZRealloc(this->base, this->length);
+        
+        if (!temp)
+        {
+            return;
+        }
+
         this->base = temp;
         this->length = this->length;
         this->allocSize = this->length;
@@ -84,7 +101,9 @@ namespace Aurora::Memory
 
         if (length == 0)
         {
-            Reset();
+            this->length = 0;
+            this->writePtr = this->base;
+            this->readPtr = this->base;
             return true;
         }
 
@@ -109,7 +128,7 @@ namespace Aurora::Memory
         {
             auto scale = GetAllocationPower();
             oldLength = this->length;
-            newLength = AuMax(AuUInt(length), AuUInt(((this->allocSize / scale) + 1) * scale));
+            newLength = AuMax(AuUInt(length), (((this->allocSize * 2) / scale) + 1) * scale);// AuUInt(((this->allocSize / scale) + 2) * scale));  < this spends too much time spinning
 
             nextPtr = ZRealloc(this->base, newLength);
             if (!nextPtr)

Include/Aurora/Memory/ByteBuffer_Position.inl

@@ -11,19 +11,37 @@ namespace Aurora::Memory
 {
     bool ByteBuffer::WriterTryGoForward(AuUInt32 offset)
     {
+        if (!offset)
+        {
+            return true;
+        }
+
+        auto old = writePtr - base;
         auto n = Write(nullptr, offset);
-        SysAssert(n == offset); // TODO: writer go back
+        if (n != offset)
+        {
+            this->writePtr = this->base + old;
+            return false;
+        }
+
         return true;
     }
 
     bool ByteBuffer::ReaderTryGoForward(AuUInt32 offset)
     {
+        if (!offset)
+        {
+            return true;
+        }
+
+        auto old = readPtr - base;
         auto n = Read(nullptr, offset, false);
         if (n != offset)
         {
-            SysAssert(ReaderTryGoBack(n));
+            this->readPtr = this->base + old;
             return false;
         }
+
         return true;
     }
 
@@ -70,12 +88,12 @@ namespace Aurora::Memory
 
             if (readOffset < offset)
             {
-                auto absPosition = offset - readOffset;
+                auto absPosition = (AuUInt)offset - readOffset;
                 auto goAround = absPosition;
                 auto backIdx = length - goAround;
 
                 auto writeOffset = writePtr - base;
-                if (writeOffset > backIdx)
+                if ((AuUInt)writeOffset > (AuUInt)backIdx)
                 {
                     return false;
                 }
@@ -118,7 +136,7 @@ namespace Aurora::Memory
         {
             if ((readPtr < writePtr) && (endAtWrite))
             {
-                return length - (writePtr - readPtr);
+                return writePtr - readPtr;
             }
             else
             {
@@ -180,6 +198,11 @@ namespace Aurora::Memory
         return true;
     }
 
+    void ByteBuffer::ResetReadPointer()
+    {
+        readPtr = base;
+    }
+
     AuUInt ByteBuffer::GetReadOffset() const
     {
         if (flagCircular)

Include/Aurora/Memory/ByteBuffer_ReadWrite.inl

@@ -66,11 +66,22 @@ namespace Aurora::Memory
         else
         {
             auto offset = writePtr - base;
+
+            if ((AuUInt)offset > (AuUInt)length)
+            {
+                SysPushErrorIO();
+                this->flagWriteError = true;
+                return 0;
+            }
+
             auto overhead = length - offset;
 
             AuUInt len = AuMin(overhead, requestLength);
 
-            if ((len != requestLength) && (flagExpandable))
+            if ((len != requestLength))
+            {
+
+                if ((flagExpandable))
                 {
                     if (!Resize(offset + requestLength))
                     {
@@ -80,6 +91,11 @@ namespace Aurora::Memory
                     overhead = length - offset;
                     len = AuMin(overhead, requestLength);
                 }
+                else
+                {
+                    if (len == 0) return 0;
+                }
+            }
 
             if (buffer)
             {
@@ -87,11 +103,11 @@ namespace Aurora::Memory
             }
 
             writePtr += len;
+            SysAssert(writePtr <= base + length);
             return len;
         }
     }
 
-
     AuUInt ByteBuffer::Read(void *out, AuUInt requestedLength, bool peek)
     {
         AuUInt linearOverhead = 0, toWriteOverhead = 0, linearReadable = 0, toWriteReadable = 0;
@@ -124,8 +140,11 @@ namespace Aurora::Memory
             }
 
             if (toWriteOverhead)
+            {
+                if (out)
                 {
                     AuMemcpy(reinterpret_cast<AuUInt8 *>(out) + linearOverhead, base, toWriteReadable);
+                }
 
                 if (!peek)
                 {
@@ -144,6 +163,13 @@ namespace Aurora::Memory
         }
         else
         {
+            if (readPtr > writePtr)
+            {
+                SysPushErrorIO();
+                this->flagReadError = true;
+                return 0;
+            }
+
             AuUInt len = AuMin(AuUInt(writePtr - readPtr), requestedLength);
 
             if (out)
@@ -159,5 +185,4 @@ namespace Aurora::Memory
             return len;
         }
     }
-
 }

Include/Aurora/Memory/ByteBuffer_Strings.inl

@@ -0,0 +1,137 @@
+/***
+    Copyright (C) 2022 J Reece Wilson (a/k/a "Reece"). All rights reserved.
+
+    File: ByteBuffer_Strings.inl
+    Date: 2022-2-15
+    Author: Reece
+***/
+#pragma once
+
+namespace Aurora::Memory
+{
+    bool ByteBuffer::WriteString(const AuString &string, EStringType type, Locale::ECodePage codepage)
+    {
+        ByteBufferPushWriteState a(*this);
+        AuStreamReadWrittenPair_t len {};
+        Memory::ByteBuffer temp;
+
+        if (codepage != Locale::ECodePage::eUTF8)
+        {
+            len = Aurora::Locale::Encoding::EncodeUTF8(string, {}, codepage);
+            if (len == AuStreamReadWrittenPair_t {0, 0})
+            {
+                return false;
+            }
+
+            temp = Memory::ByteBuffer(len.second);
+            len = Aurora::Locale::Encoding::EncodeUTF8(string, Memory::MemoryViewStreamWrite(temp.begin(), temp.end()), codepage);
+            if (len == AuStreamReadWrittenPair_t {0, 0})
+            {
+                return false;
+            }
+        }
+        else
+        {
+            len = AuStreamReadWrittenPair_t {AuUInt32(string.size()), AuUInt32(string.size())};
+        }
+     
+        switch (type)
+        {
+        case EStringType::eStringByte:
+        {
+            if (len.second > 255)
+            {
+                return {};
+            }
+
+            if (!Write<AuUInt8>(static_cast<AuUInt8>(len.second)))
+            {
+                return {};
+            }
+            break;
+        }
+        case EStringType::eStringWord:
+        {
+            if (len.second > std::numeric_limits<AuUInt16>::max())
+            {
+                return {};
+            }
+
+            if (!Write<AuUInt16>(static_cast<AuUInt16>(len.second)))
+            {
+                return {};
+            }
+            break;
+        }
+        case EStringType::eStringDword:
+        {
+            if (len.second > std::numeric_limits<AuUInt32>::max())
+            {
+                return {};
+            }
+
+            if (!Write<AuUInt32>(static_cast<AuUInt32>(len.second)))
+            {
+                return {};
+            }
+            break;
+        }
+        case EStringType::eStringQword:
+        {
+            if (!Write<AuUInt64>(static_cast<AuUInt64>(len.second)))
+            {
+                return {};
+            }
+            break;
+        }
+        }
+
+        if (codepage != Locale::ECodePage::eUTF8)
+        {
+            if (!Write(temp.base, temp.length))
+            {
+                return {};
+            }
+        }
+        else
+        {
+            if (!Write(string.data(), string.size()))
+            {
+                return {};
+            }
+        }
+
+        if (type == EStringType::eStringTerminated)
+        {
+            if (codepage == Locale::ECodePage::eUTF16 || codepage == Locale::ECodePage::eUTF16BE)
+            {
+                if (!Write<AuUInt16>(0))
+                {
+                    return {};
+                }
+            }
+            else if (codepage == Locale::ECodePage::eUTF32 || codepage == Locale::ECodePage::eUTF32BE)
+            {
+                if (!Write<AuUInt32>(0))
+                {
+                    return {};
+                }
+            }
+            else
+            {
+                if (!Write<AuUInt8>(0))
+                {
+                    return {};
+                }
+            }
+        }
+
+        return true;
+    }
+
+    bool ByteBuffer::ReadString(AuString &string, EStringType type, Locale::ECodePage codepage)
+    {
+        // TODO: ...
+        return {};
+    }
+}

Include/Aurora/Memory/ByteBuffer_Utility.inl

@@ -0,0 +1,79 @@
+/***
+    Copyright (C) 2022 J Reece Wilson (a/k/a "Reece"). All rights reserved.
+
+    File: ByteBuffer_Utility.inl
+    Date: 2022-2-15
+    Author: Reece
+***/
+#pragma once
+
+namespace Aurora::Memory
+{
+    bool ByteBuffer::Trim(AuUInt tail)
+    {
+        if (this->length > tail)
+        {
+            return false;
+        }
+
+        return Resize(this->length - tail);
+    }
+
+    bool ByteBuffer::Fill(AuUInt length, AuUInt8 magicCharacter)
+    {
+        ByteBufferPushWriteState a(*this);
+
+        auto maxLinear = this->writePtr - AuMin(this->readPtr, this->base + this->length);
+        maxLinear = AuMin(AuUInt(maxLinear), length);
+
+        if (maxLinear)
+        {
+            AuMemset(this->writePtr, magicCharacter, maxLinear);
+            this->writePtr += maxLinear;
+        }
+
+        if (maxLinear != length)
+        {
+            auto delta = length - maxLinear;
+
+            // HACK: 
+            Memory::ByteBuffer inner(delta);
+            if (!inner)
+            {
+                return {};
+            }
+
+            AuMemset(inner.base, magicCharacter, inner.length);
+        
+            if (!Write(inner.base, inner.length))
+            {
+                return {};
+            }
+        }
+
+        return true;
+    }
+
+    bool ByteBuffer::Pad(AuUInt16 aPowOf2, AuUInt8 magicCharacter)
+    {
+        ByteBufferPushWriteState a(*this);
+
+        auto oldLen = this->length;
+        auto newLen = ((this->length + aPowOf2) & ~(aPowOf2 - 1));
+        
+        if (!Resize(newLen))
+        {
+            return {};
+        }
+
+        auto old = this->writePtr - this->base;
+        this->writePtr = this->base + oldLen;
+        if (!this->Fill(newLen - oldLen))
+        {
+            return {};
+        }
+
+        this->writePtr = this->base + old;
+        return true;
+    }
+}

Include/Aurora/Memory/ByteBuffer_Utils.inl

@@ -41,6 +41,7 @@ namespace Aurora::Memory
 
     AuUInt8 &ByteBuffer::operator [](AuUInt idx) const
     {
+        SysAssert(idx < length);
         return *(data() + idx);
     }
 
@@ -119,7 +120,6 @@ namespace Aurora::Memory
         return !IsEmpty() && !HasStreamError();
     }
 
-
     bool ByteBuffer::IsEmpty() const
     {
         return !length || !base;

Include/Aurora/Memory/ByteBuffer_WriteFrom.inl

@@ -0,0 +1,16 @@
+/***
+    Copyright (C) 2022 J Reece Wilson (a/k/a "Reece"). All rights reserved.
+
+    File: ByteBuffer_WriteFrom.inl
+    Date: 2022-2-15
+    Author: Reece
+***/
+#pragma once
+
+namespace Aurora::Memory
+{
+    bool ByteBuffer::WriteFrom(ByteBuffer &buffer, AuUInt length)
+    {
+        return {};
+    }
+}

Include/Aurora/Memory/ExtendStlLikeSharedPtr.hpp

@@ -35,25 +35,29 @@ namespace Aurora::Memory
             return *this;
         }
 
-        TType_t &operator*() const
+        template<typename TType2_t = TType_t>
+        TType2_t &operator*() const
         {
             throwif();
             return Base_t::operator*();
         }
 
-        TType_t *operator->() const
+        template<typename TType2_t = TType_t>
+        TType2_t *operator->() const
         {
             throwif();
             return Base_t::operator->();
         }
 
-        TType_t &operator*()
+        template<typename TType2_t = TType_t>
+        TType2_t &operator*()
         {
             throwif();
             return Base_t::operator*();
         }
 
-        TType_t *operator->()
+        template<typename TType2_t = TType_t>
+        TType2_t *operator->()
         {
             throwif();
             return Base_t::operator->();

Include/Aurora/Memory/Memory.hpp

@@ -83,7 +83,7 @@ namespace Aurora::Memory
         {
         }
 
-        [[nodiscard]] T *allocate(std::size_t n)
+        [[nodiscard]] T *allocate(Types::size_t n)
         {
             if (auto p = (NewArray<T>(n)))
             {
@@ -93,7 +93,7 @@ namespace Aurora::Memory
             throw std::bad_alloc();
         }
 
-        void deallocate(T *p, std::size_t n) noexcept
+        void deallocate(T *p, Types::size_t n) noexcept
         {
             _Free(p);
         }
@@ -146,3 +146,5 @@ namespace Aurora::Memory
 }
 
 #include "ByteBuffer.hpp"
+#include "ByteBufferPushReadState.hpp"
+#include "ByteBufferPushWriteState.hpp"

Include/Aurora/Memory/MemoryView.hpp

@@ -18,11 +18,10 @@ namespace Aurora::Memory
         using StdArray_t = AuConditional_t<Readonly_b, const AuArray<T, Z>, AuArray<T, Z>>;
         
         /*
-            YadaYada(MemoryView(tempstring/array/etc)) should be legal, right?
+            MethodName(MemoryView(tempstring/array/etc)) should be legal, right?
         
-            ---------------------
             Temporary objects are destroyed as the last step in evaluating the full-expression that (lexically) contains the point where they were created.
-            ---------------------
+
             A full-expression is:
                 [...]
                 * an expression that is not a subexpression of another expression and that is not otherwise part of a full-expression.
@@ -103,9 +102,9 @@ namespace Aurora::Memory
         }
 
         template<typename T>
-        T *Begin() const
+        AuConditional_t<Readonly_b, const T *, T *> Begin() const
         {
-            return reinterpret_cast<T *>(ptr);
+            return reinterpret_cast<AuConditional_t<Readonly_b, const T *, T *>>(ptr);
         }
 
         template<typename T>
@@ -119,6 +118,11 @@ namespace Aurora::Memory
             return ptr && length;
         }
 
+        operator bool() const
+        {
+            return HasMemory();
+        }
+
         Void_t /*const*/ ptr;
         AuUInt /*const*/ length;
     };
@@ -169,12 +173,32 @@ namespace Aurora::Memory
             outVariable = 0;
         }
         
+        constexpr MemoryViewStream(MemoryView<Readonly_b> in) : MemoryView<Readonly_b>(in.ptr, in.length), outVariable(unused)
+        {
+            outVariable = 0;
+        }
+
+        constexpr MemoryViewStream(MemoryView<Readonly_b> in, AuUInt &len) : MemoryView<Readonly_b>(in.ptr, in.length), outVariable(len)
+        {
+            outVariable = 0;
+        }
+
         template<typename T, int Z>
         constexpr MemoryViewStream(T(&a)[Z]) : MemoryView<Readonly_b>(a), outVariable(unused)
         {
             outVariable = 0;
         }
 
+        bool HasMemory() const
+        {
+            return this->ptr && this->length;
+        }
+
+        operator bool() const
+        {
+            return HasMemory();
+        }
+
         AuUInt &outVariable;
     private:
         AuUInt unused;

Include/Aurora/Memory/_ByteBuffer.hpp

@@ -5,3 +5,6 @@
 #include "ByteBuffer_Memory.inl"
 #include "ByteBuffer_Utils.inl"
 #include "ByteBuffer_Position.inl"
+#include "ByteBuffer_Utility.inl"
+#include "ByteBuffer_Strings.inl"
+#include "ByteBuffer_WriteFrom.inl"

Include/Aurora/Parse/Base64.hpp

@@ -11,13 +11,21 @@ namespace Aurora::Parse
 {
     AUKN_SYM bool Base64Decode(const AuString &in, Memory::ByteBuffer &decoded, bool url = false);
 
-    AUKN_SYM bool Base64Encode(const void *buffer, AuMach length, AuString &encoded, bool url = false);
+    AUKN_SYM bool Base64Encode(const Memory::MemoryViewRead &input, AuString &encoded, bool url = false);
+    
     static bool Base64Encode(const AuString &in, AuString &encoded, bool url = false)
     {
-        return Base64Encode(in.data(), in.size(), encoded, url);
+        return Base64Encode(Memory::MemoryViewRead {in}, encoded, url);
     }
+
     static bool Base64Encode(const Memory::ByteBuffer &in, AuString &encoded, bool url = false)
     {
-        return Base64Encode(in.data(), in.size(), encoded, url);
+        return Base64Encode(Memory::MemoryViewRead {in}, encoded, url);
+    }
+
+    template<typename T, typename Z>
+    static bool Base64Encode(T begin, Z end, AuString &encoded, bool url = false)
+    {
+        return Base64Encode(Memory::MemoryViewRead {begin, end}, encoded, url);
     }
 }

Include/Aurora/Parse/Hex.hpp

@@ -9,14 +9,14 @@
 
 namespace Aurora::Parse
 {
-    enum class EHexDump
+    AUE_DEFINE(EHexDump,
-    {
+    (
         eString,
         eZeroXSpace,
         eHexEditor,
         eCLiteral,
         eJSLiteral
-    };
+    ));
 
     AUKN_SYM void ByteToHex(AuUInt8 val, char(&hex)[2]);
     AUKN_SYM bool HexToInt  (const char *hex, AuUInt32 length, AuUInt64 &val);

Include/Aurora/Parse/Parser.hpp

@@ -8,6 +8,10 @@
 #pragma once
 
 #include <Aurora/Data/Data.hpp>
+#include <Aurora/IO/Character/ICharacterProvider.hpp>
+#include <Aurora/IO/Character/ICharacterProviderEx.hpp>
+#include <Aurora/IO/Character/IBufferedCharacterConsumer.hpp>
+#include <Aurora/IO/Character/Providers.hpp>
 
 namespace Aurora::Parse
 {
@@ -112,6 +116,7 @@ namespace Aurora::Parse
         ParsedObject result;
     };
 
+#if 0
     using ConsumeStream_cb = AuFunction<bool(AuUInt8 &)>;
 
     static ConsumeStream_cb StringToConsumable(const AuString &str, AuMach &index)
@@ -128,17 +133,15 @@ namespace Aurora::Parse
         };
         return getc;
     }
+#endif
 
   
     struct ParseState
     {
-        ParseState(ConsumeStream_cb &str) : stringstream(str)
+        ParseState(const AuSPtr<Aurora::IO::Character::ICharacterProvider> &stream) : stream(stream)
         {}
 
-        ParseState(ConsumeStream_cb &&str) : stringstream(AuMove(str))
+        AuSPtr<Aurora::IO::Character::ICharacterProvider> stream;
-        {}
-
-        ConsumeStream_cb stringstream;
 
         AuUInt8 *additionalTokens {};
         AuUInt16 countOfTokens {};
@@ -149,31 +152,41 @@ namespace Aurora::Parse
     };
 
     AUKN_SYM void VaildateStructure(const ParseObject &object);
-    AUKN_SYM bool ConsumeToken(ParsableTag type, ConsumeStream_cb getc, ParseValue &out);
+    AUKN_SYM bool ConsumeToken(ParsableTag type, const AuSPtr<Aurora::IO::Character::ICharacterProvider> &getc, ParseValue &out);
 
     static bool ConsumeToken(ParsableTag type, const AuString &str, AuMach &index, ParseValueEx &out)
     {
-        return ConsumeToken(type, StringToConsumable(str, index), out);
+        auto strStream = IO::Character::ProviderFromStringUnique(str, index);
+        if (!strStream) return false;
+        if (!ConsumeToken(type, AuUnsafeRaiiToShared(strStream), out)) return false;
+        index = strStream->GetPosition();
+        return true;
     }
 
     static bool ConsumeToken(ParsableTag type, const AuString &str, ParseValueEx &out)
     {
-        AuMach index{};
+        auto strStream = IO::Character::ProviderFromStringUnique(str);
-        return ConsumeToken(type, StringToConsumable(str, index), out);
+        if (!strStream) return false;
+        return ConsumeToken(type, AuUnsafeRaiiToShared(strStream), out);
     }
 
     AUKN_SYM bool Parse(ParseState &state, const ParseObject &structure, ParseResult &result);
 
     static bool Parse(ParseResult &result, const ParseObject &structure, const AuString &str, AuMach &index)
     {
-        ParseState state(StringToConsumable(str, index));
+        auto strStream = IO::Character::ProviderFromStringUnique(str, index);
-        return Parse(state, structure, result);
+        if (!strStream) return false;
+        ParseState state(AuUnsafeRaiiToShared(strStream));
+        if (!Parse(state, structure, result)) return false;
+        index = strStream->GetPosition();
+        return true;
     }
 
     static bool Parse(ParseResult &result, const ParseObject &structure, const AuString &str)
     {
-        AuMach index{};
+        auto strStream = IO::Character::ProviderFromStringUnique(str, 0);
-        ParseState state(StringToConsumable(str, index));
+        if (!strStream) return false;
+        ParseState state(AuUnsafeRaiiToShared(strStream));
         return Parse(state, structure, result);
     }