[*] Re-do AuAsync reference counting
[+] IWorkItem::SetSchedSteadyTimeNsAbs
[*] Irrelevant IIOProcessor sources are now discarded in evaluating whether or not a thread-pool in special running mode should shutdown
[*] Transition WorkItems to only use steady time
[*] Refactor AsyncConfig
[*] Drop default SMT spin time from hundreds of cycles to ~32 so that we can sit nicely at the bottom of task manager unless the application calls for extra responsivity
[+] brotli now respects CompressInfo::uBlockSize
[*] refactored CompressInfo::uLz4BlockSize -> CompressInfo::uBlockSize
[*] zstd decompressor now respects user arguments
[*] zstd compressor now respects user arguments
[*] brotli decompressor now respects user arguments
[*] brotli compressor now respects user arguments
This will help us reduce cpu usage and latency at the cost of 32 bytes.
We are now hopelessly oversized: 136 bytes for a single primitive. 104 was barely passble.
[*] revert win32 net tx hardening - do not use the Reset function to null pSocket
[*] fix a bytebuffer circular buffer path
[*] update how TLS protocol stacks can snap the stack
[+] AuMemory::ByteBuffer.flagNoFree
[*] SetBuffer(MemoryViewRead readView, bool bMoveWriteHeadForReaders = true);
...added bMoveWriteHeadForReaders
[-] AuMemory::SetBuffer(const void *in, AuUInt length)
[-] AuMemory::SetBuffer(const AuList<AuUInt8> &buffer)
[*] Harden AuMemory::ByteBuffer::RemainingBytes bc it is a high level api that is not called excessively. Invalid head states should not mess with ::CanRead/::CanWrite
[*] Harden AuMemory::ByteBuffer::RemainingWrite bc it is a high level api that is not called excessively. Invalid head states should not mess with ::CanRead/::CanWrite
[+] IConditionMutex::TryLock()
... Timed is still too much of an ask for this interface, i think. There's a good reason as to why this isn't a iwaitable. This condvar interface must be as common and primitive as possible.
[*] Refactor NoncanonicalInput
[*] Fix various keys resort in NUL getting added to ConsoleTTYs input field
[*] Fix multichar right key input (finish TODO note)
[*] Hex grammar: literals may end with 'h' now that i think about it. also added wiresharks seperator. let's continue to simply make sure we cover all forms of hex in a reasonably liberal fashion to account for user input from various human written and software form.
(Nonuniform style of non-hex illegal chars are fine so long as it looks and reads like a legal array of hex characters. Comas must come directly after literals, possible [0x] prefixes are fine, whitespaces too, copy/pasted '[', ']', '{', '}', 'reasonable' new line rejection, etc)
[+] Upcoming failure categories
[+] Updated SysPushXXX prototypes
[*] Restore bandwidth OnTick position for extrapolate (using current frame stats, ref to last) fractional lerps into the future with ref to average
[*] Compression.cpp AuList<AuUInt8> upgrade was incomplete & could've been improved with modern apis
[+] IOProcessor::WakeupThread
[+] NT: Begin hacking in timeBeginPeriod (must spam it in some places)
[+] ConsoleTTY (more specifically the win32 calls) are too slow to run on the mainthread. Delegate to worker.
[*] AuTime.CurrentClockSteady
[*] AuTime.CurrentClockSteadyMS
[*] AuTime.CurrentClockSteadyNS
[+] IOPipeRequest::uMinBytesToRead
[+] (secret api intended for unix users) AuIO::NewLSOSHandleEx
[*] Fix quirks when running Gtk under an io processor
...CtxYield shouldn't spin while work (improper breakout on remote update)
[*] EFileOpenMode::eWrite should assume O_CREAT semantics making eReadWrite somewhat redundant. OpenWrite + eWrite should reasonably work with file-appends. it should not mean force create + cucked GetLength().
[*] ISocket::Shutdown(*bool bNow*), allowing for flush of the send channel when false
[*] Fix StartRead and StartWrite after shutdown (NT)
[*] Amended dead-lock
[+] IOPipeRequestAIO::uStartOffset
[*] Fix Zstd frames being finicky
[*] Fix compression interceptor invalid weak reference to pipe that
[*] Fix WinFileStream::GetOffset using wrong enumeration
[*] Null ByteBuffer flags on init bc not all operators and constructors account for them
[will prevent bugs in the future]
[*] Prevent AuByteBuffer::begin from exploding user code by providing normalized begin positions
[*] Update AuByteBuffer string write
[*] Redefine AuByteBuffer::data() as the bytebuffers base
[*] Various pointer normalization improvements in ByteBuffer_Position.inl
[*] Update comments
[*] Update KCryptoAES: use streamable buffers. deprecate legacy vec<8>::size() based logic - AuByteBuffer (replacing vec8)::begin, ::end still works.
[*] IO pipe buffers should no longer be marked as circular
[*] ByteBuffer::IsValid() will now return true if there is no underlying buffer so long as the object is marked with the dynamic scaling flag (no contents isnt an invalid state; alloc failure on construct and ad-hoc failure is. latterly, specify the write flag on failure in the ByteBuffer::Write method.) the alloc error can be safely unwound by state raii helpers bc realloc and frens will not free or dirty the state. write-fail-aware may roll back the write head, like how readers can roll back the read head if the stream is incomplete.
> GetForcedMinRounds
> GenSalt
> HashPW
> HashPWEx
> CheckPassword
> CheckPasswordEx
[*] Refactor AuCompression APIs
[*] Clean up AuTryConstructs
[+] Internal compression API for compression based interceptors
[+] Root-level input stream arg check for all compression apis (harden)
[*] Clean up AuCompression code
[+] Solar Designer / OpenWall blowfish crypt
[*] BlowCrypt: accept length input parameter
[*] Split locale into 2 source files
[-] Ugly comment from Open.Win32.cpp. TODO: Readd later. Might warn on empty string bc it makes sense given, "." and "/" normalizes to nothing, and memory pre-idc-if-drops are dropped siliently.
(prepend makes sense for outbound stacks, where you might register the compression/crypto stages first, then return an object to the caller who may wish to prepend the input processors. http might need it.)
=============================================================================
Network ]====================================================================
=============================================================================
[+] Added (very) early Aurora::IO::Net implementation
[+] AuNet::EHostnameType
[+] AuNet::EIPProtocol
[+] AuNet::ENetworkError
[+] AuNet::ETransportProtocol
[+] AuNet::INetInterface
[+] AuNet::INetSrvDatagram
[+] AuNet::INetSrvResolve
[+] AuNet::INetSrvSockets
[+] AuNet::INetSrvWorkers
[+] AuNet::INetWorker
[+] AuNet::IPAddress
[+] AuNet::IResolver
[+] AuNet::ISocket
[+] AuNet::IResolver
[+] AuNet::ISocketBase
[+] AuNet::ISocketChannel
[+] AuNet::ISocketDriver
[+] AuNet::ISocketDriverFactory
[+] AuNet::ISocketServer
[+] AuNet::ISocketServerDriver
[+] AuNet::NetEndpoint
[+] AuNet::NetError
[+] AuNet::NetHostname
(+implementation)
=============================================================================
Protocol ]===================================================================
=============================================================================
[+] IProtocolInterceptor
[+] IProtocolInterceptorEx
[+] IProtocolStack
(+implementation)
=============================================================================
TLS ]========================================================================
=============================================================================
[+] ITLSContext
[+] TLSProtocolRecv
[+] TLSProtocolSend
(+implementation)
=============================================================================
IO Bug Fixes ]===============================================================
=============================================================================
[*] IOProcessor::SubmitIOWorkItem should signal the CvEvent, forcing at least once future tick (wont optimize with if in tick & not yet dispatched work items)
[*] Split IOPipeWork in into IOPipeProcessor header
[+] IOPipeWork::GetBuffer (internal reallocation)
[*] Harden against IAsyncTransactions without a loop source
[*] Missing null `if (processor->listener)` in IOProcessor
[*] Solved some soft-lock conditions under Linux's LoopQueue (added deferred commits)
[*] Quick hack: IOProcessor::HasItems() should OR the early can-tick check function.
=============================================================================
Other ]======================================================================
=============================================================================
[+] Linux: LSSignalCatcher
[+] `static void AuResetMember(Aurora::Memory::ByteBuffer &ref)` for AuROXTL
[*] Attempt to enforce a normalization and don't overwrite-readptr-under-istreamwriters policy in ByteBuffer_ReadWrite (circular buffers)
[*] Bad ECC ctors
=============================================================================
Known issues ]===============================================================
=============================================================================
> Linux net is nowhere near done
> UDP socket emulation layer isn't implemented
> Ciphersuite API is a stub
> Private key API is a stub
> ...therefore no TLS servers
> Missing thread safety precautions under net
> Net implementation is still beri early
[*] Fixed build errors post refactor
[*] Don't create an epoll for a read-poll, reuse fd
[+] AuByteBuffer:Allocate(AuUInt length, AuUInt alignment)
[*] NT+Linux: Ensure IO buffers are aligned to AuHwInfo::GetPageSize()
[*] Fix bug in IOProcessor where work items were not enough to cont execution in a loop checking ::HasItems
[+] PromiseCallbackFunctional constructor with one functor arg
[+] Begin work on IO futexes for io release on process/thread exit
[+] Linux ::readdir iteration
[+] AuConsole buffering API
[*] Fix sleep as to not get interrupted by signals
[*] Switch the type of FS lock used under Linux
[*] Linux: Use new IPCHandle encoding scheme
[*] Fix undefined behaviour: unintialized timeout values (AuLoop/Linux)
[*] Fix undefined behaviour: ConsoleTTY clear line was called of a color of a random value on stack
[-] Remainings of std dir iterator
[*] Fix pthread_kill (aka send signal to pthread handle) always kills process. This is what you expect bc signal handler inheritance.
[*] Reformat the build Aurora.json file
[+] Added clang warning ignores to the build file
[*] Fix: UNIX need to use STDOUT_FILENO. Was using CRT handle in place of fd by mistake.
[+] Linux implementation for IO yield (AuIO::IOYield() - UNIX::LinuxOverlappedYield())
[*] Fix: Linux async end of stream processing. res 0 = zero bytes consumed. <= was detecting this as an error of code 0. Should succeed with zero bytes.
[+] Linux LoopQueue missing epilogue hook for the IO processor
[*] Various refactors and minor bug fixes
[*] Linux fix: Handle pipe EOS as zero
[*] Linux fix: thread termination via a user signal of 77. Need a force terminate.
[*] IPC handle: fix improper int to bool cast in the header setup within ToString
[*] Linux fix: HWInfo CPU topology regression
[-] Linux fix: remove SIGABRT handler
[*] Missing override in compression, exit, and consoletty headers.
[+] Unix Syslog logger backend
[*] NT async file: error 38 "Reached the end of the file." should be handled as a zero length packet much like ERROR_BROKEN_PIPE for stream read interface consistency
[*] FIX: Missing `this->isStream = isStream;`
[*] FIX: Blocking APIs under async objects under NT were using illegal calls that were for some reason not failing
[+] IOProcessor::StartSimpleIOWatch(const AuSPtr<IIOWaitableItem> &object, const AuSPtr<IIOSimpleEventListener> &listener)
[+] IOProcessor::StartSimpleLSWatch(const AuSPtr<Loop::ILoopSource> &source, const AuSPtr<IIOSimpleEventListener> &listener)
[*] IOAdapterAsyncStream should reset the transactions IO state upon reaching end of segment (should this be per tick?) or upon stream error - otherwise, we end up spinning on a stuck event forever
[*] Fix non-linear path under read of the AuByteBuffer
[*] Fix various other nonlinear conditions under AuByteBuffer
[*] IOProcessor releases registered io item from queue upon request
[*] Fix ConsoleMessage::Write -> enumeration of color should be casted to a uint8
[+] Error telemetry under async task creation
[*] Fix various lock ups and non-blocking spins related to erroneous InternalRunOne impl. Residual preemptive batching was fucking with modern io.
[*] Cleanup TaskFrom/JobFrom. More work required to clean up legacy piss
[+] IIOBufferedStreamAvailable callback
[+] IIOProcessor singleshot work items / IIOProcessorWorkUnit
[+] IOPipeCallback description of a pipes destination
[+] IOPipeInputData description of a pipes source
[+] IOPipeRequest, IOPipeRequestAIO, IOPipeRequestBasic
[+] IPipeBackend hooks for on start/end hooks of IOPipeRequestBasics
[*] Update IOAdapaterAsyncStream implementation to better support caller buffering
[*] Updated IAsyncStreamReader to include a warm/dequeue API for direct async usage
[*] Fix NT IO regressions
[*] Fix ThreadPool shutdown on an unregistered thread
[*] Fix race condition in Async.NT.cpp & fix signalable state to closely match Linux (dunno how this was passing before)
[*] Refactor IOProcessorWorkUnit -> IIOProcessorWorkUnit
[*] Update experimental header to include the changes
[+] if ^::eString, [+] ...isControlSequence, isAltSequence, isShiftSequence
[*] Update semaphore from class to struct types
[*] GetProfileDomain and frens should check the result string length before returning true
[*] Expand watcher API -> Breaking NT
[*] Reexpand loop queue API -> Breaking NT
[*] Linux CPUInfo clean up
[*] Bug fix: mkdir should set execute flag... because directories are special
[*] Refactor: Cleanup base64
[*] Bug fix: UNIX path normalization
[*] Bug fix: missing O_CREAT flag (au auto-creates)
[*] Normalize line endings
[*] Treat SIGTERM the same as SIGINT. SIGINT is somewhat of an arachic signal meaning, "hey dumb unix app, fuck the process group, start reading from stdin to listen to the user." Nowadays, this doesn't mean anything other than "hey, a human asked us to terminate from a TTY" - basically the same as SIGTERM, except SIGTERM is more likely to be a scheduled or otherwise expected shutdown event.
The intention is to quickly improve and add util apis, enhance functionality given current demands, go back to the build pipeline, finish that, publish runtime tests, and then use what we have to go back to to linux support with a more stable api.
[+] AuMakeSharedArray
[+] Technet ArgvQuote
[+] Grug subsystem (UNIX signal thread async safe ipc + telemetry flusher + log flusher.)
[+] auEndianness -> Endian swap utils
[+] AuGet<N>(...)
[*] AUE_DEFINE conversion for
ECompresionType, EAnsiColor, EHashType, EStreamError, EHexDump
[+] ConsoleMessage ByteBuffer serialization
[+] CmdLine subsystem for parsing command line arguments and simple switch/flag checks
[*] Split logger from console subsystem
[+] StartupParameters -> A part of a clean up effort under Process
[*] Refactor SysErrors header + get caller hack
[+] Atomic APIs
[+] popcnt
[+] Ring Buffer sink
[+] Added more standard errors
Catch,
Submission,
LockError,
NoAccess,
ResourceMissing,
ResourceLocked,
MalformedData,
InSandboxContext,
ParseError
[+] Added ErrorCategorySet, ErrorCategoryClear, GetStackTrace
[+] IExitSubscriber, ETriggerLevel
[*] Write bias the high performance RWLockImpl read-lock operation operation
[+] ExitHandlerAdd/ExitHandlerRemove (exit subsystem)
[*] Updated API style
Digests
[+] CpuId::CpuBitCount
[+] GetUserProgramsFolder
[+] GetPackagePath
[*] Split IStreamReader with an inl file
[*] BlobWriter/BlobReader/BlobArbitraryReader can now take shared pointers to bytebuffers. default constructor allocates a new scalable bytebuffer
[+] ICharacterProvider
[+] ICharacterProviderEx
[+] IBufferedCharacterConsumer
[+] ProviderFromSharedString
[+] ProviderFromString
[+] BufferConsumerFromProvider
[*] Parse Subsystem uses character io bufferer
[*] Rewritten NT's high perf semaphore to use userland SRW/ConVars [like mutex, based on generic semaphore]
[+] ByteBuffer::ResetReadPointer
[*] Bug fix bytebuffer base not reset on free and some scaling issues
[+] ProcessMap -> Added kSectionNameStack, kSectionNameFile, kSectionNameHeap for Section
[*] ProcessMap -> Refactor Segment to Section. I was stupid for keeping a type conflict hack API facing
[+] Added 64 *byte* fast RNG seeds
[+] File Advisorys/File Lock Awareness
[+] Added extended IAuroraThread from OS identifier caches for debug purposes
[*] Tweaked how memory is reported on Windows. Better consistency of what values mean across functions.
[*] Broke AuroraUtils/Typedefs out into a separate library
[*] Update build script
[+] Put some more effort into adding detail to the readme before rewriting it, plus, added some media
[*] Improved public API documentation
[*] Bug fix `SetConsoleCtrlHandler`
[+] Locale TimeDateToFileNameISO8601
[+] Console config stdOutShortTime
[*] Begin using internal UTF8/16 decoders when platform support isnt available (instead of stl)
[*] Bug fixes in decoders
[*] Major bug fix, AuMax
[+] RateLimiter
[+] Binary file sink
[+] Log directory sink
[*] Data header usability (more operators)
[+] AuRemoveRange
[+] AuRemove
[+] AuTryRemove
[+] AuTryRemoveRange
[+] auCastUtils
[+] Finish NewLSWin32Source
[+] AuTryFindByTupleN, AuTryRemoveByTupleN
[+] Separated AuRead/Write types, now in auTypeUtils
[+] Added GetPosition/SetPosition to FileWriter
[*] Fix stupid AuMin in place of AuMax in SpawnThread.Unix.Cpp
[*] Refactored Arbitrary readers to SeekingReaders (as in, they could be atomic and/or parallelized, and accept an arbitrary position as a work parameter -> not Seekable, as in, you can simply set the position)
[*] Hack back in the sched deinit
[+] File AIO loop source interop
[+] Begin to prototype a LoopQueue object I had in mind for NT, untested btw
[+] Stub code for networking
[+] Compression BaseStream/IngestableStreamBase
[*] Major: read/write locks now support write-entrant read routines.
[*] Compression subsystem now uses the MemoryView concept
[*] Rewrite the base stream compressions, made them less broken
[*] Update hashing api
[*] WriterTryGoForward and ReaderTryGoForward now revert to the previous relative index instead of panicing
[+] Added new AuByteBuffer apis
Trim, Pad, WriteFrom, WriteString, [TODO: ReadString]
[+] Added ByteBufferPushReadState
[+] Added ByteBufferPushWriteState
[*] Move from USC-16 to full UTF-16. Win32 can handle full UTF-16.
[*] ELogLevel is now an Aurora enum
[+] Raised arbitrary limit in header to 255, the max filter buffer
[+] Explicit GZip support
[+] Explicit Zip support
[+] Added [some] compressors
et al
[*] Minor refactors
[*] Begin refactoring the Processes subsystem
[*] Added empty files ready to move the gross c++ functional callback in the parse subsystem to a dedicated interface w/ utils
[*] Broke out Win32 Process into an NT base variant (Process.NT.cpp) with Win32 overloaded process exit pokes (Process.Win32.cpp)
[*] Added const modifier to cpuid functions
[*] Added OnExit to planned API interface
[*] BlobReader now holds onto a handle of a ByteBuffer, allowing for shared usage -> aiding in the mitigation of allocations/second and needless heavy copy of objects
[+] AuRemoveConst
[*] Support circular reference in Aurora pipelines Include, added support for early Aurora::Build AuroraEnum
[+] Added SWInfo API
[+] AU_COPY_MOVE, AU_MOVE, AU_COPY to go with AU_NO_... variants
[+] Adding GetProcessId
[*] Fix deadlock in the async subsystem (NoLockShutdown vs Shutdown in exception handler)
[+] Added ProccessMap NT variant
[+] Added ToolHelp image profiling
[*] Improved exception awareness
[*] Delegated SpawnThread to isolated TU, ready for reuse for RunAs and XNU Open - now with horrible evil alloc that could fail
[+] Added header for future api 'UtilRun'
[*] Improve NT core detection
[*] Changed small affinity bitmap to AuUInt64 instead of AuUInt32
[+] Added data structure to hold cpuids/affinity masks
[+] Implemented logger sinks
[+] Implemented logger glue logic
[*] Began migrating older loggers to sink-based default devices
[*] Minor refactors
[*] Improved internal exception discarding, not yet nothrow capable
[*] Minor create directory fix
[*] Dont fail on non-blocking /dev/urand
[+] Added if not initialized check under RNG/unix fclose
[+] Add missing debug apis ready for report to linux
[*] Update build script for WIP branch
[+] Added explicit AU_WHAT macro, I think people are familiar with this nomenclatures, not sure.
[+] Added try catch around logger write line. I'm sure something will explode around here sooner or later
Reece, [07/11/2021 20:10]
technically legal template odr
Reece, [07/11/2021 20:11]
either we resolve the vtable from an api that returns a shared pointer of the implementation. The inline keyword should help with linking. Seems to work.
Reece, [07/11/2021 20:11]
or we just have the C/FTask impl in which case some compilers will C-inline the jmp target through the final hints. Connfirmed under MSVC and GCC.
[*] Split up Aurora Async
[*] Split Async app into seperate ThreadPool concept
[*] Fix various OSThread bugs and tls transfer issues
[*] Set default affinity to 0xFFFFFFFF
[*] Update Build script
[+] Add AuTuplePopFront
[+] New Network Interface (unimplemented)
[*] Stub out the interfaces required for a better logger
[*] Fix Win32 ShellExecute bug; windows 11 struggles without explicit com init per the docs - now deferring to thread pool
[*] Update gitignore
[*] Follow XDG home standard
[*] Refactor some namespaces to use the shorthand aliases
[*] Various stability fixes
