scuffed-code/icu4c/source/common/udataswp.h
Markus Scherer fe3eb3ed5c
ICU-13530 add UCPTrie/CodePointTrie, switch normalization to use it (#48)
* ICU-13530 copy C/C++ files UTrie2 -> UTrie3

X-SVN-Rev: 40754

* ICU-13530 UTrie3 new files copied from UTrie2: rename types/functions/macros

X-SVN-Rev: 40755

* ICU-13530 debug-print building each UTrie2

X-SVN-Rev: 40756

* ICU-13530 remove two-byte-UTF-8 errorValue block; move highValue from end of data array into header; add errorValue to header

X-SVN-Rev: 40762

* ICU-13530 UTrie3 U16_NEXT/PREV: errorValue for unpaired surrogates

X-SVN-Rev: 40763

* ICU-13530 no more separate values for lead surrogate code units

X-SVN-Rev: 40764

* ICU-13530 change from 11:5 trie bits to 10:6 for simpler UTF-8 code

X-SVN-Rev: 40766

* ICU-13530 UTrie2 build UTrie3 as well, print sizes

X-SVN-Rev: 40767

* ICU-13530 debug-print countSame, sumOverlaps, countInitial

X-SVN-Rev: 40768

* ICU-13530 debug-print whether trie is for CanonIterData

X-SVN-Rev: 40769

* ICU-13530 no index-shift for BMP data, no separate index-2 for 2-byte UTF-8; builder changes incomplete

X-SVN-Rev: 40777

* ICU-13530 remove errorValue and highStart from UNewTrie3

X-SVN-Rev: 40778

* ICU-13530 rewrite UTrie3 builder code

X-SVN-Rev: 40783

* ICU-13530 UTrie3 bug fixes

X-SVN-Rev: 40788

* ICU-13530 fully re-inline _UTRIE3_U8_NEXT()

X-SVN-Rev: 40790

* ICU-13530 find most common all-same data block for dataNullBlock and initialValue

X-SVN-Rev: 40792

* ICU-13530 UTrie3 iterator functions take start and return the end of a range, rather than callback call for each range

X-SVN-Rev: 40800

* ICU-13530 mask off unused data value bits before building a UTrie3 with values less than 32 bits wide

X-SVN-Rev: 40803

* ICU-13530 split utrie3builder.h out of utrie3.h

X-SVN-Rev: 40804

* ICU-13530 separate types UTrie3 vs. UTrie3Builder, implement builder as wrapper over C++ class Trie3Builder in .cpp

X-SVN-Rev: 40809

* ICU-13530 function to make a UTrie3Builder from a UTrie3

X-SVN-Rev: 40810

* ICU-13530 debug-print some data; some cleanup

X-SVN-Rev: 40865

* ICU-13530 BMP 10:6 but supplementary 10:6:4

X-SVN-Rev: 40984

* ICU-13530 move errorValue & highValue to the end of the data table, minimal padding to 4 bytes

X-SVN-Rev: 41011

* ICU-13530 index-1 table gap of index-2 null blocks

X-SVN-Rev: 41018

* ICU-13530 test with more than 128k compacted data

X-SVN-Rev: 41034

* ICU-13530 supplementary bits 11:5:4 saves a little space

X-SVN-Rev: 41039

* ICU-13530 supplementary bits 6:5:5:4 instead of gap: about same size but simpler

X-SVN-Rev: 41050

* ICU-13530 remove unnecessary utrie3_clone(built trie)

X-SVN-Rev: 41058

* ICU-13530 remove unnecessary UTrie3StringIterator

X-SVN-Rev: 41059

* ICU-13530 back to UTRIE3_GET...() macros *returning* data values

X-SVN-Rev: 41060

* ICU-13530 fast vs. small

X-SVN-Rev: 41066

* ICU-13530 always load NFC data, add simple normalization performance test

X-SVN-Rev: 41110

* ICU-13530 change normalization main trie to UTrie3 with special values for lead surrogates; forbid non-inert surrogate code *points* because unable to store values different from code *units*; runtime code work around that for code point lookup and iteration; adjust UTS 46 for normalization no longer mapping unpaired surrogates to U+FFFD

X-SVN-Rev: 41122

* ICU-13530 simplenormperf bug fix and NFC base line

X-SVN-Rev: 41126

* ICU-13530 move normalization getRange skipping lead surrogates to API getRangeSkipLead()

X-SVN-Rev: 41182

* ICU-13530 switch CanonIterData and gennorm2 Norms to UTrie3

X-SVN-Rev: 41183

* ICU-13530 remove unused overwrite parameter from setRange()

X-SVN-Rev: 41184

* ICU-13530 getRange skip lead -> fixed surrogates

X-SVN-Rev: 41219

* ICU-13530 minor cleanup

X-SVN-Rev: 41221

* ICU-13530 UTS 46 code map unpaired surrogates to U+FFFD before normalization

X-SVN-Rev: 41224

* ICU-13530 minor internal-docs cleanup

X-SVN-Rev: 41225

* ICU-13530 rename UTrie3 to UCPTrie, and other name changes

X-SVN-Rev: 41226

* ICU-13530 add 8-bit data option; add type-any & valueBits-any for fromBinary(); macros consistently source type then data width

X-SVN-Rev: 41234

* ICU-13530 scrub the API docs for the proposal

X-SVN-Rev: 41319

* ICU-13530 tag internal definitions as such, or move them to an internal header

X-SVN-Rev: 41320

* ICU-13530 Java API skeleton

X-SVN-Rev: 41326

* ICU-13530 API feedback: ValueWidth, MutableCodePointTrie, base CodePointMap, ...

X-SVN-Rev: 41382

* ICU-13530 add UCPTrie valueWidth field and padding, and combine data pointers into a union

X-SVN-Rev: 41408

* ICU-13530 switch some macros to using dataAccess parameter: separate index vs. data lookups, no macro variant for each value width

X-SVN-Rev: 41409

* ICU-13530 StringIterator is no longer a java.util.Iterator (bad fit)

X-SVN-Rev: 41455

* ICU-13530 CodePointTrie.java code complete

X-SVN-Rev: 41518

* ICU-13530 finish Java port incl test; keep C++ parallel

* ICU-13530 adjust API for feedback: rename HandleValue to FilterValue, change getRange+getRangeFixedSurr(bool allSurr) to enum RangeOption+getRange(enum option); change remaining C macros to use dataAccess for 16/32/8-bit value widths; fix/clarify some API docs

* ICU-13530 add javadoc

* ICU-13530 document UCPTrie binary data format

* ICU-13530 update .nrm formatVersion 3->4, document change in surrogate handling with new trie

* ICU-13530 re-hardcode NFC data

* move trie swapper code into new file; add new files to Windows project files; turn off trie debugging

* ICU-13530 minor cleanup

* ICU-13530 test more range starts; fix a C test leak

* ICU-13530 regenerate Java data from scratch

* ICU-13530 review feedback changes: API docs typos, more @internal, C++11 field initializers, fix potential leak in MutableCodePointTrie::fromUCPTrie()

* ICU-13530 rename interface FilterValue to ValueFilter
2018-09-27 14:27:38 -07:00

405 lines
12 KiB
C

// © 2016 and later: Unicode, Inc. and others.
// License & terms of use: http://www.unicode.org/copyright.html
/*
*******************************************************************************
*
* Copyright (C) 2003-2014, International Business Machines
* Corporation and others. All Rights Reserved.
*
*******************************************************************************
* file name: udataswp.h
* encoding: UTF-8
* tab size: 8 (not used)
* indentation:4
*
* created on: 2003jun05
* created by: Markus W. Scherer
*
* Definitions for ICU data transformations for different platforms,
* changing between big- and little-endian data and/or between
* charset families (ASCII<->EBCDIC).
*/
#ifndef __UDATASWP_H__
#define __UDATASWP_H__
#include <stdarg.h>
#include "unicode/utypes.h"
/* forward declaration */
U_CDECL_BEGIN
struct UDataSwapper;
typedef struct UDataSwapper UDataSwapper;
/**
* Function type for data transformation.
* Transforms data, or just returns the length of the data if
* the input length is -1.
* Swap functions assume that their data pointers are aligned properly.
*
* Quick implementation outline:
* (best to copy and adapt and existing swapper implementation)
* check that the data looks like the expected format
* if(length<0) {
* preflight:
* never dereference outData
* read inData and determine the data size
* assume that inData is long enough for this
* } else {
* outData can be NULL if length==0
* inData==outData (in-place swapping) possible but not required!
* verify that length>=(actual size)
* if there is a chance that not every byte up to size is reached
* due to padding etc.:
* if(inData!=outData) {
* memcpy(outData, inData, actual size);
* }
* swap contents
* }
* return actual size
*
* Further implementation notes:
* - read integers from inData before swapping them
* because in-place swapping can make them unreadable
* - compareInvChars compares a local Unicode string with already-swapped
* output charset strings
*
* @param ds Pointer to UDataSwapper containing global data about the
* transformation and function pointers for handling primitive
* types.
* @param inData Pointer to the input data to be transformed or examined.
* @param length Length of the data, counting bytes. May be -1 for preflighting.
* If length>=0, then transform the data.
* If length==-1, then only determine the length of the data.
* The length cannot be determined from the data itself for all
* types of data (e.g., not for simple arrays of integers).
* @param outData Pointer to the output data buffer.
* If length>=0 (transformation), then the output buffer must
* have a capacity of at least length.
* If length==-1, then outData will not be used and can be NULL.
* @param pErrorCode ICU UErrorCode parameter, must not be NULL and must
* fulfill U_SUCCESS on input.
* @return The actual length of the data.
*
* @see UDataSwapper
* @internal ICU 2.8
*/
typedef int32_t U_CALLCONV
UDataSwapFn(const UDataSwapper *ds,
const void *inData, int32_t length, void *outData,
UErrorCode *pErrorCode);
/**
* Convert one uint16_t from input to platform endianness.
* @internal ICU 2.8
*/
typedef uint16_t U_CALLCONV
UDataReadUInt16(uint16_t x);
/**
* Convert one uint32_t from input to platform endianness.
* @internal ICU 2.8
*/
typedef uint32_t U_CALLCONV
UDataReadUInt32(uint32_t x);
/**
* Convert one uint16_t from platform to input endianness.
* @internal ICU 2.8
*/
typedef void U_CALLCONV
UDataWriteUInt16(uint16_t *p, uint16_t x);
/**
* Convert one uint32_t from platform to input endianness.
* @internal ICU 2.8
*/
typedef void U_CALLCONV
UDataWriteUInt32(uint32_t *p, uint32_t x);
/**
* Compare invariant-character strings, one in the output data and the
* other one caller-provided in Unicode.
* An output data string is compared because strings are usually swapped
* before the rest of the data, to allow for sorting of string tables
* according to the output charset.
* You can use -1 for the length parameters of NUL-terminated strings as usual.
* Returns Unicode code point order for invariant characters.
* @internal ICU 2.8
*/
typedef int32_t U_CALLCONV
UDataCompareInvChars(const UDataSwapper *ds,
const char *outString, int32_t outLength,
const UChar *localString, int32_t localLength);
/**
* Function for message output when an error occurs during data swapping.
* A format string and variable number of arguments are passed
* like for vprintf().
*
* @param context A function-specific context pointer.
* @param fmt The format string.
* @param args The arguments for format string inserts.
*
* @internal ICU 2.8
*/
typedef void U_CALLCONV
UDataPrintError(void *context, const char *fmt, va_list args);
struct UDataSwapper {
/** Input endianness. @internal ICU 2.8 */
UBool inIsBigEndian;
/** Input charset family. @see U_CHARSET_FAMILY @internal ICU 2.8 */
uint8_t inCharset;
/** Output endianness. @internal ICU 2.8 */
UBool outIsBigEndian;
/** Output charset family. @see U_CHARSET_FAMILY @internal ICU 2.8 */
uint8_t outCharset;
/* basic functions for reading data values */
/** Convert one uint16_t from input to platform endianness. @internal ICU 2.8 */
UDataReadUInt16 *readUInt16;
/** Convert one uint32_t from input to platform endianness. @internal ICU 2.8 */
UDataReadUInt32 *readUInt32;
/** Compare an invariant-character output string with a local one. @internal ICU 2.8 */
UDataCompareInvChars *compareInvChars;
/* basic functions for writing data values */
/** Convert one uint16_t from platform to input endianness. @internal ICU 2.8 */
UDataWriteUInt16 *writeUInt16;
/** Convert one uint32_t from platform to input endianness. @internal ICU 2.8 */
UDataWriteUInt32 *writeUInt32;
/* basic functions for data transformations */
/** Transform an array of 16-bit integers. @internal ICU 2.8 */
UDataSwapFn *swapArray16;
/** Transform an array of 32-bit integers. @internal ICU 2.8 */
UDataSwapFn *swapArray32;
/** Transform an array of 64-bit integers. @internal ICU 53 */
UDataSwapFn *swapArray64;
/** Transform an invariant-character string. @internal ICU 2.8 */
UDataSwapFn *swapInvChars;
/**
* Function for message output when an error occurs during data swapping.
* Can be NULL.
* @internal ICU 2.8
*/
UDataPrintError *printError;
/** Context pointer for printError. @internal ICU 2.8 */
void *printErrorContext;
};
U_CDECL_END
U_CAPI UDataSwapper * U_EXPORT2
udata_openSwapper(UBool inIsBigEndian, uint8_t inCharset,
UBool outIsBigEndian, uint8_t outCharset,
UErrorCode *pErrorCode);
/**
* Open a UDataSwapper for the given input data and the specified output
* characteristics.
* Values of -1 for any of the characteristics mean the local platform's
* characteristics.
*
* @see udata_swap
* @internal ICU 2.8
*/
U_CAPI UDataSwapper * U_EXPORT2
udata_openSwapperForInputData(const void *data, int32_t length,
UBool outIsBigEndian, uint8_t outCharset,
UErrorCode *pErrorCode);
U_CAPI void U_EXPORT2
udata_closeSwapper(UDataSwapper *ds);
/**
* Read the beginning of an ICU data piece, recognize magic bytes,
* swap the structure.
* Set a U_UNSUPPORTED_ERROR if it does not look like an ICU data piece.
*
* @return The size of the data header, in bytes.
*
* @internal ICU 2.8
*/
U_CAPI int32_t U_EXPORT2
udata_swapDataHeader(const UDataSwapper *ds,
const void *inData, int32_t length, void *outData,
UErrorCode *pErrorCode);
/**
* Convert one int16_t from input to platform endianness.
* @internal ICU 2.8
*/
U_CAPI int16_t U_EXPORT2
udata_readInt16(const UDataSwapper *ds, int16_t x);
/**
* Convert one int32_t from input to platform endianness.
* @internal ICU 2.8
*/
U_CAPI int32_t U_EXPORT2
udata_readInt32(const UDataSwapper *ds, int32_t x);
/**
* Swap a block of invariant, NUL-terminated strings, but not padding
* bytes after the last string.
* @internal
*/
U_CAPI int32_t U_EXPORT2
udata_swapInvStringBlock(const UDataSwapper *ds,
const void *inData, int32_t length, void *outData,
UErrorCode *pErrorCode);
U_CAPI void U_EXPORT2
udata_printError(const UDataSwapper *ds,
const char *fmt,
...);
/* internal exports from putil.c -------------------------------------------- */
/* declared here to keep them out of the public putil.h */
/**
* Swap invariant char * strings ASCII->EBCDIC.
* @internal
*/
U_CAPI int32_t U_EXPORT2
uprv_ebcdicFromAscii(const UDataSwapper *ds,
const void *inData, int32_t length, void *outData,
UErrorCode *pErrorCode);
/**
* Copy invariant ASCII char * strings and verify they are invariant.
* @internal
*/
U_CFUNC int32_t
uprv_copyAscii(const UDataSwapper *ds,
const void *inData, int32_t length, void *outData,
UErrorCode *pErrorCode);
/**
* Swap invariant char * strings EBCDIC->ASCII.
* @internal
*/
U_CFUNC int32_t
uprv_asciiFromEbcdic(const UDataSwapper *ds,
const void *inData, int32_t length, void *outData,
UErrorCode *pErrorCode);
/**
* Copy invariant EBCDIC char * strings and verify they are invariant.
* @internal
*/
U_CFUNC int32_t
uprv_copyEbcdic(const UDataSwapper *ds,
const void *inData, int32_t length, void *outData,
UErrorCode *pErrorCode);
/**
* Compare ASCII invariant char * with Unicode invariant UChar *
* @internal
*/
U_CFUNC int32_t
uprv_compareInvAscii(const UDataSwapper *ds,
const char *outString, int32_t outLength,
const UChar *localString, int32_t localLength);
/**
* Compare EBCDIC invariant char * with Unicode invariant UChar *
* @internal
*/
U_CFUNC int32_t
uprv_compareInvEbcdic(const UDataSwapper *ds,
const char *outString, int32_t outLength,
const UChar *localString, int32_t localLength);
/**
* \def uprv_compareInvWithUChar
* Compare an invariant-character strings with a UChar string
* @internal
*/
#if U_CHARSET_FAMILY==U_ASCII_FAMILY
# define uprv_compareInvWithUChar uprv_compareInvAscii
#elif U_CHARSET_FAMILY==U_EBCDIC_FAMILY
# define uprv_compareInvWithUChar uprv_compareInvEbcdic
#else
# error Unknown charset family!
#endif
// utrie_swap.cpp -----------------------------------------------------------***
/**
* Swaps a serialized UTrie.
* @internal
*/
U_CAPI int32_t U_EXPORT2
utrie_swap(const UDataSwapper *ds,
const void *inData, int32_t length, void *outData,
UErrorCode *pErrorCode);
/**
* Swaps a serialized UTrie2.
* @internal
*/
U_CAPI int32_t U_EXPORT2
utrie2_swap(const UDataSwapper *ds,
const void *inData, int32_t length, void *outData,
UErrorCode *pErrorCode);
/**
* Swaps a serialized UCPTrie.
* @internal
*/
U_CAPI int32_t U_EXPORT2
ucptrie_swap(const UDataSwapper *ds,
const void *inData, int32_t length, void *outData,
UErrorCode *pErrorCode);
/**
* Swaps a serialized UTrie, UTrie2, or UCPTrie.
* @internal
*/
U_CAPI int32_t U_EXPORT2
utrie_swapAnyVersion(const UDataSwapper *ds,
const void *inData, int32_t length, void *outData,
UErrorCode *pErrorCode);
/* material... -------------------------------------------------------------- */
#if 0
/* udata.h */
/**
* Public API function in udata.c
*
* Same as udata_openChoice() but automatically swaps the data.
* isAcceptable, if not NULL, may accept data with endianness and charset family
* different from the current platform's properties.
* If the data is acceptable and the platform properties do not match, then
* the swap function is called to swap an allocated version of the data.
* Preflighting may or may not be performed depending on whether the size of
* the loaded data item is known.
*
* @param isAcceptable Same as for udata_openChoice(). May be NULL.
*
* @internal ICU 2.8
*/
U_CAPI UDataMemory * U_EXPORT2
udata_openSwap(const char *path, const char *type, const char *name,
UDataMemoryIsAcceptable *isAcceptable, void *isAcceptableContext,
UDataSwapFn *swap,
UDataPrintError *printError, void *printErrorContext,
UErrorCode *pErrorCode);
#endif
#endif