Usage

utf.hpp Usage

The public interface of the utf.hpp library is very simple, consisting of the class template stringview which, as the name implies, provides an immutable view of an existing string. Importantly, it does not own the string data, and it does not create a copy of the string you pass to it. It simply stores two pointers (or iterators) delimiting the string data, and provides helper functions to treat this data as if it were the specified encoding. This also means that it is very cheap to create, and can be freely created at the cost of copying a pair of pointers.

The stringview template is specialized on two parameters, a type tag specifying the encoding the underlying data uses, and an iterator type specifying the types of iterators it should store.

Finally, a convenience utf::make_stringview function will infer these parameters from the iterators it is called with.

Examples

Note that where convenient, these examples use Unicode character types, and Unicode string literals as defined in C++11, but this is not necessary. You can use any other integer type of the appropriate size (int, uint16_t, signed char or whatever else you like).

Create a stringview from string literals

utf::stringview<utf::utf8, const char*> str_utf8 = utf::make_stringview(u8"some UTF-8 data");
utf::stringview<utf::utf16, const char*> str_utf16 = utf::make_stringview(u"some UTF-16 data");
utf::stringview<utf::utf32, const char*> str_utf32 = utf::make_stringview(U"some UTF-32 data");

Create a stringview from iterators

std::string src_utf8 = <some UTF-8 data>;

auto str_utf8 = utf::make_stringview(src_utf8.begin(), src_utf8.end());

std::vector<uint16_t> src_utf16 = <some UTF-16 data>;
auto str_utf16 = utf::make_stringview(src_utf16.begin(), src_utf16.end();

char32_t* src_utf32 = <some UTF-32 data>;
auto str_utf32 = utf::make_stringview(src_utf32, src_utf32 + length);

Convert from one encoding to another

auto utf16_str = utf::make_stringview(u"some UTF-16 data");

// converting into a pre-allocated buffer:
std::vector<char> utf8_buf;
// resize it appropriately
// how big would the string be if encoded as UTF-8?
size_t utf8_len = str.codeunits<utf::utf8>();
// resize the buffer
utf8_buf.resize(utf8_len);
// convert the string, writing into utf8_buf
str.to<utf::utf8>(utf8_buf.begin());

// converting into a buffer allocated on the fly:
std::vector<uint32_t> utf32_buf;
str.to<utf::utf32>(std::back_inserter(utf32_buf));

Iterate over the code points of a string

// A UTF-16 string containing 4 code units, representing the three code points, HIGH-SPEED TRAIN WITH BULLET NOSE, space, ø
char16_t src[] = {0xd83d, 0xde85, 0x20, 0xf8};

auto sv = utf::make_stringview(src);

// will print out the code points represented by the string, the values 0x1f685, 0x20, 0xf8
std::for_each(sv.begin(), sv.end(), [](utf::codepoint_type c){
    std::cout << (uint32_t)*c << ' ';
});

Synopsis

namespace utf {
  // type tags for the different encodings
  struct utf8;
  struct utf16; // uses native endianness
  struct utf32;

  typedef char32_t codepoint_type;
}

namespace utf {
  // Specialized for iterator type of the underlying string,
  // and deduces an encoding based on the value type's size
    template <typename Iter, typename E = implementation-defined-default>
  struct stringview {
  
    // constructor: takes a pair of iterators
    stringview(const Iter first, const Iter last);

    // returns iterators for on-the-fly decoding of a string
    // from its current encoding to Unicode code points
    codepoint_iterator<Iter> begin() const;
    codepoint_iterator<Iter> end() const;
    
    // check that the string is valid under its current encoding 
    bool validate() const;
    
    // number of code points in the string
    size_t codepoints() const;

    // length of the string in bytes, in its current encoding
    size_t bytes() const;
    
    // length of the string, in bytes, if encoded as EDest
    template <typename EDest>
    size_t bytes() const;
    
    // length of the string, in code units, in its current encoding
    size_t codeunits() const;

    // length of the string, in code units, if encoded as EDest
    template <typename EDest>
    size_t codeunits() const;
    
    // encode the string as EDest, writing the output to dest
    // returns an iterator pointing to the end of the output
    template <typename EDest, typename OutIt>
    OutIt to(OutIt dest) const;
  };

  // return a stringview over the specified iterator range
  // deducing its encoding from the size of the iterator's value_type
  template <typename Iter>
  stringview<Iter> make_stringview(Iter first, Iter last);

  // return a stringview over the specified array
  // deducing its encoding from the size of the element type
  template <typename T, size_t N>
  stringview<const T*> make_stringview(T (&arr)[N]);
}

When creating a stringview, the encoding E's default parameter is the type tag of the encoding that matches the size of std::iterator_traits<Iter>::value_type>. In other words, an iterator over char will yield a default encoding of utf8, and an iterator over uint32_t will yield utf32 as the encoding.

The stringview is immutable, and none of the member functions allow you to modify the underlying string data. Instead, to actually perform a conversion, an output iterator is supplied, and the return value is an iterator pointing to the end of the generated sequence, in the same style as std::transform and other standard library algorithms.

Building and running

to run the unit tests, you need the Catch unit test framework, and set up your include path so that catch/catch.hpp can be found. Then, simply build tests.cpp and run it.
by default, the C++11 Unicode character types (or typedefs mapping the same names to appropriate integer types) are expected to be available. To target C++03, define UTFHPP_NO_CPP11.
to build the samples, simply build samples.cpp. No external includes necessary.
to include utf.hpp into your project, just… #include "utf.hpp". (or #include <utf/utf.hpp>, or however you've set up your include paths)

Navigation Menu