# \[Solved\] Handling invalid UTF-8 characters

**URL:** <https://discourse.libcinder.org/t/solved-handling-invalid-utf-8-characters/1197>\
**Category:** Using Cinder\
**Created:** [June 19, 2018, 10:00am UTC](https://discourse.libcinder.org/t/solved-handling-invalid-utf-8-characters/1197 "2018-06-19T10:00:12Z")\
**Posts on this page:** 5\
**Page:** 1

<div class="post-metadata">

**Author:** ![Gazoo](https://yyz2.discourse-cdn.com/flex030/user_avatar/discourse.libcinder.org/gazoo/32/61_2.png) [@Gazoo](https://discourse.libcinder.org/u/Gazoo)\
**Post date:** [June 19, 2018, 10:00am UTC](https://discourse.libcinder.org/t/solved-handling-invalid-utf-8-characters/1197/1 "2018-06-19T10:00:12Z")

</div>

Hey Embers,

**TL;DR version:**

I’d like to use cinder’s `toUtf16()` function on a string like this “ñêQRV”, but this leads to an invalid UTF-8 exception. I’m guessing I have to convert them to characters representing their unicode value, as the character itself isn’t a proper UTF-8 character? Any tips/hints on doing this?

**Long version:**

I’m reading in text from [Mp3 ID3 tags](https://en.wikipedia.org/wiki/ID3) and apparently it occasionally uses a (now) deprecated encoding. Specifically, “UCS-2 encoded Unicode with BOM” according to the [wikipedia page](https://en.wikipedia.org/wiki/ID3). I spent at least an hour trying to find the ‘right way’ to properly decode it, until I eventually gave up and settled for [a solution provided by a kind soul on the github for the id3 decode library](https://github.com/larsbs/id3v2lib/issues/7). I’m left with an `std::string` which will occasionally contain what is apparently invalid UTF-8 characters because when I use `toUtf16()` on it, I’m dealt an exception highlighting this.

I know [UTF-16 is generally to be avoided](https://discourse.libcinder.org/t/utf8-encoding-and-cinder-and-cross-platform/124) as per the ever helpful @paul.houx, but since I only plan to use UTF-16 internally on the windows platform for now, I’ve opted to make an exception.

I use it primarily so I can remove a single character at a time from a string and be sure it is a ‘complete’ character rather than a portion of one.

So I’m wondering what I’m to do - perhaps convert the characters to proper UTF-8 using `toUtf8()`? But if that’s the approach, how do I do this when these characters are in an `std::string` to begin with?

Thanks in advance,

Gazoo

---

<div class="post-metadata">

**Author:** ![lithium](https://yyz2.discourse-cdn.com/flex030/user_avatar/discourse.libcinder.org/lithium/32/38_2.png) [@lithium](https://discourse.libcinder.org/u/lithium)\
**Post date:** [June 19, 2018, 1:45pm UTC](https://discourse.libcinder.org/t/solved-handling-invalid-utf-8-characters/1197/2 "2018-06-19T13:45:00Z")

</div>

Since you’re on windows only, you might be able to do the [MultiByteToWideChar](https://msdn.microsoft.com/en-us/library/windows/desktop/dd319072(v=vs.85).aspx) + `std::wstring` / `std::u16string` dance. Apologies in advance.

---

<div class="post-metadata">

**Author:** ![benjaminbojko](https://yyz2.discourse-cdn.com/flex030/user_avatar/discourse.libcinder.org/benjaminbojko/32/591_2.png) [@benjaminbojko](https://discourse.libcinder.org/u/benjaminbojko)\
**Post date:** [June 19, 2018, 1:55pm UTC](https://discourse.libcinder.org/t/solved-handling-invalid-utf-8-characters/1197/3 "2018-06-19T13:55:33Z")

</div>

Speaking a bit out of my comfort zone here, but on Windows we’ve had good luck with `wstring_convert` and bundled things in these conversion methods: [https://github.com/bluecadet/Cinder-BluecadetText/blob/develop/src/bluecadet/text/Text.h#L270-L288](https://github.com/bluecadet/Cinder-BluecadetText/blob/develop/src/bluecadet/text/Text.h#L270-L288). Perhaps there’s something helpful there.

---

<div class="post-metadata">

**Author:** ![xumo](https://yyz2.discourse-cdn.com/flex030/user_avatar/discourse.libcinder.org/xumo/32/75_2.png) [@xumo](https://discourse.libcinder.org/u/xumo)\
**Post date:** [June 19, 2018, 2:06pm UTC](https://discourse.libcinder.org/t/solved-handling-invalid-utf-8-characters/1197/4 "2018-06-19T14:06:46Z")

</div>

I use latin windows (spanish and portuguese). So when tryiing to draw system messages I use this replace method:

```
#include "utf8cpp/checked.h"

    std::string text = "Texto en español ";
    std::string temp;
    //UTF8 valid checkout
   utf8::replace_invalid(text.begin(), text.end(), back_inserter(temp));

```

Then I can pass it to somenthing like a ci::TextBox.

---

<div class="post-metadata">

**Author:** ![Gazoo](https://yyz2.discourse-cdn.com/flex030/user_avatar/discourse.libcinder.org/gazoo/32/61_2.png) [@Gazoo](https://discourse.libcinder.org/u/Gazoo)\
**Post date:** [June 20, 2018, 9:29am UTC](https://discourse.libcinder.org/t/solved-handling-invalid-utf-8-characters/1197/5 "2018-06-20T09:29:28Z")

</div>

Much appreciate all of the answers. It would seem to me that @xumo’s suggestion is the most cross-platform friendly. So I opted for that approach.

Honestly, I wasn’t even aware that Cinder had this functionality available.

Thanks everybody!

Cheers,

Gazoo