I have run into a problem when I use POST characters above \x7F to Dancer 1.3093 and then applying from_json to them.

I have discussed the issue with ambs, who reported an issue with to_json in versions prior to 1.3093, and he has helpfully (and speedily) found a fix, but it's still unclear why it needs fixing, and whether this is something that Dancer should fix, so I'm posting to the list for wider input, ideas, discussion.

To recreate:

Create a new app:

        dancer -a MyWeb::App

and replace your index.tt and App.pm with those attached (alternatively, use the diff attached).

Load it into your browser, click the button, and it sends {"q":"café"} to the server, which processes it fine and returns that word. All good so far.

Notice that in MyWeb/App.pm, the to_json has a flag utf8=>0. This is the mysterious fix.

Now, remove that flag, so the line reads

        my $data = from_json( param('json'));

... and reload the app, click the button, and you will get an 500 internal error response, reading:

        {
          "exception" : "malformed UTF-8 character in JSON string, at character offset 9 (before \"\\x{98bd}\") at /usr/lib/perl5/site_perl/5.10/JSON.pm line 171.\n",
          "error" : "malformed UTF-8 character in JSON string, at character offset 9 (before \"\\x{98bd}\") at /usr/lib/perl5/site_perl/5.10/JSON.pm line 171.\n"
       }

(NB: in earlier versions, such as 1.3072, you won't get an error.)

What puzzles me most here is the reference to \x{98bd} - I have no idea how from_json is getting \x98bd. What gets sent is json=%7B%22q%22%3A%22caf%C3%A9%22%7D  - c3+a9 being utf8 for \xe9 i.e é.

Ambs says "Now, why you need to make utf8 to false, because the string is in UTF8 but doesn't have the utf8 flag on. So, when asking to parse it as utf8 it will double encode the thing (I think)."

The question I have is "Should the utf8 flag be on anyway?" - is this something Dancer should be doing?

It seems odd to me that Dancer makes available to the user a utf8 string without the utf8 flag, but perhaps there is a good reason for it? (or I have misunderstood?)

Possibly Relevant links...

- https://github.com/sukria/Dancer/pull/740/files
- https://github.com/sukria/Dancer/issues/749
- https://github.com/sukria/Dancer/pull/726

Daniel