Hi All, I will return JSON data from a route. If I don't use Dancer2 framework, I do it as follow and it works: sub json_branch_list { my $sth_branch = $dbh->prepare (qq(SELECT BranchID,Branch FROM branch WHERE Deleted=0 )); $sth_branch->execute || die 'SQL_ERROR'; my $branch_list = []; while (my $branch = $sth_branch->fetchrow_hashref) { push @$branch_list, { BranchID => $branch->{BranchID}, Branch => $branch->{Branch} }; } use JSON; my $result = { total => scalar(@$branch_list), branch => $branch_list, success => JSON::true }; print $q->header(-content_type => "application/json; charset='utf8'); print to_json($result); exit; } At Dancer2, when I write below definition : use JSON; it does not allow with a warning that to_json and from_json functions are duplicated. I of course know that Dancer2 is using following definition set serializer => 'JSON'; But I still could not achieve returnin JSON data: Here my route is get '/json_branch_list' => sub { my $sth_branch = database->prepare (qq(SELECT BranchID,Branch FROM branch WHERE Deleted=0 )); $sth_branch->execute || die 'SQL_ERROR'; my $branch_list = []; while (my $branch = $sth_branch->fetchrow_hashref) { push @$branch_list, { BranchID => $branch->{BranchID}, Branch => $branch->{Branch} }; } set serializer => 'JSON'; return $branch_list; }; No data is displayed -- Kadir Beyazlı Computer Engineer GSM : +90 535 821 50 00
Hi All, I understood my mistake, last line of my code was: return $branch_list; but it should be return @\$branch_list; But I have one more problem, how I can I set character set of JSON data to utf8? I was doing it as follow as I wrote at my 1st mail print $q->header(-content_type => "application/json; charset='utf8'); Now, how I will do it? Also why am I unable to use : use JSON which is general way? On Sun, Sep 13, 2015 at 7:54 PM, Kadir Beyazlı <kadirbeyazli@gmail.com> wrote:
Hi All,
I will return JSON data from a route. If I don't use Dancer2 framework, I do it as follow and it works:
sub json_branch_list {
my $sth_branch = $dbh->prepare (qq(SELECT BranchID,Branch FROM branch WHERE Deleted=0 ));
$sth_branch->execute || die 'SQL_ERROR';
my $branch_list = [];
while (my $branch = $sth_branch->fetchrow_hashref) {
push @$branch_list, { BranchID => $branch->{BranchID}, Branch => $branch->{Branch} }; }
use JSON;
my $result = { total => scalar(@$branch_list), branch => $branch_list, success => JSON::true };
print $q->header(-content_type => "application/json; charset='utf8');
print to_json($result);
exit; }
At Dancer2, when I write below definition :
use JSON;
it does not allow with a warning that to_json and from_json functions are duplicated. I of course know that Dancer2 is using following definition
set serializer => 'JSON';
But I still could not achieve returnin JSON data: Here my route is
get '/json_branch_list' => sub {
my $sth_branch = database->prepare (qq(SELECT BranchID,Branch FROM branch WHERE Deleted=0 ));
$sth_branch->execute || die 'SQL_ERROR';
my $branch_list = [];
while (my $branch = $sth_branch->fetchrow_hashref) {
push @$branch_list, { BranchID => $branch->{BranchID}, Branch => $branch->{Branch} }; }
set serializer => 'JSON';
return $branch_list; };
No data is displayed
-- Kadir Beyazlı Computer Engineer GSM : +90 535 821 50 00
-- Kadir Beyazlı Computer Engineer GSM : +90 535 821 50 00
On 13/09/2015 17:54, Kadir Beyazlı wrote:
At Dancer2, when I write below definition :
use JSON;
it does not allow with a warning that to_json and from_json functions are duplicated. I of course know that Dancer2 is using following definition
set serializer => 'JSON';
But I still could not achieve returnin JSON data: Here my route is
get '/json_branch_list' => sub {
my $sth_branch = database->prepare (qq(SELECT BranchID,Branch FROM branch WHERE Deleted=0 ));
$sth_branch->execute || die 'SQL_ERROR';
my $branch_list = [];
while (my $branch = $sth_branch->fetchrow_hashref) {
push @$branch_list, { BranchID => $branch->{BranchID}, Branch => $branch->{Branch} }; }
set serializer => 'JSON';
return $branch_list; };
No data is displayed
What happens if you do: get '/json_branch_list' => sub { [...] my $ref = { BranchID => $branch->{BranchID}, Branch => $branch->{Branch} }; return to_json($ref); } Or even, depending on what is in $branch and what you want to do with it downstream just return the hashref as-is: return to_json($branch); As far I as I know there is no need for 'use JSON', just 'use Dancer2'. Works for me anyway. -- Richard Jones
Hi John, Now it is OK me. I am using as follow while (my $branch = $sth_branch->fetchrow_hashref) { push @$branch_list, { BranchID => $branch->{BranchID}, Branch => 'ali' }; } my $result = { total => scalar(@$branch_list), branch => $branch_list, success => 'true' }; set serializer => 'JSON'; return $result; If I use set serializer => 'JSON'; I dont need to use to_json as I understand. This is good for me I also understood why I dont need to define use JSON Because it is already ready for me. Thanks On Sun, Sep 13, 2015 at 8:29 PM, Richard Jones <ra.jones@dpw.clara.co.uk> wrote:
On 13/09/2015 17:54, Kadir Beyazlı wrote:
At Dancer2, when I write below definition :
use JSON;
it does not allow with a warning that to_json and from_json functions are duplicated. I of course know that Dancer2 is using following definition
set serializer => 'JSON';
But I still could not achieve returnin JSON data: Here my route is
get '/json_branch_list' => sub {
my $sth_branch = database->prepare (qq(SELECT BranchID,Branch FROM branch WHERE Deleted=0 ));
$sth_branch->execute || die 'SQL_ERROR';
my $branch_list = [];
while (my $branch = $sth_branch->fetchrow_hashref) {
push @$branch_list, { BranchID => $branch->{BranchID}, Branch => $branch->{Branch} }; }
set serializer => 'JSON';
return $branch_list; };
No data is displayed
What happens if you do:
get '/json_branch_list' => sub { [...] my $ref = { BranchID => $branch->{BranchID}, Branch => $branch->{Branch} }; return to_json($ref); }
Or even, depending on what is in $branch and what you want to do with it downstream just return the hashref as-is: return to_json($branch);
As far I as I know there is no need for 'use JSON', just 'use Dancer2'. Works for me anyway.
-- Richard Jones
_______________________________________________ dancer-users mailing list dancer-users@dancer.pm http://lists.preshweb.co.uk/mailman/listinfo/dancer-users
-- Kadir Beyazlı Computer Engineer GSM : +90 535 821 50 00
Hi John, Now my only problem is encoding JSON data. My data is Turkish, I was setting charset as follow to display correcty print $q->header(-content_type => "application/json; charset='utf8'); How will I do it at Dancer2? I used following 1st and 5th lines but did not solve problem 1 use Encode; 2 while (my $branch = $sth_branch->fetchrow_hashref) { 3 push @$branch_list, 4 { BranchID => $branch->{BranchID}, 5 Branch => encode_utf8($branch->{Branch}), }; } On Sun, Sep 13, 2015 at 8:43 PM, Kadir Beyazlı <kadirbeyazli@gmail.com> wrote:
Hi John,
Now it is OK me. I am using as follow
while (my $branch = $sth_branch->fetchrow_hashref) {
push @$branch_list, { BranchID => $branch->{BranchID}, Branch => 'ali' }; }
my $result = { total => scalar(@$branch_list), branch => $branch_list, success => 'true' };
set serializer => 'JSON'; return $result;
If I use
set serializer => 'JSON';
I dont need to use to_json as I understand. This is good for me
I also understood why I dont need to define
use JSON
Because it is already ready for me.
Thanks
On Sun, Sep 13, 2015 at 8:29 PM, Richard Jones <ra.jones@dpw.clara.co.uk> wrote:
On 13/09/2015 17:54, Kadir Beyazlı wrote:
At Dancer2, when I write below definition :
use JSON;
it does not allow with a warning that to_json and from_json functions are duplicated. I of course know that Dancer2 is using following definition
set serializer => 'JSON';
But I still could not achieve returnin JSON data: Here my route is
get '/json_branch_list' => sub {
my $sth_branch = database->prepare (qq(SELECT BranchID,Branch FROM branch WHERE Deleted=0 ));
$sth_branch->execute || die 'SQL_ERROR';
my $branch_list = [];
while (my $branch = $sth_branch->fetchrow_hashref) {
push @$branch_list, { BranchID => $branch->{BranchID}, Branch => $branch->{Branch} }; }
set serializer => 'JSON';
return $branch_list; };
No data is displayed
What happens if you do:
get '/json_branch_list' => sub { [...] my $ref = { BranchID => $branch->{BranchID}, Branch => $branch->{Branch} }; return to_json($ref); }
Or even, depending on what is in $branch and what you want to do with it downstream just return the hashref as-is: return to_json($branch);
As far I as I know there is no need for 'use JSON', just 'use Dancer2'. Works for me anyway.
-- Richard Jones
_______________________________________________ dancer-users mailing list dancer-users@dancer.pm http://lists.preshweb.co.uk/mailman/listinfo/dancer-users
-- Kadir Beyazlı Computer Engineer GSM : +90 535 821 50 00
-- Kadir Beyazlı Computer Engineer GSM : +90 535 821 50 00
I would try setting it in the config file. See http://search.cpan.org/~sukria/Dancer2-0.01/lib/Dancer2/Config.pod. There are YAML and JSON examples for setting UTF-8. --john On 9/13/2015 11:56 AM, Kadir Beyazlı wrote:
Hi John,
Now my only problem is encoding JSON data. My data is Turkish, I was setting charset as follow to display correcty
print $q->header(-content_type => "application/json; charset='utf8');
How will I do it at Dancer2?
I used following 1st and 5th lines but did not solve problem
1 use Encode; 2 while (my $branch = $sth_branch->fetchrow_hashref) {
3 push @$branch_list, 4 { BranchID => $branch->{BranchID}, 5 Branch => encode_utf8($branch->{Branch}), }; }
On Sun, Sep 13, 2015 at 8:43 PM, Kadir Beyazlı <kadirbeyazli@gmail.com> wrote:
Hi John,
Now it is OK me. I am using as follow
while (my $branch = $sth_branch->fetchrow_hashref) {
push @$branch_list, { BranchID => $branch->{BranchID}, Branch => 'ali' }; }
my $result = { total => scalar(@$branch_list), branch => $branch_list, success => 'true' };
set serializer => 'JSON'; return $result;
If I use
set serializer => 'JSON';
I dont need to use to_json as I understand. This is good for me
I also understood why I dont need to define
use JSON
Because it is already ready for me.
Thanks
On Sun, Sep 13, 2015 at 8:29 PM, Richard Jones <ra.jones@dpw.clara.co.uk> wrote:
On 13/09/2015 17:54, Kadir Beyazlı wrote:
At Dancer2, when I write below definition :
use JSON;
it does not allow with a warning that to_json and from_json functions are duplicated. I of course know that Dancer2 is using following definition
set serializer => 'JSON';
But I still could not achieve returnin JSON data: Here my route is
get '/json_branch_list' => sub {
my $sth_branch = database->prepare (qq(SELECT BranchID,Branch FROM branch WHERE Deleted=0 ));
$sth_branch->execute || die 'SQL_ERROR';
my $branch_list = [];
while (my $branch = $sth_branch->fetchrow_hashref) {
push @$branch_list, { BranchID => $branch->{BranchID}, Branch => $branch->{Branch} }; }
set serializer => 'JSON';
return $branch_list; };
No data is displayed
What happens if you do:
get '/json_branch_list' => sub { [...] my $ref = { BranchID => $branch->{BranchID}, Branch => $branch->{Branch} }; return to_json($ref); }
Or even, depending on what is in $branch and what you want to do with it downstream just return the hashref as-is: return to_json($branch);
As far I as I know there is no need for 'use JSON', just 'use Dancer2'. Works for me anyway.
-- Richard Jones
_______________________________________________ dancer-users mailing list dancer-users@dancer.pm http://lists.preshweb.co.uk/mailman/listinfo/dancer-users
-- Kadir Beyazlı Computer Engineer GSM : +90 535 821 50 00
-- John J. McDermott, CPLP Learning and Performance Consultant jjm at jkintl.com 575/737-8556 Check out my security blog posts Add an A for the Arts To STEM and get STEAM and a strong engine to move forward.
On 14/09/2015 4:32 am, John McDermott, CPLP wrote:
I would try setting it in the config file. See http://search.cpan.org/~sukria/Dancer2-0.01/lib/Dancer2/Config.pod. There are YAML and JSON examples for setting UTF-8.
That version of Dancer2::Config is definitely outdated. It's rarely a good idea to link to CPAN resources using a specific version number. Better to use: https://metacpan.org/pod/distribution/Dancer2/lib/Dancer2/Config.pod Which will always give you the latest version. Regards, Russell. ( with thanks to Dave Cross !)
You're right. I need to update this laptop's links. Sorry. --john On 9/13/2015 5:26 PM, Russell Jenkins wrote:
On 14/09/2015 4:32 am, John McDermott, CPLP wrote:
I would try setting it in the config file. See http://search.cpan.org/~sukria/Dancer2-0.01/lib/Dancer2/Config.pod. There are YAML and JSON examples for setting UTF-8.
That version of Dancer2::Config is definitely outdated.
It's rarely a good idea to link to CPAN resources using a specific version number. Better to use:
https://metacpan.org/pod/distribution/Dancer2/lib/Dancer2/Config.pod
Which will always give you the latest version.
Regards, Russell.
( with thanks to Dave Cross !) _______________________________________________ dancer-users mailing list dancer-users@dancer.pm http://lists.preshweb.co.uk/mailman/listinfo/dancer-users
-- John J. McDermott, CPLP Learning and Performance Consultant jjm at jkintl.com 575/737-8556 Check out my security blog posts Add an A for the Arts To STEM and get STEAM and a strong engine to move forward.
On 14/09/2015 3:56 am, Kadir Beyazlı wrote:
Now my only problem is encoding JSON data. My data is Turkish, I was setting charset as follow to display correcty
print $q->header(-content_type => "application/json; charset='utf8');
How will I do it at Dancer2?
Hey Kadir. You do no need to specify your JSON content as being UTF-8. RFC4627 states: "JSON text SHALL be encoded in Unicode. The default encoding is UTF-8" and goes on to explain how to determine the encoding of the octet stream. Furthermore, the IANA assignment for 'application/json' includes the following note: "No "charset" parameter is defined for this registration. Adding one really has no effect on compliant recipients." Hope that helps, Russell.
Hi Russel, On Mon, Sep 14, 2015 at 2:19 AM, Russell Jenkins <russell.jenkins@strategicdata.com.au> wrote:
On 14/09/2015 3:56 am, Kadir Beyazlı wrote:
Now my only problem is encoding JSON data. My data is Turkish, I was setting charset as follow to display correcty
print $q->header(-content_type => "application/json; charset='utf8');
How will I do it at Dancer2?
Hey Kadir.
You do no need to specify your JSON content as being UTF-8.
RFC4627 states: "JSON text SHALL be encoded in Unicode. The default encoding is UTF-8" and goes on to explain how to determine the encoding of the octet stream. [KB] I did not specify UTF-8, but my display problem for Turkish characters is still going on. I am returning JSON data from file myproject.pm and using JSON data at file report.js. My files (myproject.pm and report.js) character sets are also UTF-8. At old method, when I set content to UTF-8, it was solved
Furthermore, the IANA assignment for 'application/json' includes the following note: "No "charset" parameter is defined for this registration. Adding one really has no effect on compliant recipients."
Hope that helps, Russell.
_______________________________________________ dancer-users mailing list dancer-users@dancer.pm http://lists.preshweb.co.uk/mailman/listinfo/dancer-users
-- Kadir Beyazlı Computer Engineer GSM : +90 535 821 50 00
2015-09-14 10:19 GMT+03:00 Kadir Beyazlı <kadirbeyazli@gmail.com>:
[KB] I did not specify UTF-8, but my display problem for Turkish characters is still going on. I am returning JSON data from file myproject.pm and using JSON data at file report.js. My files (myproject.pm and report.js) character sets are also UTF-8. At old method, when I set content to UTF-8, it was solved
You could look with `curl`, are the headers wrong or data not properly encoded : curl -v http://mysite.com/json_route wbr, -- Kõike hääd, Gunnar
Hi WK, On Mon, Sep 14, 2015 at 12:02 PM, WK <wanradt@gmail.com> wrote:
2015-09-14 10:19 GMT+03:00 Kadir Beyazlı <kadirbeyazli@gmail.com>:
[KB] I did not specify UTF-8, but my display problem for Turkish characters is still going on. I am returning JSON data from file myproject.pm and using JSON data at file report.js. My files (myproject.pm and report.js) character sets are also UTF-8. At old method, when I set content to UTF-8, it was solved
You could look with `curl`, are the headers wrong or data not properly encoded :
curl -v http://mysite.com/json_route [KB] I checked, it is UTF-8. But my Turkish characters are still broken. My country's character set is iso-8859-9 but UTF8 was always enough to display correct data. I will search about it. Thanks
wbr, -- Kõike hääd,
Gunnar _______________________________________________ dancer-users mailing list dancer-users@dancer.pm http://lists.preshweb.co.uk/mailman/listinfo/dancer-users
-- Kadir Beyazlı Computer Engineer GSM : +90 535 821 50 00
2015-09-14 15:44 GMT+03:00 Kadir Beyazlı <kadirbeyazli@gmail.com>:
[KB] I checked, it is UTF-8. But my Turkish characters are still broken. My country's character set is iso-8859-9 but UTF8 was always enough to display correct data. I will search about it.
I had noticed that if I have set STDOUT to have utf-8 encoded (as does Dancer) and if JSON is set to use utf-8 (as seems case in Dancer), I end up with twice encoded data. So, try like this now: return to_json( $ref, { utf8 => 0 } ); Wbr, -- Kõike hääd, Gunnar
Hi WK, On Mon, Sep 14, 2015 at 4:29 PM, WK <wanradt@gmail.com> wrote:
2015-09-14 15:44 GMT+03:00 Kadir Beyazlı <kadirbeyazli@gmail.com>:
[KB] I checked, it is UTF-8. But my Turkish characters are still broken. My country's character set is iso-8859-9 but UTF8 was always enough to display correct data. I will search about it.
I had noticed that if I have set STDOUT to have utf-8 encoded (as does Dancer) and if JSON is set to use utf-8 (as seems case in Dancer), I end up with twice encoded data. So, try like this now:
return to_json( $ref, { utf8 => 0 } ); [KB] It worked! You are genius!
Wbr, -- Kõike hääd,
Gunnar _______________________________________________ dancer-users mailing list dancer-users@dancer.pm http://lists.preshweb.co.uk/mailman/listinfo/dancer-users
-- Kadir Beyazlı Computer Engineer GSM : +90 535 821 50 00
2015-09-14 15:44 GMT+03:00 Kadir Beyazlı <kadirbeyazli@gmail.com>:
[KB] I checked, it is UTF-8. But my Turkish characters are still broken. My country's character set is iso-8859-9 but UTF8 was always enough to display correct data. I will search about it.
Actually, right way to emit JSON is to use serializer, so the right way is something like that: ======= use Dancer2; set serializer => 'JSON'; get '/json' => sub { my $ref = { BranchID => 'ülane kõnnib öösiti õues', Branch => 'šaakal kasutab ainult žiletti', }; return $ref; }; start; ======= If you don't use serializer, you have wrong content-type. If you set content-type by hand, you have wrong content-lenght, etc Wbr, -- Kõike hääd, Gunnar
Hi, On Mon, Sep 14, 2015 at 5:03 PM, WK <wanradt@gmail.com> wrote:
2015-09-14 15:44 GMT+03:00 Kadir Beyazlı <kadirbeyazli@gmail.com>:
[KB] I checked, it is UTF-8. But my Turkish characters are still broken. My country's character set is iso-8859-9 but UTF8 was always enough to display correct data. I will search about it.
Actually, right way to emit JSON is to use serializer, so the right way is something like that:
======= use Dancer2; set serializer => 'JSON';
get '/json' => sub { my $ref = { BranchID => 'ülane kõnnib öösiti õues', Branch => 'šaakal kasutab ainult žiletti', }; return $ref; };
start; =======
If you don't use serializer, you have wrong content-type. If you set content-type by hand, you have wrong content-lenght, etc content-type is text/html at both cases (with and without serializer), not application/json when i curl url. But it works And when I use serializer, other routes fail because other routes don't return json data. Even if I define serializer in the route which will return json, it effects all routes
Wbr, -- Kõike hääd,
Gunnar _______________________________________________ dancer-users mailing list dancer-users@dancer.pm http://lists.preshweb.co.uk/mailman/listinfo/dancer-users
-- Kadir Beyazlı Computer Engineer GSM : +90 535 821 50 00
2015-09-15 0:18 GMT+03:00 Kadir Beyazlı <kadirbeyazli@gmail.com>:
content-type is text/html at both cases (with and without serializer), not application/json when i curl url.
It is very strange. If you use my example, save it as script and run it, you got content type "text/html"?
But it works And when I use serializer, other routes fail because other routes don't return json data.
You need to run 2 applications: one for html and other for serialized data (JSON). Sawyer X wrote wonderful article about it in last advent calendar: http://advent.perldancer.org/2014/11 It is pretty easy and I assure you, it gives right content types too ;) Wbr, -- Kõike hääd, Gunnar
Hi WK, On Tue, Sep 15, 2015 at 1:05 AM, WK <wanradt@gmail.com> wrote:
2015-09-15 0:18 GMT+03:00 Kadir Beyazlı <kadirbeyazli@gmail.com>:
content-type is text/html at both cases (with and without serializer), not application/json when i curl url.
It is very strange. If you use my example, save it as script and run it, you got content type "text/html"?
But it works And when I use serializer, other routes fail because other routes don't return json data.
You need to run 2 applications: one for html and other for serialized data (JSON). Sawyer X wrote wonderful article about it in last advent calendar:
http://advent.perldancer.org/2014/11
It is pretty easy and I assure you, it gives right content types too ;)
Thank you very much! Everything is OK now.
Wbr, -- Kõike hääd,
Gunnar _______________________________________________ dancer-users mailing list dancer-users@dancer.pm http://lists.preshweb.co.uk/mailman/listinfo/dancer-users
-- Kadir Beyazlı Computer Engineer GSM : +90 535 821 50 00
It seems that there is still a problem with Dancer2 serializing JSON output and double encoding UTF-8 data. I’ve thoroughly read: https://github.com/PerlDancer/Dancer2/issues/686 https://github.com/PerlDancer/Dancer2/pull/520 Setting the options in 520 results in: "Body must be bytes and should not contain wide characters (UTF-8 strings)” error https://github.com/PerlDancer/Dancer2/pull/647 I guess that since #686 is still open that there is no work around for those of us who need to use serializer: JSON and have UTF-8 encoded data coming from our database? -Robert
On Sep 14, 2015, at 4:19 PM, Kadir Beyazlı <kadirbeyazli@gmail.com> wrote:
Hi Russel,
On Mon, Sep 14, 2015 at 2:19 AM, Russell Jenkins <russell.jenkins@strategicdata.com.au> wrote:
On 14/09/2015 3:56 am, Kadir Beyazlı wrote:
Now my only problem is encoding JSON data. My data is Turkish, I was setting charset as follow to display correcty
print $q->header(-content_type => "application/json; charset='utf8');
How will I do it at Dancer2?
Hey Kadir.
You do no need to specify your JSON content as being UTF-8.
RFC4627 states: "JSON text SHALL be encoded in Unicode. The default encoding is UTF-8" and goes on to explain how to determine the encoding of the octet stream. [KB] I did not specify UTF-8, but my display problem for Turkish characters is still going on. I am returning JSON data from file myproject.pm and using JSON data at file report.js. My files (myproject.pm and report.js) character sets are also UTF-8. At old method, when I set content to UTF-8, it was solved
Furthermore, the IANA assignment for 'application/json' includes the following note: "No "charset" parameter is defined for this registration. Adding one really has no effect on compliant recipients."
Hope that helps, Russell.
_______________________________________________ dancer-users mailing list dancer-users@dancer.pm http://lists.preshweb.co.uk/mailman/listinfo/dancer-users
-- Kadir Beyazlı Computer Engineer GSM : +90 535 821 50 00 _______________________________________________ dancer-users mailing list dancer-users@dancer.pm http://lists.preshweb.co.uk/mailman/listinfo/dancer-users
JSON is enabled in config.yml serializer: "JSON" engines: serializer: JSON: pretty: 1 allow_blessed: 1 canonical: 0 utf8: 1 YAML is working fine as a serializer using the the following method: JSON prints mojibake when using the serializer using the following method: get '/establishments' => sub { my $sql = 'select * from establishments'; my $sth = database->prepare($sql) or die database->errstr; $sth->execute or die $sth->errstr; #the following is broken in Dancer2 return \%{$sth->fetchall_hashref('id')}; }; JSON is disabled in config.yml #serializer: "JSON" engines: serializer: JSON: pretty: 1 allow_blessed: 1 canonical: 0 utf8: 1 returns fine when disabling the serializer JSON in the config.yml file. get '/establishments' => sub { my $sql = 'select * from establishments'; my $sth = database->prepare($sql) or die database->errstr; $sth->execute or die $sth->errstr; return to_json(\%{$sth->fetchall_hashref('id')}, { utf8 => 0 }); }; My question is, is this still the expected behavior due to issue #686 still being open? Can I contribute monetarily to help get this issue fixed ASAP? -Robert
On Sep 27, 2015, at 5:05 AM, Robert Smith <spamfree@wansecurity.com> wrote:
It seems that there is still a problem with Dancer2 serializing JSON output and double encoding UTF-8 data.
I’ve thoroughly read:
https://github.com/PerlDancer/Dancer2/issues/686
https://github.com/PerlDancer/Dancer2/pull/520
Setting the options in 520 results in: "Body must be bytes and should not contain wide characters (UTF-8 strings)” error
https://github.com/PerlDancer/Dancer2/pull/647
I guess that since #686 is still open that there is no work around for those of us who need to use serializer: JSON and have UTF-8 encoded data coming from our database?
-Robert
On Sep 14, 2015, at 4:19 PM, Kadir Beyazlı <kadirbeyazli@gmail.com> wrote:
Hi Russel,
On Mon, Sep 14, 2015 at 2:19 AM, Russell Jenkins <russell.jenkins@strategicdata.com.au> wrote:
On 14/09/2015 3:56 am, Kadir Beyazlı wrote:
Now my only problem is encoding JSON data. My data is Turkish, I was setting charset as follow to display correcty
print $q->header(-content_type => "application/json; charset='utf8');
How will I do it at Dancer2?
Hey Kadir.
You do no need to specify your JSON content as being UTF-8.
RFC4627 states: "JSON text SHALL be encoded in Unicode. The default encoding is UTF-8" and goes on to explain how to determine the encoding of the octet stream. [KB] I did not specify UTF-8, but my display problem for Turkish characters is still going on. I am returning JSON data from file myproject.pm and using JSON data at file report.js. My files (myproject.pm and report.js) character sets are also UTF-8. At old method, when I set content to UTF-8, it was solved
Furthermore, the IANA assignment for 'application/json' includes the following note: "No "charset" parameter is defined for this registration. Adding one really has no effect on compliant recipients."
Hope that helps, Russell.
_______________________________________________ dancer-users mailing list dancer-users@dancer.pm http://lists.preshweb.co.uk/mailman/listinfo/dancer-users
-- Kadir Beyazlı Computer Engineer GSM : +90 535 821 50 00 _______________________________________________ dancer-users mailing list dancer-users@dancer.pm http://lists.preshweb.co.uk/mailman/listinfo/dancer-users
_______________________________________________ dancer-users mailing list dancer-users@dancer.pm http://lists.preshweb.co.uk/mailman/listinfo/dancer-users
On 27/09/2015 11:54 am, Robert Smith wrote:
JSON is enabled in config.yml
serializer: "JSON" engines: serializer: JSON: pretty: 1 allow_blessed: 1 canonical: 0 utf8: 1
YAML is working fine as a serializer using the the following method: JSON prints mojibake when using the serializer using the following method:
get '/establishments' => sub { my $sql = 'select * from establishments'; my $sth = database->prepare($sql) or die database->errstr; $sth->execute or die $sth->errstr; #the following is broken in Dancer2 return \%{$sth->fetchall_hashref('id')}; };
Provided you get characters (not octets) back from the database the above "just works". If you are getting octets back they will get encoded again (with their default serializer params). All Dancer2 serializers expect characters and return octets for adding to the response body. Note the #686 is about a different issue; the to_json, to_yaml and to_dumper keywords. Their implementation also returns octets, so when those keywords are used as the return form a route the response is double encoded. Hope that helps, Russell.
Dear Russel, please see additional information below. I really appreciate your help.
On Sep 27, 2015, at 10:41 PM, Russell Jenkins <russell.jenkins@strategicdata.com.au> wrote:
On 27/09/2015 11:54 am, Robert Smith wrote:
JSON is enabled in config.yml
serializer: "JSON" engines: serializer: JSON: pretty: 1 allow_blessed: 1 canonical: 0 utf8: 1
YAML is working fine as a serializer using the the following method: JSON prints mojibake when using the serializer using the following method:
get '/establishments' => sub { my $sql = 'select * from establishments'; my $sth = database->prepare($sql) or die database->errstr; $sth->execute or die $sth->errstr; #the following is broken in Dancer2 return \%{$sth->fetchall_hashref('id')}; };
Provided you get characters (not octets) back from the database the above "just works". If you are getting octets back they will get encoded again (with their default serializer params). All Dancer2 serializers expect characters and return octets for adding to the response body.
It doesn’t "just work” however it seems. If this was true, shouldn’t the YAML serializer be having the same issue? Whatever my database is returning works perfectly fine with the YAML serializer, but not the JSON serializer. Here is my config.yml appname: "BeaconTrackerAPI" layout: "main" charset: "UTF-8" serializer: "JSON" engines: serializer: JSON: pretty: 1 allow_blessed: 1 canonical: 0 # ascii: 1 #utf8: 1 template: "simple" plugins: Database: driver: 'Pg' database: ‘***************' host: 'localhost' port: 5432 username: ‘*******' password: '********' connection_check_threshold: 10 on_connect_do: ["SET NAMES 'utf8'"] Here is the method used in both the JSON and YAML tests: get '/establishments' => sub { my $sql = 'select * from establishments'; my $sth = database->prepare($sql) or die database->errstr; $sth->execute or die $sth->errstr; #return to_json(\%{$sth->fetchall_hashref('id')}, { utf8 => 0 }); #the following is broken in Dancer2 serializer return \%{$sth->fetchall_hashref('id')}; }; Here is the output to the browser (broken): { "1" : { "client_id" : "1123", "name" : "WANSecurity, K.K.", "address" : "æ±äº¬éƒ½æ¸‹è°·åŒºä¸ŠåŽŸï¼’−４８−１２ ã¨ã‚ˆä»£ã€…木上原コーãƒï¼ƒï¼’ï¼ï¼‘", "id" : "1" } } Here is output from my terminal via psql (Looks good): beacon_tracker=> select * from establishments; id | name | address | client_id ----+-------------------+-----------------------------------------------------------+----------- 1 | WANSecurity, K.K. | 東京都渋谷区上原２−４８ | 1123 Is this the expected behavior? Here is the output from YAML when using the YAML serializer (Looks good): --- 1: address: 東京都渋谷区上原２−４８ client_id: 1123 id: 1 name: 'WANSecurity, K.K.' YAML looks exactly like what I would expect to see, right out of the box. Something strange is happening with the JSON serializer it seems. Am I missing anything? -Robert
Note the #686 is about a different issue; the to_json, to_yaml and to_dumper keywords. Their implementation also returns octets, so when those keywords are used as the return form a route the response is double encoded.
Hope that helps, Russell.
_______________________________________________ dancer-users mailing list dancer-users@dancer.pm http://lists.preshweb.co.uk/mailman/listinfo/dancer-users
2015-09-28 18:54 GMT+03:00 Robert Smith <spamfree@wansecurity.com>:
Here is the output to the browser (broken):
{ "1" : { "client_id" : "1123", "name" : "WANSecurity, K.K.", "address" : "æ ±äº¬éƒ½æ¸‹è°·åŒºä¸ŠåŽŸï¼’âˆ’ï¼”ï¼˜âˆ’ï¼‘ï¼’ ã ¨ã‚ˆä»£ã€…æœ¨ä¸ŠåŽŸã‚³ãƒ¼ãƒ ï¼ƒï¼’ï¼ ï¼‘", "id" : "1" } }
...
Is this the expected behavior?
I think, it is. I am almost sure, your data is encoded correctly, but browser does not show it correctly. 'application/json' should be always encoded in utf-8, but browser may assume it being Latin-1 (or iso8859-1). You could try requesting with curl and look, what you get. I got correct output with your conf. And you may read about same issue here too: https://github.com/PerlDancer/Dancer2/issues/995 If you need it to show up correctly in browser too, you may set header yourself "Content-type: application/json;charset=utf-8" Wbr, -- Kõike hääd, Gunnar
Hi Robert. Sorry for this terse reply, I've gotta run. On 29/09/2015 1:54 am, Robert Smith wrote:
It doesn’t "just work” however it seems.
Cut it down to a basic test case. eg: === use Dancer2; set serializer => 'JSON'; get '/' => sub { return +{ 1 => { address=> '東京都渋谷区上原２−４８', client_id => '1123', id => '1', name => 'WANSecurity, K.K.', }, }; }; dance; === Save to a file, and run with plackup. Again, you need to ensure you get *characters* back from the database. Please read the DBD::Pg docs on the pg_enable_utf8 connection attribute.. Cheers, Russell.
Wow you were so right on with this! Who’d have guessed that mozilla would try to display JSON in latin charset!? Thank you for your support. Please let me know where I can make a donation to the project. -Robert
On Sep 29, 2015, at 4:31 PM, Russell Jenkins <russell.jenkins@strategicdata.com.au> wrote:
Hi Robert.
Sorry for this terse reply, I've gotta run.
On 29/09/2015 1:54 am, Robert Smith wrote:
It doesn’t "just work” however it seems.
Cut it down to a basic test case. eg:
=== use Dancer2; set serializer => 'JSON'; get '/' => sub { return +{ 1 => { address=> '東京都渋谷区上原２−４８', client_id => '1123', id => '1', name => 'WANSecurity, K.K.', }, }; }; dance; ===
Save to a file, and run with plackup.
Again, you need to ensure you get *characters* back from the database. Please read the DBD::Pg docs on the pg_enable_utf8 connection attribute..
Cheers, Russell.
_______________________________________________ dancer-users mailing list dancer-users@dancer.pm http://lists.preshweb.co.uk/mailman/listinfo/dancer-users
On Sep 26, 2015 1:05 PM, "Robert Smith" <spamfree@wansecurity.com> wrote:
I guess that since #686 is still open that there is no work around for those of us who need to use serializer: JSON and have UTF-8 encoded data coming from our database?
Use ascii JSON encoding? I've never been fond of utf8 JSON; making it ascii is usually only trivially increasing the length but doing away with all transport encoding issues. I guess it might make a difference if you needed unicode characters not supported in javascript/JSON, though.
I'm storing Chinese, Japanese and Korean characters in a UTF8 encoded database. I thought that JS and by extension JSON natively deal with Unicode characters. In fact I'm sure it I've never had any problems with JS in this matter. It would be nice to have an extension to the JSON serializer like no_encode : 1 I don't understand what the problem with the double wide character check in Lint is all about. When setting utf8 : 0 What's the problem with that? -Robert Sent from my iPhone
On Sep 27, 2015, at 1:40 PM, Yitzchak Scott-Thoennes <sthoenna@gmail.com> wrote:
On Sep 26, 2015 1:05 PM, "Robert Smith" <spamfree@wansecurity.com> wrote:
I guess that since #686 is still open that there is no work around for those of us who need to use serializer: JSON and have UTF-8 encoded data coming from our database?
Use ascii JSON encoding?
I've never been fond of utf8 JSON; making it ascii is usually only trivially increasing the length but doing away with all transport encoding issues.
I guess it might make a difference if you needed unicode characters not supported in javascript/JSON, though. _______________________________________________ dancer-users mailing list dancer-users@dancer.pm http://lists.preshweb.co.uk/mailman/listinfo/dancer-users
participants (7)
-
John McDermott, CPLP -
Kadir Beyazlı -
Richard Jones -
Robert Smith -
Russell Jenkins -
WK -
Yitzchak Scott-Thoennes