Dear dancers, I have an openldap server and I do some search in there. I wrote a perl module to handle the basic operations in the ldap. So when I search for for example to my displayName, the ldap modul found the following: battila@igornb:~$ ./test_ldap.pl LdapPerson module is successfully loaded! displayname: Bárdi Attila battila@igornb:~$ test.pl: *#!/usr/bin/perl -wuse strict;use warnings;use LdapPerson;my $somebody = LdapPerson->new();$somebody = LdapPerson->search('attila.bardi@example.com <attila.bardi@example.com>');print "displayname: " . $somebody->get_displayName() . "\n"* But with: *use Dancer ':syntax';* the "*Bárdi Attila*" become "*Bárdi Attila*" package MyWeb::App; use Dancer ':syntax'; use LdapPerson; our $VERSION = '0.1'; get '/' => sub { my $somebody = LdapPerson->new(); $somebody = LdapPerson->search('attila.bardi@example.com'); return $somebody->get_displayName(); }; true; from config.yml: *# when the charset is set to UTF-8 Dancer will handle for you# all the magic of encoding and decoding. You should not care# about unicode within your app when this setting is set (recommended).charset: "UTF-8"* Can anyone help me what is the problem and how can I solve it? Best regrds, Attila
On 12/18/2013 03:03 PM, Attila Bárdi wrote:
Dear dancers,
I have an openldap server and I do some search in there. I wrote a perl module to handle the basic operations in the ldap. So when I search for for example to my displayName, the ldap modul found the following:
battila@igornb:~$ ./test_ldap.pl LdapPerson module is successfully loaded! displayname: Bárdi Attila battila@igornb:~$
test.pl:
*#!/usr/bin/perl -wuse strict;use warnings;use LdapPerson;my $somebody = LdapPerson->new();$somebody = LdapPerson->search('attila.bardi@example.com <attila.bardi@example.com>');print "displayname: " . $somebody->get_displayName() . "\n"*
But with: *use Dancer ':syntax';*
the "*Bárdi Attila*" become "*Bárdi Attila*"
package MyWeb::App; use Dancer ':syntax'; use LdapPerson;
our $VERSION = '0.1';
get '/' => sub { my $somebody = LdapPerson->new(); $somebody = LdapPerson->search('attila.bardi@example.com'); return $somebody->get_displayName(); };
true;
from config.yml:
*# when the charset is set to UTF-8 Dancer will handle for you# all the magic of encoding and decoding. You should not care# about unicode within your app when this setting is set (recommended).charset: "UTF-8"*
Can anyone help me what is the problem and how can I solve it?
LDAP + UTF-8 seems to be tricky, as I learned from writing Dancer::Plugin::LDAP. Maybe you can find some wisdom in its source code? Regards Racke -- LinuXia Systems => http://www.linuxia.de/ Expert Interchange Consulting and System Administration ICDEVGROUP => http://www.icdevgroup.org/ Interchange Development Team
Hey Stefan, as you see the the ldap gives me the correct result, till the Dancer module is not loaded. I think it is a Dancer error not an ldap. I created a more simple test case: battila@igornb:~$ dancer -a MyWeb::App ... battila@igornb:~$ cd MyWeb-App battila@igornb:~/MyWeb-App$ vi lib/ChrachterMistake.pm #!/usr/bin/perl -w package ChrachterMistake; use strict; use warnings; sub gimmySomething { return "Bárdi Attila"; } 1; battila@igornb:~/MyWeb-App$ vi lib/MyWeb/App.pm package MyWeb::App; use Dancer ':syntax'; use ChrachterMistake; our $VERSION = '0.1'; get '/' => sub { return ChrachterMistake::gimmySomething(); }; true; battila@igornb:~/MyWeb-App$ perl ./bin/app.pl [9950] core @0.000010> loading Dancer::Handler::Standalone handler in /usr/share/perl5/Dancer/Handler.pm l. 45 [9950] core @0.000181> loading handler 'Dancer::Handler::Standalone' in /usr/share/perl5/Dancer.pm l. 474
Dancer 1.311 server 9950 listening on http://0.0.0.0:3000 == Entering the development dance floor ...
With browser I connect to the http://localhost:3000/ and the result is: Bárdi Attila The Dancer does something with the return value of the function. No ldap in the chain at all. Best regards, Attila On Wed, Dec 18, 2013 at 4:31 PM, Stefan Hornburg (Racke) <racke@linuxia.de>wrote:
On 12/18/2013 03:03 PM, Attila Bárdi wrote:
Dear dancers,
I have an openldap server and I do some search in there. I wrote a perl module to handle the basic operations in the ldap. So when I search for for example to my displayName, the ldap modul found the following:
battila@igornb:~$ ./test_ldap.pl LdapPerson module is successfully loaded! displayname: Bárdi Attila battila@igornb:~$
test.pl:
*#!/usr/bin/perl -wuse strict;use warnings;use LdapPerson;my $somebody = LdapPerson->new();$somebody = LdapPerson->search(' attila.bardi@example.com <attila.bardi@example.com>');print "displayname: " . $somebody->get_displayName() . "\n"*
But with: *use Dancer ':syntax';*
the "*Bárdi Attila*" become "*Bárdi Attila*"
package MyWeb::App; use Dancer ':syntax'; use LdapPerson;
our $VERSION = '0.1';
get '/' => sub { my $somebody = LdapPerson->new(); $somebody = LdapPerson->search('attila.bardi@example.com'); return $somebody->get_displayName(); };
true;
from config.yml:
*# when the charset is set to UTF-8 Dancer will handle for you# all the magic of encoding and decoding. You should not care# about unicode within your app when this setting is set (recommended).charset: "UTF-8"*
Can anyone help me what is the problem and how can I solve it?
LDAP + UTF-8 seems to be tricky, as I learned from writing Dancer::Plugin::LDAP. Maybe you can find some wisdom in its source code?
Regards Racke
-- LinuXia Systems => http://www.linuxia.de/ Expert Interchange Consulting and System Administration ICDEVGROUP => http://www.icdevgroup.org/ Interchange Development Team
_______________________________________________ dancer-users mailing list dancer-users@dancer.pm http://lists.preshweb.co.uk/mailman/listinfo/dancer-users
On 12/18/2013 05:04 PM, Attila Bárdi wrote:
Hey Stefan,
as you see the the ldap gives me the correct result, till the Dancer module is not loaded. I think it is a Dancer error not an ldap.
I created a more simple test case:
battila@igornb:~$ dancer -a MyWeb::App ... battila@igornb:~$ cd MyWeb-App battila@igornb:~/MyWeb-App$ vi lib/ChrachterMistake.pm
#!/usr/bin/perl -w package ChrachterMistake; use strict; use warnings;
sub gimmySomething { return "Bárdi Attila"; }
1;
battila@igornb:~/MyWeb-App$ vi lib/MyWeb/App.pm
package MyWeb::App; use Dancer ':syntax'; use ChrachterMistake;
our $VERSION = '0.1';
get '/' => sub { return ChrachterMistake::gimmySomething(); };
true;
You have to use use utf8; if you put hardcoded UTF-8 strings into your module. Regards Racke -- LinuXia Systems => http://www.linuxia.de/ Expert Interchange Consulting and System Administration ICDEVGROUP => http://www.icdevgroup.org/ Interchange Development Team
Hey Stefan, you are right! Danke! I tried the* use urf8;* with ldap and it did not work. I will check the source of *Dancer::Plugin::LDAP.* Best regards, Attila On Wed, Dec 18, 2013 at 5:09 PM, Stefan Hornburg (Racke) <racke@linuxia.de>wrote:
On 12/18/2013 05:04 PM, Attila Bárdi wrote:
Hey Stefan,
as you see the the ldap gives me the correct result, till the Dancer module is not loaded. I think it is a Dancer error not an ldap.
I created a more simple test case:
battila@igornb:~$ dancer -a MyWeb::App ... battila@igornb:~$ cd MyWeb-App battila@igornb:~/MyWeb-App$ vi lib/ChrachterMistake.pm
#!/usr/bin/perl -w package ChrachterMistake; use strict; use warnings;
sub gimmySomething { return "Bárdi Attila"; }
1;
battila@igornb:~/MyWeb-App$ vi lib/MyWeb/App.pm
package MyWeb::App; use Dancer ':syntax'; use ChrachterMistake;
our $VERSION = '0.1';
get '/' => sub { return ChrachterMistake::gimmySomething(); };
true;
You have to use
use utf8;
if you put hardcoded UTF-8 strings into your module.
Regards Racke
-- LinuXia Systems => http://www.linuxia.de/ Expert Interchange Consulting and System Administration ICDEVGROUP => http://www.icdevgroup.org/ Interchange Development Team
_______________________________________________ dancer-users mailing list dancer-users@dancer.pm http://lists.preshweb.co.uk/mailman/listinfo/dancer-users
Hey Stefan, unfortunately I have not found anything related to my problem in your source. Best regards, Attila On Wed, Dec 18, 2013 at 7:01 PM, Attila Bárdi <attila.bardi@gmail.com>wrote:
Hey Stefan,
you are right! Danke!
I tried the* use urf8;* with ldap and it did not work.
I will check the source of *Dancer::Plugin::LDAP.*
Best regards, Attila
On Wed, Dec 18, 2013 at 5:09 PM, Stefan Hornburg (Racke) <racke@linuxia.de
wrote:
On 12/18/2013 05:04 PM, Attila Bárdi wrote:
Hey Stefan,
as you see the the ldap gives me the correct result, till the Dancer module is not loaded. I think it is a Dancer error not an ldap.
I created a more simple test case:
battila@igornb:~$ dancer -a MyWeb::App ... battila@igornb:~$ cd MyWeb-App battila@igornb:~/MyWeb-App$ vi lib/ChrachterMistake.pm
#!/usr/bin/perl -w package ChrachterMistake; use strict; use warnings;
sub gimmySomething { return "Bárdi Attila"; }
1;
battila@igornb:~/MyWeb-App$ vi lib/MyWeb/App.pm
package MyWeb::App; use Dancer ':syntax'; use ChrachterMistake;
our $VERSION = '0.1';
get '/' => sub { return ChrachterMistake::gimmySomething(); };
true;
You have to use
use utf8;
if you put hardcoded UTF-8 strings into your module.
Regards Racke
-- LinuXia Systems => http://www.linuxia.de/ Expert Interchange Consulting and System Administration ICDEVGROUP => http://www.icdevgroup.org/ Interchange Development Team
_______________________________________________ dancer-users mailing list dancer-users@dancer.pm http://lists.preshweb.co.uk/mailman/listinfo/dancer-users
2013/12/18 Attila Bárdi <attila.bardi@gmail.com>
I have an openldap server and I do some search in there.
So when I search for for example to my displayName, the ldap modul found the following:
battila@igornb:~$ ./test_ldap.pl LdapPerson module is successfully loaded! displayname: Bárdi Attila battila@igornb:~$
test.pl:
*#!/usr/bin/perl -wuse strict;use warnings;use LdapPerson;*
Don't you get any "wide character" warnings? As far as I understand you have problem with decoding your input from LDAP. When you use your module without Dancer you seem not decode your input and Perl treats your input by bytes and on print it outputs these same bytes. So, when you have UTF-8 enabled environment those bytes are shown as right characters even when they are not meant as such. Now, using Dancer you tell explicitly to use UTF-8 and because your input is not decoded properly (not marked as UTF-8) Dancer will treat this string as Latin1 and decodes bytes after ASCII-set again to its inner Unicode and on printing again to UTF-8. This causes such output as you have. You could add to your test.pl one line to test your output filters: print join( ' ', PerlIO::get_layers('STDOUT') ), "\n"; If the last filter is not "utf8", my conclusion should be right. -- Wbr, Kõike hääd, Gunnar
Hey, battila@igornb:~/Dancer/MyWebApp/lib$ ./test_ldap.pl LdapPerson module is successfully loaded! displayname: Bárdi Attila unix perlio battila@igornb:~/Dancer/MyWebApp/lib$ Hmml, there is no utf8. I found the following while I search for the solution and before I read you e-mail: the ldap does not set the utf8 flag (I don't know why) so I tried to ask it to set the flag. I used the *raw => qr/(?i:^jpegPhoto|;binary)/ *option at the bind, but it just confused the thing, because it converted to utf8. I also tried to use several utf8, unicode modules without success. But I don't know the solution. On Wed, Dec 18, 2013 at 10:00 PM, WK <wanradt@gmail.com> wrote:
2013/12/18 Attila Bárdi <attila.bardi@gmail.com>
I have an openldap server and I do some search in there.
So when I search for for example to my displayName, the ldap modul found the following:
battila@igornb:~$ ./test_ldap.pl LdapPerson module is successfully loaded! displayname: Bárdi Attila battila@igornb:~$
test.pl:
*#!/usr/bin/perl -wuse strict;use warnings;use LdapPerson;*
Don't you get any "wide character" warnings?
As far as I understand you have problem with decoding your input from LDAP.
When you use your module without Dancer you seem not decode your input and Perl treats your input by bytes and on print it outputs these same bytes. So, when you have UTF-8 enabled environment those bytes are shown as right characters even when they are not meant as such.
Now, using Dancer you tell explicitly to use UTF-8 and because your input is not decoded properly (not marked as UTF-8) Dancer will treat this string as Latin1 and decodes bytes after ASCII-set again to its inner Unicode and on printing again to UTF-8. This causes such output as you have.
You could add to your test.pl one line to test your output filters:
print join( ' ', PerlIO::get_layers('STDOUT') ), "\n";
If the last filter is not "utf8", my conclusion should be right.
-- Wbr, Kõike hääd,
Gunnar
_______________________________________________ dancer-users mailing list dancer-users@dancer.pm http://lists.preshweb.co.uk/mailman/listinfo/dancer-users
2013/12/19 Attila Bárdi <attila.bardi@gmail.com>
But I don't know the solution.
You must deal with decoding in your module, but in your script it should work like this: #!/usr/bin/perl use strict; use warnings; use Encode; binmode STDOUT, 'utf8'; my $somebody = LdapPerson->new(); $somebody = LdapPerson->search('attila.bardi@example.com'); print "displayname: " . decode('utf-8', $somebody->get_displayName() ) . "\n" In Dancer you must omit setting binmode on STDOUT, because Dancer sets it itself. Hope it gives some hints how to deal with such problem. -- Wbr, Kõike hääd, Gunnar
Hey Gunnar, yesterday I also tried this. But it cost nothing to try it again. If it still does not help I will limit the users to use English character set. And after I've implemented all of the function I want to see in the webapp I can give another try to solve this. Thanks for your help! Best regards, Attila On Thu, Dec 19, 2013 at 9:08 AM, WK <wanradt@gmail.com> wrote:
2013/12/19 Attila Bárdi <attila.bardi@gmail.com>
But I don't know the solution.
You must deal with decoding in your module, but in your script it should work like this:
#!/usr/bin/perl use strict; use warnings; use Encode; binmode STDOUT, 'utf8';
my $somebody = LdapPerson->new(); $somebody = LdapPerson->search('attila.bardi@example.com'); print "displayname: " . decode('utf-8', $somebody->get_displayName() ) . "\n"
In Dancer you must omit setting binmode on STDOUT, because Dancer sets it itself. Hope it gives some hints how to deal with such problem.
-- Wbr, Kõike hääd,
Gunnar
_______________________________________________ dancer-users mailing list dancer-users@dancer.pm http://lists.preshweb.co.uk/mailman/listinfo/dancer-users
Hello do you test your result with Encode::Detect::Detector ? I use it for RSS feed paser , in same document a part of this is in utf8 and other part in windows 1252 or I use Test::utf8::is_sane_utf8 ( some title are twice encoded ) see some advice http://juerd.nl/site.plp/perluniadvice bye HB Le 19/12/2013 09:50, Attila Bárdi a écrit :
Hey Gunnar,
yesterday I also tried this. But it cost nothing to try it again.
If it still does not help I will limit the users to use English character set. And after I've implemented all of the function I want to see in the webapp I can give another try to solve this.
Thanks for your help!
Best regards, Attila
On Thu, Dec 19, 2013 at 9:08 AM, WK <wanradt@gmail.com <mailto:wanradt@gmail.com>> wrote:
2013/12/19 Attila Bárdi <attila.bardi@gmail.com <mailto:attila.bardi@gmail.com>>
But I don't know the solution.
You must deal with decoding in your module, but in your script it should work like this:
#!/usr/bin/perl use strict; use warnings; use Encode; binmode STDOUT, 'utf8';
my $somebody = LdapPerson->new(); $somebody = LdapPerson->search('attila.bardi@example.com <mailto:attila.bardi@example.com>'); print "displayname: " . decode('utf-8', $somebody->get_displayName() ) . "\n"
In Dancer you must omit setting binmode on STDOUT, because Dancer sets it itself. Hope it gives some hints how to deal with such problem.
-- Wbr, Kõike hääd,
Gunnar
_______________________________________________ dancer-users mailing list dancer-users@dancer.pm <mailto:dancer-users@dancer.pm> http://lists.preshweb.co.uk/mailman/listinfo/dancer-users
_______________________________________________ dancer-users mailing list dancer-users@dancer.pm http://lists.preshweb.co.uk/mailman/listinfo/dancer-users
Hey Hugues, thanks for the link, I will read it! Best regards, Attila On Thu, Dec 19, 2013 at 9:59 AM, Hugues <hugues@max4mail.com> wrote:
Hello do you test your result with Encode::Detect::Detector ?
I use it for RSS feed paser , in same document a part of this is in utf8 and other part in windows 1252 or I use Test::utf8::is_sane_utf8 ( some title are twice encoded ) see some advice http://juerd.nl/site.plp/perluniadvice
bye HB
Le 19/12/2013 09:50, Attila Bárdi a écrit :
Hey Gunnar,
yesterday I also tried this. But it cost nothing to try it again.
If it still does not help I will limit the users to use English character set. And after I've implemented all of the function I want to see in the webapp I can give another try to solve this.
Thanks for your help!
Best regards, Attila
On Thu, Dec 19, 2013 at 9:08 AM, WK <wanradt@gmail.com> wrote:
2013/12/19 Attila Bárdi <attila.bardi@gmail.com>
But I don't know the solution.
You must deal with decoding in your module, but in your script it should work like this:
#!/usr/bin/perl use strict; use warnings; use Encode; binmode STDOUT, 'utf8';
my $somebody = LdapPerson->new(); $somebody = LdapPerson->search('attila.bardi@example.com'); print "displayname: " . decode('utf-8', $somebody->get_displayName() ) . "\n"
In Dancer you must omit setting binmode on STDOUT, because Dancer sets it itself. Hope it gives some hints how to deal with such problem.
-- Wbr, Kõike hääd,
Gunnar
_______________________________________________ dancer-users mailing list dancer-users@dancer.pm http://lists.preshweb.co.uk/mailman/listinfo/dancer-users
_______________________________________________ dancer-users mailing listdancer-users@dancer.pmhttp://lists.preshweb.co.uk/mailman/listinfo/dancer-users
_______________________________________________ dancer-users mailing list dancer-users@dancer.pm http://lists.preshweb.co.uk/mailman/listinfo/dancer-users
Hey Gunnar, this is it. I don't remember what did I do yesterday night and why it was not working, but now I tested it and it is perfect! The solution was what you wrote: just omit setting the *binmode STDOUT, 'utf8';* and use the rest. And that is it. I would like to thank all of you for your help! Best regards, Attila On Thu, Dec 19, 2013 at 9:08 AM, WK <wanradt@gmail.com> wrote:
2013/12/19 Attila Bárdi <attila.bardi@gmail.com>
But I don't know the solution.
You must deal with decoding in your module, but in your script it should work like this:
#!/usr/bin/perl use strict; use warnings; use Encode; binmode STDOUT, 'utf8';
my $somebody = LdapPerson->new(); $somebody = LdapPerson->search('attila.bardi@example.com'); print "displayname: " . decode('utf-8', $somebody->get_displayName() ) . "\n"
In Dancer you must omit setting binmode on STDOUT, because Dancer sets it itself. Hope it gives some hints how to deal with such problem.
-- Wbr, Kõike hääd,
Gunnar
_______________________________________________ dancer-users mailing list dancer-users@dancer.pm http://lists.preshweb.co.uk/mailman/listinfo/dancer-users
As I know it is better to write: use open qw(:std :utf8); It will mark all channels (STDIN, STDOUT, STDERR) to work with utf8 (not only STDOUT, as you wrote) On Thursday, December 19, 2013, Attila Bárdi wrote:
Hey Gunnar,
this is it. I don't remember what did I do yesterday night and why it was not working, but now I tested it and it is perfect! The solution was what you wrote: just omit setting the *binmode STDOUT, 'utf8';* and use the rest. And that is it.
I would like to thank all of you for your help!
Best regards, Attila
On Thu, Dec 19, 2013 at 9:08 AM, WK <wanradt@gmail.com <javascript:_e({}, 'cvml', 'wanradt@gmail.com');>> wrote:
2013/12/19 Attila Bárdi <attila.bardi@gmail.com <javascript:_e({}, 'cvml', 'attila.bardi@gmail.com');>>
But I don't know the solution.
You must deal with decoding in your module, but in your script it should work like this:
#!/usr/bin/perl use strict; use warnings; use Encode; binmode STDOUT, 'utf8';
my $somebody = LdapPerson->new(); $somebody = LdapPerson->search('attila.bardi@example.com<javascript:_e({}, 'cvml', 'attila.bardi@example.com');> '); print "displayname: " . decode('utf-8', $somebody->get_displayName() ) . "\n"
In Dancer you must omit setting binmode on STDOUT, because Dancer sets it itself. Hope it gives some hints how to deal with such problem.
-- Wbr, Kõike hääd,
Gunnar
_______________________________________________ dancer-users mailing list dancer-users@dancer.pm <javascript:_e({}, 'cvml', 'dancer-users@dancer.pm');> http://lists.preshweb.co.uk/mailman/listinfo/dancer-users
participants (5)
-
Attila Bárdi -
Hugues -
Stefan Hornburg (Racke) -
WK -
Иван Бессарабов