If I understand your problem correctly, the main problem is in configuration.

By default testing uses development environment and there is set "show_errrors = 1". That causes on die (or croak) to show full stack tree. Setting "show_errrors = 0" should give you simple 500 page.

Also, there is Dancer::Exceptions and this gives us special keyword "raise". Maybe it is better to use it? I read about it here:

http://advent.perldancer.org/2011/6 

Hope i got you right.