There once was a project at Mandriva where each packager was to supposed to come up with a basic functionality test for his/her package(s), although I don't think it got much traction aside from those of us who were getting paid to work on the distro and whose manager made the tests a deliverable.
iirc there was even some automation framework so they they could be run by machine.