1. [numerical analysis] Said of an algorithm or computational method that tends to blow up because of accumulated roundoff error or poor convergence properties.
2. [obs.] Software that bypasses the defined OS interfaces to do things (like screen, keyboard, and disk I/O) itself, often in a way that depends on the hardware of the machine it is running on or which is nonportable or incompatible with other pieces of software. In the MS-DOS world, there was a folk theorem (nearly true) to the effect that (owing to gross inadequacies and performance penalties in the OS interface) all interesting applications were ill-behaved. See also bare metal. Oppose well-behaved. See also mess-dos.
3. In modern usage, a program is called ill-behaved if it uses interfaces to the OS or other programs that are private, undocumented, or grossly non-portable. Another way to be ill-behaved is to use headers or files that are theoretically private to another application.