What to do to run GUI Java app in chrooted environment?

Make sure the environment variable XAUTHORITY is set (usually to ~/.Xauthority). Then, in the host:

$ xauth list
latitude/unix:0  MIT-MAGIC-COOKIE-1  d4474d13c

Now, within the chroot environment:

# chroot some-debian
# xauth add latitude/unix:0 MIT-MAGIC-COOKIE-1 d4474d13c
# xcalc

If you're trying to get any chrooted app to show up in X11, you will need a couple of things set up correctly. One is a valid DISPLAY environment variable, second is a proper Xauthority file, and third and most important, access to the socket used by X11/Xorg. X11 can use either a TCP network socket or a Unix Domain socket. A TCP socket will be easier to use from a chroot, but most modern desktops have TCP turned off with -nolisten tcp or something similar set on the X server. TCP sockets start at TCP port 6000 for display :0 up through 6063 for the last display. As long as TCP sockets are listening, you shouldn't need to do anything more with sockets. You still need to worry about DISPLAY and Xauthority. Unix domain sockets require a little more work and, at least on Linux, reside under /tmp/.X11-unix/X? where ? is a number from 0 to 63. If your chroot resided on the same filesystem as /tmp then you can use hard links, at least on Linux. Symbolic links won't leave a chroot. Replace ? with the correct display number:

mkdir /path/to/chroot/tmp
chmod 1777 /path/to/chroot/tmp
mkdir /path/to/chroot/tmp/.X11-unix
chmod 1777 /path/to/chroot/tmp/.X11-unix
ln -f /tmp/.X11-unix/X? /path/to/chroot/tmp/.X11-unix/X?

mkdir and chmod only need to be done once, but the socket file will need to be recreated with ln on each X session. If /tmp is not on the same file system, life is harder and you will need some kind of hackery such as making /tmp/.X11-unix a symbolic link to the corresponding directory under the chroot. Next, make sure the DISPLAY environment variable is set and matches what is used by your terminal and other X11 apps. Lastly, to copy over the .Xauthority, use xauth. You need to find the matching cookie for your X11 session and this cookie will be different for every session. Use xauth list $DISPLAY to print out cookies for your DISPLAY. The name will look like host:? or host/unix:? where host is the hostname of the computer and ? is the display number. The display number can be retrieved with echo $DISPLAY and it will be the number following the colon (:) and before any period (.) To copy over the Xauthority, use something like this:

xauth extract /path/to/chroot/.Xauthority host/unix:1
xauth -f /path/to/chroot/.Xauthority list

The second command simple list the copied entries. If you need to copy while using sudo or other command to change users, try something like this:

xauth extract - host/unix:1 | sudo xauth -f /path/to/chroot/.Xauthority merge -

If the place you put the .Xauthority file is not the home directory for the chrooted user, you will have to set the XAUTHORITY environment variable:

export XAUTHORITY=/path/to/chroot/.Xauthority

As you can probably tell, it's not typical to run a GUI app under a chroot.

I don't think there's anything you need to do specifically for Java apps, but you do have to do something for GUI apps in general.

I've done this on my home computer, but I'm not at home right now so bear with me. I have a script that mounts everything into my chroot like /dev and /proc, but it also copies ~/.X* as well. If I remember correctly, in your home folder there are files that start with .Xauth, and you copy those into the home folder of your chroot environment. In my experience these files need to be copied every time that X is restarted.

What I would do to test this method out was I'd just run a simple X application like xterm. It was just a fast and simple app I could test with. xterm should give you an error message if it can't connect to the X server.

I happen to have the full X11 package installed into my chroot, but I don't know enough about X to know if chroot'd apps run on a chroot'd X server or on the host X server.