I suspect that you can't use a zcut value bigger than acell_z as you do not have q-point
sampling in that direction. So discarding that situations, the results you get are not so awful,
even if not very satisfactory.
Of course it is, while for the cylinder there is an analytic expression as shown in the prb, here for the box each component of the box cutoff is constructedBy the way, it seems to me that the implementation of the box cutoff in the code is different from that of the PRB 2006 paper.
by summation over all G components and integration over the BZ (that's the reason why you need to calculate the RIM before constructing the
cutoff). As you need a sum over all the G'-vector to construct each G components of the coulomb cutoff, there is the possibility that the use of a
small plane wave cutoff can affect the quality of the results.
This shape of the cutoff has been tested for finite boxes, while it has been not, for isolating surfaces.
For your porblem, you can have a look to this paper:
Sohrab Ismail-Beigi, PRB 73,233103 (2006), here there is another receipt that should not be difficult to implement.
Cheers,
Daniele