Hi Huang,
I would be happy to help but I would need more information.
What is a GCH?
Where is you HWP, between the SLM and the cube or before the cube?
What is your grating? Is it the mask you display on the SLM?
I am not sure I see how it can work with a PBS.
You are suppose to feed your SLM with one given polarization (horizontal I think) because only at this polarization you would have a nice phase modulation. Light reflected off your SLM have the same polarization (I think it is how it works for the Holoeye LETO).
So, your PBS an only provide H or V polarization and the incident and reflected light of your SLM have the same polarization. I do not find a way to have your modulated beam reflected by the PBS instead of transmitted, but I can be wrong. You would need to provide me more information.
Anyway, before using the SLM, you should characterize the complex modulation as explain in my tutorial. It is the safest way to be sure that everything works the way it should.
About the diffraction pattern. If you send a ideal square signal between the phases 0 and pi/2, you still have a high 0 order because of the DC background. This is due to the fact that the average of your signal is not zero (0.5+0.5j in this case). If you calculate the Fourier transform, you have half the energy in the 0 order and a quarter in each +1/-1 orders. That is what you seem to observe.
If you take instead a grating with a square wave between 0 and pi or between -pi/2 and pi/2, the average value is 0 and then all the energy is in the +1/-1 orders (half of the energy in each).
I enclose an image of the power spectra (intensity of the FFT) for a square wave with two phase values 0 and pi/2 (left) and for a square wave with phase values -pi/2 and pi/2 (right). You see that in the first case you have half the energy in the 0th order and not in the other one.
Best,
Sebastien