Using self-developed half mould direct shear apparatus, this paper carries out the direct shear test between soil and steel or stone. Different vertical load varies from 50kPa to 200 kPa. Through numerical simulation of shear test of soil and structure under different conditions of stress via PFC2D, this paper studies the shear stress variation and deformation of soil at the interface with the structure. It is concluded that strain softening occurs to soil in the process of shear and the shear stress of soil which rises as vertical stress grows. Shear zone of some thickness appears at the interface when soil and structure interact with each other, which means that the interaction zone covers not only the contact surface but also the soil off the surface to some extent, the range of action in soil extents 5 times’ particle size away from the interface. The surface roughness of structure imposes slight effects upon the change of shear stress.