The physical interpretation is similar to fluxes of gases. The neutrons exhibit a net flow in the direction of least density. This is a natural consequence of greater collision densities at positions of greater neutron densities.
Consider neutrons passing through the plane at x=0 from left to right as the result of collisions to the left of the plane. Since the concentration of neutrons and the flux is larger for negative values of x, there are more collisions per cubic centimeter on the left. Therefore more neutrons are scattered from left to right, then the other way around. Thus the neutrons naturally diffuse toward the right.