Just get a couple of other smaller type jobs coming in as well if you can, then technically you are not subbing to one contractor and effectively a company in it's own right.
Do you provide your own tools, insurance, vehicle etc? If you do then you should be outside the scope of IR35.