Well I figured it out. When I pulled the axle everything just fell out and so when I put it back together the part that holds the caliper has a slot that you have to slide on the swing arm. Unfortunately I had to remove the axle bolt and slide that piece out. I have an old bolt from my RSTD and used that and slide it in so that the wheel didn't come apart completely and was able to put the part in it's proper place.
Hope this might help anybody else that runs into this problem. Took it for a ride and all is well.