It is almost unheard of for a rear brake drum to wear out on a motorcycle, because about 90% of the braking is done by the front. You will need to remove the wheel again, and pull out the brake assembly to see what is going on it there. If nothing looks broken, but it is really dirty and corroded, you will need to take things apart and clean them until they are spotless. You can use sandpaper, a sanding sponge, or plumbers cloth to sand the inside of the drum until it is bright and shiny everywhere. Remove the shoes and check them. If they are pretty worn, I'd replace them. If not, clean them up really good with brake parts cleaner and a parts cleaning brush, sand the friction surface to remove any glazing, and also sand the pivot point on one end, and the flat surface where the cam goes on the other until they are completely free of dirt and corrosion. Do the same thing to the pivot pin, and the cam. Remove the cam and sand the shaft until it is clean and shiny, then find a wire brush to clean out the hole in the backing plate.
Simple Green is a good all purpose cleaner for the wheel, and the nasty black mess you are likely to find inside the brake assembly. Use it full strength from a spray bottle. Again, get everything clean and remove all traces of corrosion.
Now that it is all clean, get some high temp silicone grease (or plumbers grease) put it on the cam shaft, and in the hole in the backing plate with a Q-Tip, and install the shaft back into the hole. Now put some grease on the pivot shaft and the flat surfaces of the actuating cam, and also on the shoes where the pivot goes, and where they fit against the cam. Don't get any grease on the shoes. Reinstall the shoes. Install the arm on the cam shaft on the outside of the backing plate. Now test everything for prober operation to make sure it is not sticking or binding somewhere. If it is, find out where, and fix it before installing the wheel.
As for the swing arm, That is not as easy as it should be. And probably not necessary. I have 70,000 miles and have never had the swing arm off. But my bike was bought new, garage kept, and seriously overmaintained. It is dusty (this is AZ), but has no rust or corrosion anywhere. Cleaned up it would look new. I do the splines every 10,000 miles, and check, clean and lubricate the rear brake at the same time.
With the wheel and shocks off, move the swing arm up and down, and try to move it side to side. If it moves up and down with no resistance, and you can't feel any side play in it, I'd leave it alone. If I ever do have to pull my swing arm for those reasons, I will be installing new bearings anyway. Unlike the splines and the wheel bearings, there is nothing urgent about the swing arm bearings. Just check for play whenever you have the rear wheel off, which should be about every 10,000 miles.
The swing arm can be a pain to remove for two reasons. One, the goats belly and mufflers have to come off first, and that in itself can be a frustrating job. You usually wind up having to at least loosen the header bolts at the engine, and everything will be stuck together. Once you get the exhaust system off, then you have to remove the side plates, and pull out the pivot pins, which are usually also stuck. The bolt required is a common metric bolt, not sure of the size. Many bolts of that size are used on the bike.
When you finally get it off, then you have to get it back on, and that is even less fun. You have to set the clearance between the swing arm and frame on the left side. I used a KLR650 valve shim of the correct thickness when I did it on my first Vulcan, but it can be done other ways. You can use the right combination of feeler gauges, or you can find something, like a washer, that fits right before you remove the swing arm, to get it put back the way it was. Certainly not rocket science,, but quite frankly I'd rather pull the engine out of a car.
I am a motorcyclist, NOT a biker.
1997 Vulcan 750, purchased about a week ago
2006 Sportster 1200 Low
2013 Royal Enfield Bullet 500, converted to carb
2001 Yamaha XT225, heavily modified
2004 Honda Rebel 250
1979 Vespa P200E
2002 Vulcan 750 parts bike
1994 Yamaha XT225 parts bike