bad headlight relay was the problem
Guess what ? Sure enough, its something wrong with the headlight relay inside the junction box. I was fiddling with it, checking continuity of wires, when all of a sudden I felt a small mechanical jolt and the headlight goes on.
since the headlamp-malfunction light wasn't lighting up as well (see first post) I wanted to test that. Basically I wanted to recreate the problem and try the solution again too. I disconnected the headlight. I get the bike running. I didn't feel the relay fire so no power was flowing to the headlight socket and the headlight-malfunction light also didn't kick in. Then I fiddled around some more with the junction box, pin 6 & 7, and I felt the relay kick. Sure enough the headlamp-malfunction indicator light lit up, as expected, since the headlight was disconnected. I connected the headlight and I got light! Then the headlight-malfunction indicator went out, as expected, since everything was good.
It seems to me the relay is somehow getting stuck. I created a bypass to circumvent the relay as a temporary fix until I get a new junction box. The junction box doesn't seem to be serviceable, the backplate looks riveted, or I'd open it up and try to fix the relay. The bypass involves pin 6 & 7 or there abouts. If you follow the Clymer schematics it actually shows the inside circuitry of the junction box.