Causality is different for shadow walkers. "A causes B" is replaced by "I want B, therefore A also exists".
Or to put it another way - time runs backwards.
So the 'start' of Amber isn't Dworkin drawing the Pattern, it hasn't happened yet - what *will* happen is some Shadow walker, probably using Logrus, deciding to spend the time to acquire an Amber Devotee!
The existence of Amber is not required to *want* this, but now that the shadow walker does want, and acquire, their Devotee, that Devotee will come into existence. At some point, the Devotee will want to learn where they came from, and walk to their Amberite parent - bringing that parent into existence. They in turn will wonder where they came from...sooner or later, someone walks to Oberon, who walks to Dworkin, who is too mad to care. Or possibly Dworkin did not exist until Corwin walked to him: in which case Dworkin's madness has as its' true cause Corwin's centuries of being locked up.
The original shadow walker is probably a descendant of Merlin; an orphan is more likely to dream of being the child of a king than of a pauper, especially given Amberite egos. Also, once called into existence, a king is likely to tinker with shadow to make it more likely that they are the ancestors of powerful shadow walkers, providing them with a degree of control over the most powerful beings in existence. In turn, Merlin is probably walking Corwin's history into existence as Corwin explains it to him at the end of the first series. A major advantage of this theory is that the dysfunctional nature of Amberite society is explained by Merlin being a prat.
Of course, with multiple different bloodlines, why the eventual convergence? Well, friendship is part of it; friends often wish to be related, so Luke and Merlin would be happy to 'discover' that they were cousins. Hatred also creates links: so Dalt proves the evil of Amber by walking his mother's rape into existence. The emphasis on one Amberite parent means that the number of ancestral lines isn't increasing; and once a handful of 'siblings' have created themselves, the trend will be reinforced - but with individuals who aren't particular well designed for it, thus the family feuds.