Personally a bigger fan of psionic storm, the damage scales insanely, but it does eat mana. I never chug mana pots, the skill fires with the mana cost if you're just spamming, it only fails to heal you if it can't consume 10% max mana and I have enough regen to refill ~21k mana in 10-15 seconds of no casting. Casters preference for sure though.
Have you tried the build in Scosglen? I find the biggest issue with magic damage builds seems to be that the magic resistant mobs of most areas are unwarrantedly tanky due to the lack of magic pen available in the game. Dealing with heavily fire resistant mobs is straightforward for fire builds, just build 150+ fire pen and you can still melt them, the same option doesn't really exist for magic builds. I find the big trees in Scosglen near impossibly tanky for magic damage, to the point where I would just ignore them because it's not worth the time investment. Do you encounter similar issues?