I don't think I've experienced headaches without caffeine though I am an addict caffeine. When most people get it from coffee, I also get it from caffeine powder and other caffeine sources that are mixed with other multivitamins. Thankfully, if I don't have any single caffeine running through me, I am fine, no headache at all which is also surprising to me. Everyone has different response to caffeine which remains to be interesting phenomenon to me.
Have you tried chicory root or maybe maca powder for hot drinks? I have heard that chicory root can be replacement for coffee though I haven't tried it.