You are some 2-3 km in the air. There’s nothing to hold on to. You are sitting on a piece of cloth that is barely large enough to accommodate you. No railings, no seatbelts, nothing to hold on to or stop you from falling to your death if the carpet gets into turbulences or a strong gust of wind makes you lose your balance. And let’s not forget that most depictions of magic carpet flights have the carpet move very erratically because they have a mind of their own.
I’ve only ever ridden mine a few tens of meters above the ground, but you’re right, still high enough to hurt if I fell off. It’s not really as unstable as you make out, more like a floating mat in a swimming pool.