Actually it is in fact the case that Aero will eat up a tremendous amount of resources, even on a desktop or traditional laptop. Aero is an umbrella for numerous graphical and performance 'widgets' (or I've seen the term 'eye-candy' used a lot too) which sound small but add up quickly.
Coupled with the more limited graphics handling of the various Mini's onboard graphics chips, and most users will experience a 20-40% upswing in available resources by turning Aero off, and in many cases that same majority of users will never miss the changes that turning Aero off makes.
So, no you are not imagining things...Aero vs. no Aero on my Inspiron Duo made a measurable difference in response time and function speed and I have noticed basically zero change in my general experience.
Then again, I turn Aero off on all my work machines as well. Only my gaming PC runs Aero, and on that machine, I just run it because I can

Sleeping_Panda
Inspiron 1090 (Duo) since 2/23/11
~~~~~~~~~~~~~~~~~~~~~
Optiplex 760; Latitude D630; Latitude D830; Latitude E6410 (and this is just at work)
Technical Support & Training Technician
@ a DOE National Laboratory