You might have a couple of things going on.
First, yes, the Chrome processes could be dragging the box down. At minimum, you'll have two Chrome processes: one for the browser, one for the helper. Depending on how Chrome is invoked (not sure what the default is), you can have at minimum a Chrome process per tab. And if the tab has Flash, that's another process, ditto Java, etc. So, just a couple of tabs open in Chrome can spawn a lot of processes. Add in if you're trying to watch a relatively HD YouTube video or Hulu, that too will drag the box down on top of everything else.
Second, it could be how you have Finder configured. Bring up a Finder window, then click on the "gear"/settings button at the top of the window and select "Show View Options". Make sure "Calculate All Sizes" is not checked. If checked, uncheck, click on the button on the bottom labeled "Use as defaults". Oh, and I believe this can only be done when in List view.