How Soon Is Too Soon To Decorate For Halloween?

Call my crazy, but I LOVE Halloween so I started decorating my house 2 weeks ago. I have been told that this is WAY too early.

It kind of drives my husband crazy. I think if you have decorations you might as well put them up. If 5 weeks is too early to decorate for Halloween then when should I decorate?

Does it REALLY bother people if you decorate too early? It's just Halloween decorations.