So you have your bike. What can you do to make it faster? What is this most cost effective way?
Things that make you faster - aerodynamics that cheat the wind, friction reduction on moving parts and from the road, weight, not losing power in energy transfer. <--of these weight is the least important.
Here are some suggestions:
New tires - if you reduce friction between you and the road, you will go faster. If you are using mountain bike tires and you aren't on trails -GET RID OF THEM. road tires can be bought for as little as $20 pr. On road bikes you want tires that have minimal road contact - use the minimum tire width for your rim, a high profile center and a high max pressure. Decent 700cm road tires start at $20 ea.
Getting your bike fit properly for you. I'd recommend havng a more experienced cyclist give you pointers and then do some trial and error. It is free. You can also pay for professional fittings for $100-$250 but I don't recommend this unless you have disposable income and are a front of the pack cyclist.
Keep your bike clean and tuned up - $15 for a bike maintenance book and $3 for some WD-40 will be your best purchase.
Latex tubes are faster than butyl tubes. Why? I don't know. They just are.
Clipless pedals will help you go faster but that is $30+ for pedals, $60+ shoes and if you need help getting it all together there is labor cost too. The cheap and effective alternative and it is a MUST if you aren't clipless is to get some basket toe clips. $5-$15 at Target. Screw them on, adjust them for your favorite riding shoes and your off. Ideal shoes for biking are lighter and have a stiff sole - so pick the pair of your shoes that best fits that description.
An Aero Helmet - you'll look like a tool but the claim's are fantastic. In order to realize the benefit it must be one that covers your ear. These can sometimes be found for $50 on sale. Or you could just get a nice normal helmet that has good aerodynic flow $35+
The number one thing to make your bike go faster is the most effective tool and it's free. It is you. Train properly and get to your ideal weight.