Bitcoins are 'stored' in something called a Bitcoin address. They look more or less like a string of random characters, but they always start with a 1. They look something like this:
This address is safe to give to anyone, and anyone will be able to send Bitcoins to it or to see how many coins this particular address holds. Bitcoin addresses are free to create and there's an essentially limitless supply of them - it's often considered good practice to use a new address every time you want to receive bitcoins, as it makes correlating your monetary habits more difficult.
To send bitcoins to an address, a message is broadcast from the owner of the sending address to the bitcoin network that they want to send X amount of coins from their address to the new address.
BItcoin addresses are created by first picking a random number (for the all important key) and creating an ECDSA (Elliptic Curve Digital Signature Algorithm) public/private key pair with them. This operation alone generates the private key - but Bitcoin addresses are not simply public keys, but rather modified versions of them. The generated public key is then put through several SHA-256 and RIPEMD-160 operations, until eventually being converted into a format called Base-58. Base 58 is an encoding that removes the possibility of similar looking characters, such as lowercase L and uppcase I, as well as 0 and O. Finally an identifying number is added to the beginning of the address - for bitcoin addresses, this is a 1, indicating it is a public bitcoin network address.
The end result is a reasonable length string of characters that anyone can copy paste into a Bitcoin client to send coins to.