Learn Russian: Давай послушаем музыку!

Because послушать is a perfective verb (focusing on a single or completed action). When making a "let's..." suggestion with a perfective verb, Russian requires the "we" form of the future tense (мы послушаем). You only use the infinitive after давай if the verb is imperfective.

Could I say Давай слушать instead? What is the difference?

Yes, you can! Because слушать is imperfective, it stays in the infinitive after давай. Давай послушаем (perfective) suggests a specific, contained action, like "let's have a listen to this song." Давай слушать (imperfective) focuses on the process, meaning "let's spend some time listening" or "let's start listening right now."

If I am talking to a group of friends, do I change послушаем?

The verb послушаем stays exactly the same, because you are still proposing an action for "us" (1st-person plural). However, the word Давай must change to Давайте, because you are addressing more than one person. You would also use Давайте послушаем when speaking to one person formally (someone you call вы).