Improvisation is a basic DMing skill. Players go off the map, conceive devious work arounds or simply ignore adventure hooks. Sometimes life happens, you have no prep time and heavy improv is the only way a game is going to happen at all. It is a skill you should cultivate. That said, some DM’s think that improv is a reasonable replacement for effective preparation. Consider that relying on improv instead of preparation is an error as a DM. Most improv could do with a rewrite. Improvisation, even from skilled and experienced DM’s, is going to be fair to middling most of the time. You aren’t always going to come up with something compelling when you have to do it on the spot. That is the real problem with over reliance on improvisation as a DM.
In most creative arts, there is a lot preparation and lot of work that the people who eventually experience the art never see. In film, a screen play may take a year or more for a writer to complete. A novel will see several drafts before its publisher accepts it. Musicians, noodle around for hours and hours to write a song that lasts three minutes. I don’t think it is unreasonable for a DM to write and revise an encounter a few times before it sees light at the table. That editing can be the difference between an OK encounter and a great encounter. Your first instinct may be a good one but it is difficult for even the most creative DM to nail it off the cuff.
A lot of the so called story games are heavy on improvisation. There is a basic set up of where the game starts, an aesthetic or genre the game takes place in but nothing more. I’ve played some of these games with designers who are the darlings of the improv narrative focused games world. They play and think about these games all the time. It is their bread and butter. Even then, what I found is that the goal of a “satisfying narrative” often falls flat. The games were enjoyable but felt like if the GM had more time to consider other potential problems for the players to solve, it would have been the difference between good and great.
That doesn’t mean you need to write up whats on every street and the family tree of every NPC in the city. Having some knowledge of what the villains and monsters want, what their abilities and resources are, where players can get the usual sorts of things that player’s want and the NPC’s who can provide those resources is a worth while degree of preparation. Add some random tables and riffing off of the ideas the players throw out and you’ve got a game.
There is a balance point. If you like to write up detailed histories of your setting and NPC’s, do that because you like to do it but don’t expect that all of it will be useful or interesting at the table. Players find thirty minute lectures about the goblin wars to be tedious. Most players blow off long backstory homework assignments between games. No one wants to watch you dig through your binder trying to find the complete inventory of the hardware store. Prepare your settings and adventures, consider what is necessary at the table, make it easily accessible. If you have the time before you play, come back and reconsider the work you’ve done and revise it. You may have a better idea in the shower and change a few things before game time. If you do little or no prep and fly by the seat of your pants, you are going to miss opportunities to create some material that your players will remember for a long time to come.