While there is a fair share of top notch game engines out there available for indies to hit the ground running, most of them have the same common issues. No, it's not the cost :) It's lack of coherent and well engineered game code, and lack of modding capacity.
Note that I do realize that a solid indie team that works full time (or half-time; not just in the evenings :) ), has a couple of dedicated programmers and several artists (or means to outsource art), can make a game faster, and probably better using modern mainstream game engines than using GPL engines, but that's not what I am talking about here. What I am talking about is a small indie team that is short on staff and resources, trying to break through.
What does "GPL engine" mean? Without using fancy legal wording, it simply means a game engine source code released under GPL license. For example, someone (id Software for example) released engine's source code under GPL license. Someone else (Darkplaces engine author for example) grabbed it, modified hell out of it, gave it a new name, and released it for free of charge, in both binary and source code forms, with all and every change that was made to the original code. Then someone else did the same thing, and so on. Every single iteration of the original engine made, has to remain under GPL license, has to retain copyright notices, and any middleware used has to be GPL license compatible. Such engine can be freely distributed or sold.
A common misconception about using GPL engine in commercial game development is that entire game has to be released under GPL. It's not the case. Art assets (including music and sounds) are not covered by the engine's GPL license, therefore you keep it under proprietary license.
Another complaint I heard was this: "I have to release all of my code to the public. Someone can grab years of my hard work and make another game with it". Or something along those lines. It's true - someone could theoretically do it. However, starting with Doom engine and ending up with Doom 3 engine, no one grabbed the code as is and made a straight up clone. Some one definitely could have done that, and should have done that, but making a game is not a trivial task. Having an engine doesn't guarantee completion of a game.
Yet another argument against GPL engine - can't release on consoles. Again, it's true. However, releasing on consoles is nothing like releasing on PC/mobile. You'd need to get dev kit first, and without having N-amount of money and an office, you aren't going to get it. Then you need programmers who is familiar with consoles (although not necessarily when using one of the mainstream engines), and you need a QA team. Passing certification is not easy. So chances releasing on the consoles are pretty slim for your first several titles.
One thing that idTech engines have that mainstream commercial (and other GPL) engine don't offer is a full game code, tested through a long period of time. In other words, you only need to add a few things gameplay is missing, and focus on art and level design. It's an enormous time saving. Of course if you are making something entirely different, such as flight sim, or MineCraft 2, it would not help having complete game code of an FPS game. However if you are making FPS, TPS, point and click 3D adventure, top down game, etc. that have elements of Doom, Quake, Quake 2, Quake 3 or Doom 3, it would definitely work out well.
Modding is a big thing. Releasing a game is a gamble. It might take off and sell well, but might have community that cares less about modding. Or you can slowly build up sales with community that is creative and willing to mod your game, thus sustaining and prolonging life of your product. It's better have an engine that allows modding, rather than an engine that either requires you to buy a license for the engine to mod your game, or by definition can not be modded. With GPL engines, it's all possible. With most of the mainstream engine, it's a headache or a problem.
To summarize, here are key pros and cons of using GPL engines:
PROS:
+ No royalties, no payouts, no monthly fees - free to use and all profit is yours;
+ Existing polished game code (idTech engines and some derivatives);
+ Ability to modify engine to suit your needs without getting ok from the Company;
+ Support for modding;
CONS:
+ Require having talented programmer on the team;
+ Lack of solid documentation;
+ Potential lack of community that held knowledge of technology and its modding;
+ Potential lack of good tools;
+ Bound to PC/mobile platforms;
There are probably more points to add to either of the two, and I could probably go on and on, but I'd like to hear about your experience with GPL engines.
