At first blush this approach didn't work. The result was still an unstable system quickly alternating between floods and droughts. To add to the challenge the system wasn't actually closed... Oops.
Idea #3 - Closed System with New Water Transport
At this point I was ready to give up. Really and truly. It had been weeks of struggle with little that would work in a commercial game (even a small time indie one). I took a little time off while I ran 12-24 hour long simulations - I was irrationally hopping to find stability with small tweaks in parameters.
Once I pulled my head out of my butt I started to rethink the cause of the system not being closed and how I transported the water.
The cloud mechanism made a lot of sense as water transport, but it added substantial complexity and inter-connectivity that I wasn't excited about. The clouds had the added bummer of being physics based which needs to run on the main thread in Unity. This not only results in a (slight) slow down, but a decent amount of interplay between the a "cloud thread" and the main thread that's running the physics engine. This seemed like a very good potential candidate for causing the system to not be closed - errors given the complexity and syncing data between threads.
I rewrote the water transport code to get rid of the cloud and to use humidity to move water from neighbor to neighbor based on relative levels of humidity as well as the direction of the wind. This allows humidity to both spread around the map and to move with the wind. Since only the direction and magnitude of the wind is needed all of these calculations could be run off the main thread. Nice bonus!
I ran a few simulations with this new mechanism and was generally happy with the result, except the system was still not closed. After 30-60 minutes there was a noticeable error in the total amount of water in the system... Shit.
The errors per simulation cycle were tiny. Almost not there. I tracked down a few to simple small mistakes in coding. Simple mistakes that with larger numbers would have shown up much earlier and easier - when you're changing a value in the 6th or 7th decimal it can be hard to see the problem. Some of the others were harder to find.
Some of the error was simple floating point errors. Most values in the simulation are between 0 and 1, but a few such as the total water are in the hundreds. Now I don't claim to be smart enough to fully and deeply understand floating point errors, but I know they exist and I could see them creeping into my simulation. The errors in some of the larger values were bigger than the changes in other values...
Since I'm targeting PC (and maybe OSX) I'm not too worried about memory - at least not yet - I switched the whole simulation to double precision floats.
AND?
At long last the simulation was stable. Not perfect. Not balanced. But stable. I ran the simulation for over 30,000 cycles and saw an accumulated error of 0.00001 in a value that was close to 1000. I may have danced a little when this happened. Which led to the most wonderfully boring GIF I've made (watch that last digit change).