Based on my experience with so-called "big data", I believe that "Google dance" is just a part of the regular sorting algorithm of huge amounts of information (really huge). So it takes weeks until the results settle down. Google usually re-sort all the results only every few months or so (with every new update). What follows up is that the minor changes after the initial "dance" are just a part of the new sites being added to the list and put in place. But I wouldn't exclude the possibility that dancing is a part of the CTR testing to see what result attracts most clicks.
(If you are interested to know more, don't get discouraged with the amount of information below.
)
Let me back that up with some more information: The goal is that you need to sort ALL websites ever indexed for each and every keyword out there (the cut-off is at result #100). To get an idea of how big that operation is, you have the following input:
1. Number of keywords:
To get a better idea about the amount of data mentioned, let's imagine Google works only with the English language. Based on
Oxford Dictionary there are 171,146 words currently in use in the English language. Then you need to figure out an amount of potential combination of all those words. There are 2, 3, 4, or even more words for one keyword. Then add (at least semantically) meaningful questions to that list. The number of possible combinations is counted in millions.
2. Number of websites:
There are 1.88 billion websites today (and growing) based on
Statista data. Each of those websites has thousands and even millions of pages.
Imagine you have to sort all web pages (not only websites but their every page!) for each and every potential keyword out there. Those familiar with combinatorics already have an idea that the resulting number is insanely big. Even with all the huge computational power that Google has, it is impossible to compute all those results in real-time. It usually takes weeks to get the first result pages populated with meaningful results. It can take months to get results starting from page 2.
Multiply that with a lot of different languages. Then add major updates every few months to the equation (at least twice a year) and you will get an idea that it is impossible NOT to have "dancing" in place since it is simply impossible to get near-real-time results.
Google is probably is not crunching all that data all over again, so they probably use some caching instead, which saves them time. But every now and then they had to regenerate the complete database. Something like that probably happened last year, where they had their database "frozen" from May, since a big "bug" that they had, until late September. There's a high possibility that they either implemented some big (unplanned) change in their algorithms, or they had a major bug, so they most likely had to regenerate the complete database of results.
So, to put that simple: I would say that "dancing" is just a regular part of getting new results generated all over again.