The undo tablespace is a series of circular queues, a set of undo segments (made up of extents)

The undo tablespace will have a set of undo segments in it, each of these is a circular queue, the extents point to each other, the last extent in the segment points to the first extent. The size of this queue is controlled by you indirectly via your setting of undo_retention.

So, let's say you set undo_retention to 1 hour. We will ATTEMPT to hold undo in the undo segments for at least one hour.

So, as you are aware, as we are processing transactions we write UNDO to these segments.

Now, as you generate undo, we get to the end of an extent and want to advance into the next extent (the last extent remember considers the FIRST extent to be its NEXT extent - the circular queue concept).

Before we advance into the next extent - it must not contain any active transactions - if it does, we cannot advance and would simply add another new extent into this queue. We would get that extent from a) free space in the undo tablespace OR b) by extending the datafiles for the undo tablespace to get more free space (not in your case, you disabled that) OR c) by stealing an extent from some other undo segment - finding its oldest unused extent and stealing it. If all of that failed, you would receive an error and the statement you were executing would be rolled back.


Now, let's say the next extent is not active - before we advance into it, we'll check the age of the data in it. If the age of the data is more than 1 hour (our retention period) - we'll advance into it. If the age of the data is less than one hour we will try

a) add extent from free space so as to not overwrite the data
b) add extent from free space after autoextending a file
c) to steal an extent from another undo segment that is not active and has data over an hour old.


If none of that works, we'll have to advance into that extent and prematurely "expire" this undo (eg: violate your undo retention request). We will not fail (unless you set the undo retention guarantee)
http://docs.oracle.com/cd/B28359_01/server.111/b28310/undo002.htm#ADMIN11463


You can monitor all of that in v$undostat


Now, after a while, some extents in your undo tablespace might become older than one hour hold (data is no longer needed), we can and do release these extents over time as we online and offline undo segments and such. That is the "shrinking" you saw - we just freed up data we didn't need anymore, putting it back into the free space so that when we need to grow again - it is there.