Get on-site support
Tyler D. Muth, July 27, 2005 - 9:42 pm UTC
Tyler here. Oracle sells on-site support. Think of on-site support as somewhere between calling support and hiring a long-term consultant. IMHO, it's the most valuable component of partnership between Oracle and a customer when the customer has little or no prior experience with the technology. The on-site support person will be an EXPERT in the technology requested. In this case RAC. He (or she) will have installed RAC hundreds of times, will properly configure the disk and network, and will help to educate you on best practices.
If you are negotiating a deal and are at all intimidated by the install and config aspects of the technology, ask for on-site support. You'll have to work out a price, but IMHO, it's worth every penny.
Thanks,
Tyler
rac good but...
Connor, July 28, 2005 - 12:20 am UTC
Ensure you cover all the costs...RAC licenses ain't cheap
Beware of the network requirement
Peter Tran, July 28, 2005 - 8:48 am UTC
Hi,
One of the big issue with RAC which people gloss over is the importance of having a really fast network connection between the nodes. If you're thinking of using 100Mbit connection between the nodes, then you can forget about RAC.
From experience, the SAN also has to be on high-end of the performance spectrum since it has to be shared among the nodes in the RAC cluster.
Summary, on a RAC environment the SAN and network requirements are higher than a non-RAC environment.
HTH,
-Peter
July 28, 2005 - 10:21 am UTC
I agree with the comment about the interconnect - absolutely, you want a dedicated high speed private network between the two, that is virtually a requirement.
But for the disk, why? Why would 25x2 users be doing more physical IO than 50x1 users in general. If I have your block in my cache, we don't necessarily physical IO it - we send it over the interconnect (hence the requirement/desire for the private high speed network).
You can saturate a SAN as easily with a single box.
A reader, July 28, 2005 - 9:47 am UTC
Hi,
I work for one of the worlds largest financial companies, we have deployed RAC and its well worth it for our mission critical systems..
My suggestion before implementing RAC make sure you have sufficient knowledge of RAC and test it, otherwise people end blaming the product.
Thanks.
July 28, 2005 - 10:31 am UTC
My suggestion before implementing RAC make sure you have sufficient knowledge of RAC and test it, otherwise people end blaming the product
thank you, that sums up my point concisely.
RAC hardware requirements
mike, July 28, 2005 - 9:44 pm UTC
Can anybody point me to the hardware requirements for RAC on Linux? We're trying to set up a test enviroment to get familiar with it. Currently we're going to purchase 2 Linux servers and some SAN storage. What else do we need? What are the networking requirements for the 10g cluster? For example do we need extra nics, how many and what speed? I understand that 10g provides its own clustering software.
July 29, 2005 - 8:25 am UTC
You need disk that can be mounted and accessed by both machines (ASM may well be very useful here)
You'll want a private interconnect between the two, the faster the better. Separate from the public network.
Is ASM ready fo rprime time production deployment?
A reader, August 21, 2005 - 5:34 pm UTC
I heard vaiorus argument about ASM. I know ASM is a good thing and Oracle heavily promote it. ASM saves money, save time for DBA/USA for managing IO issues. Life is sweet!
But, in contrary: ASM does not support multi-channel fiber connectivity from SAN to host, ASM can not handle high volume transactions.
Am I correct pointnig out ASM's weakness? Please comment ...
August 21, 2005 - 7:22 pm UTC
where did you get that "information"?
references...
My thoughts
Bipul, August 22, 2005 - 5:41 am UTC
RAC is wonderful, once you have configured it properly. We run couple of hundreds of websites from our cluster and the DB is Oracle10g [10.1.0.4] on Sun Solaris 8 with Sun 6130 as Storge. Interconnect is gb. Note that a good interconnect is one of the main requirement for success of RAC. If your interconnect is not properly configured [or the NIC is not good enough], then your instances will wait on inter-instance communication and you won't see any benefit of RAC.
We need a very high availablity [4 9s ] and hence RAC. However, in term of performance 2 RAC nodes will not give you as good a performance as one machine with same capacity [i.e. 2 nodes each 2 cpu will perform slightly lower than 1 node with 4 cpu]. I believe this is because of overhead of inter-instance commnications.
There are several good papers on linux RAC. The one to start with is Jeff Hunters: How to build a cheap RAC. This demonstrates the working and setup of RAC at a very little cost. You can find this paper on
</code>
http://www.oracle.com/technology/pub/articles/hunter_rac10g.html
But before that I think you can have a read at this paper "You Probably Don't Need RAC" by Mogens Norgaard. Its a very good insight into what RAC is and who really needs it.
http://www.miracleas.dk/WritingsFromMogens/YouProbablyDontNeedRACUSVersion.pdf <code>
Another thing to keep in mind is choice of hardware and software [esp, clusterware]. Now you can go for Oracle clusterware, but my test [which I am sure has several flaws ! ] suggests that Oracle clusterware is not as stable as Sun Cluster 3.1.
Hope this helps.
-bipul
RAC considerations.....
Mark J Bobak, August 22, 2005 - 8:53 am UTC
My thoughts mostly fall in line with what others have said,
with one exception. In my view, if you're not buying RAC
for the high availability, I'm not sure it's cheaper. You
have to consider that the RAC tax is 50% (list price for
Enterprise Edition goes from $40k/cpu to $60k/cpu for RAC,
but don't quote me, contact your sales rep.)
That extra licensing cost could go a long ways towards
hardware when comparing one monolithic server vs. a RAC
of 3 or 4 smaller ones.
Having said that, I agree with much of what others have
said:
Make sure you have the proper training Take the time to do a test implementation. Try to hire someone who has RAC
experience, or spend some money w/ Oracle consulting for a
few weeks of someone's time who has RAC experience.
Consider that moving from single instance to RAC adds a
layer of complexity to the O/S (clustering) as well as the
Oracle code. There is overhead here. It's not free.
-Mark
Training is critical
Shawn Brockway, August 22, 2005 - 9:52 am UTC
Above and beyond all of the research and testing that has been recommended, definitely take time to get soem training before you start. The very best advice that our Oracle Sales Rep provided me when we were evaluating RAC was to attend Oracle's RAC class. Again, Oracle training is not cheap, but the course was well worth it. The instructor that I had went into to depth on many key areas. He explained things like how a single instance database uses the buffer cache and then explained how RAC uses the buffer cache highlighting and explaining the differences and the impact of the differences. I left the class with a clear understanding of how RAC works and what RAC means to applications. We haven't implemented yet because upon returning from the class I evaluated our application and found many of the flaws that were suggested in the class to lead to failures with RAC. Our application won't scale the way it needs to with RAC so for us at the time RAC would have been a disaster. Having a thorough understanding of how RAC works is critical. I could have had the best running RAC instance in the world, but as soon as the application department would have turned up their applications I would have looked like an idiot because it would have failed miserably (It did when we tested it). I knew our application would fail during the test and I knew why. Had I just read documentation and not taken the class, I would have been very surprised by our RAC test and its failure.
ASM "is" ready for Primetime
Arup Nanda, August 22, 2005 - 12:07 pm UTC
In repsonse to the reader with the subject "Is ASM ready fo rprime time production deployment", I would say with definity - yes, it is.
We use several 10g RAC on environments, and one on the mission critical database system. On one system we use ASM and we have been using that for a little more than a year now. Of all the issues we had - bugs, challenges, "opportunities for improvenent" ... - ASM had the least amount os issues. If you go ASM now - with 10.1.0.4 patchkit, you are virtually assured of the bugs the pioneers had to endure a year ago, at least on ASM.
Who told you that ASM can't handle maultiple paths from SAN to host? We use an EMC DMX SAN with connectivity to a FC switch and we use two paths. ASM does not know anything about the underlying paths. If the disks are presented to host, ASM wakes up to life; the way the disks are presented in irrelevant to ASM. As for high volume - one of our ASM instances is used for RMAN backups; so a high volume is granted. However, for that ASM instance, we use SATA drives; so I really couldn't say much about performance.
I will reiterate what others have said about RAC - it's not a HA solution. You still have a single point of failure - the SAN; but in reality, it's far more resilient than nodes. You are protected from instance failures such as shared pool issues, memory and so on.
The limitations:
1. You still can't mix and match different hardware (even if the same OS), such as Linux on Itanium and Intel x86
2. You can't have different hardware on the same RAC
You can't do rolling upgrades of nodes; remember, there is one CRS for the cluster.
3. If your SAN fails, so does the voting device and hence the cluster fails. 10gR2 has provision for multiple voting devices.
Ideal node configuration
Logan Palanisamy, September 08, 2005 - 1:57 pm UTC
Tom,
Based on our business needs, we came up with 12 as the number of CPUs needed for RAC cluster. The question is how do we split the CPUs among different node.
Configuration in the order of preference:
#1. 3 nodES with 4 CPUs each.
#2. 2 nodes with 4 CPUs each, and 2 nodes with 2 CPUs each.
I am inclined to go with #1 because #2 is a mixed configuration.
Ideally I would like to go with identical nodes for ease of maintenance (similar spfile, etc). Is there any reason not to go with a mixed configuration in a RAC setup? Are there any other problems we should be aware of?
Thanks.
September 08, 2005 - 4:40 pm UTC
equal resources is best, that is what it is expecting, you have to tweak parameters otherwise.
Testing it with desktops
Puja, September 30, 2005 - 2:04 am UTC
HI Tom
Can I have RAC setup on my desktop machines? I have two machines with similar configuration:
MICROSOFT WINDOWS 2000
SERVICE PACK4
INTEL PENTIUM 4 CPU 2.80 GHZ
1GB RAM
80 gb HARDDISK WDC WD400BB - 75FRA0
I can arrange for the software and special hardware if required. But I need to know what they are and how to configure them.
I read the document 178882.1 from metalink, and found it useful enough. But I need to know more detailed steps regarding hardware configuration. Can you post any links/ pointers?
Thanks and regards
Puja
September 30, 2005 - 9:07 am UTC
I'm not familar enought with windows to say 100%.
with 10g, you have everything you need software wise, it is recommended to have a private network for the interconnect but not mandatory.
The biggest thing for you will be the shared disk (80gb - that is small... and not shared).
You might be interested in just getting the vmware images that let you play with RAC on a single machine - to get your feet wet.
</code>
http://www.oracle.com/technology/tech/linux/vmware/index.html <code>
Must have redundant interconnect
A reader, September 30, 2005 - 10:21 am UTC
From personal experience, I would say that any production RAC implementation must have redundancy in the private interconnect. Recently, when implementing my first real life two node RAC, I tried to simulate "split brain" by disabling the private interconnect interface. Boom, the second node crashed. This happened repeatedly. A TAR with Oracle confirmed that when "split brain" happens, Oracle CSS will reboot the node which lost connectivity. This will happen for every OS platform and is documented in 10gr2 docs.
I have had no issues with ASM except if one uses OMF, they must create aliases for datafiles, controlfiles and spfile. With OMF, Oracle assigns unique names to files in ASM format. Without aliases, the REUSE clause will not work, create controlfile statement will fail, create spfile will be problematic. After a restore of controlfile from backup, it will not be possible to reuse the TEMP tablespace tempfile without alias. Oracle will create a new tempfile with the old tempfile just lying there taking up disk space. Basically, any operation which wants to overwrite an ASM format OMF will fail. With alias, there is no problem. Just use the alias name.
Conditions for implementing RAC?
George, January 24, 2006 - 8:44 pm UTC
Excellent discussion... but more questions linger for me.
My company is currently on 9i and considering a 10G RAC, mostly with hopes of improving performance without rewriting large segments of code. Initially I doubted the impact because I didn't understand how the interconnect manages disk i/o. More recently we have realized that our bottleneck is likely just the many procedures and triggers upon which our application is built.
I am currently building a test environment in 10G, but we know it will be very hard to relate that performance to our production environment. Can you provide any insight for the masses out here who need to decide between, say, a 4cpu server or 2x2 cpu servers in RAC? Or even what sort of application performance we might see going from a one 4cpu machine to a 2cpu cluster everything else being the same?
I understand that you'll probably have to include some assumptions to say anything meaningful, but I hope you will comment - (we) trust your opinion.
Thanks, George
January 25, 2006 - 1:11 pm UTC
are you saying you are having issues scaling inside of a single box? eg: if you go from 2 to 4 to 6 to 8 cpu's in a single machine - would you expect your application to scale accordingly?
RAC and Grid
A Reader, January 26, 2006 - 10:20 am UTC
I thought in 10g, the g (Grid) is to use or share resources. Do we have to have RAC to use the grid? Or do we have to have Grid to RAC? I still don't clearly understand the relationship between RAC and Grid, or the difference. Please explain.
Thanks.
January 26, 2006 - 10:45 am UTC
RAC is a (optional) component of grid.
automatic storage managment (ASM) is too.
transportable tablespaces are.
datapump is.
grid control is.
many things are.
Conditions for implementing RAC?
George, January 30, 2006 - 11:07 am UTC
I feel exposed to a degree... the question of whether our application will perform better on stronger hardware isn't something I had really even thought about until your question - and I'm not quite sure how to judge that. We used to run on some very antiquated Sun equipment and when we upgraded to RH Linux on a 2CPU Dell, we saw huge performance gains; meaning response times improved dramatically. Then we hit a high demand period where our "front door" saw many thousands of requests per minute and we discovered that our application could only handle a relatively small percentage (we started seeing deadlocks on some key tables where processes were not completing.) We've made some real improvements by moving processes out of the database onto our web server, but we don't think we've seen the last of our performance issue.
So I guess my expectation is that we would see dramatic improvments by introducing a single bigger box. And the question, although perhaps a moot point now since our discussion about availability is taking us down the RAC configuration separately, is still on the order of "how will 2-2cpu machines perform relative to 1-2cpu machine or 1-4cpu machine"?
thanks again, George
January 30, 2006 - 3:15 pm UTC
unless you can scale "in a box", it is doubtful you will scale "across boxes in a cluster".
If you can scale "in a box", it is very likely you will be able to scale "across boxes"
VIP & private network
Yogesh, February 18, 2006 - 10:13 am UTC
I have couple of doubts about RAC networking.
One scenario
Private network is implemented in four-node cluster. Node 2 failed due to some error. Something like this
1=X=3=4
How node 1,3 and 4 will interact using private interconnect? There is always TCP/IP public network, but I guess due to speed factor we can't use it for sharing memory.
Similarly how VIP will takeover the failed node?
February 18, 2006 - 4:44 pm UTC
the private interconnect is a network - a private network, the failure of a node doesn't affect it.
using RAC in production not dev or test
Sara, March 08, 2006 - 4:35 pm UTC
I am new to RAC and do not fully understand the implications as a developer of using RAC. Due to budget constraints, our development and test environments will not be RAC, however our production environment will be. I am building a data warehouse using OWB. Is there anything that I should do differently for RAC?
March 09, 2006 - 12:52 pm UTC
... our development and test environments will not be RAC ....
run away, run away - do not walk, just run away.
why don't you develop on windows and deploy on linux on ZeOS as well?
And make sure to use a single byte character set when developing but field with multi-byte.
And program using an American language but field using French Canadian or something
And remember, test and development only need say about 1 or 2 rows per table to test with, it'll be OK.
Because I'm sure there are no differences between them...
Sorry - this is a really bad idea. You'll not save a dime, you'll waste tons of money though.
Hardware, O/S and DB version restrictions for nodes in RAC
Mathew Butler, April 11, 2006 - 12:00 pm UTC
9.2 and 10GR1 and 10GR2
Are these a requirement for each node in a RAC?:
1) O/S must be the same type, version and patch level
2) DB version must be the same base version and patch level
I believe that the underlying hardware can be different (#CPU and memory) - but if it is this may require some additional tuning of the cluster. This is not recommended, as the cluster expects each node to be the same configuration.
I'm sure this is documented, but I can't find the reference on tahiti.
Say I have a number of DB to manage at version 7, 8i, 9i, 10G RAC. Can I administer all of these via grid control?
Best Regards,
April 11, 2006 - 5:58 pm UTC
OS must be the same "type" and version - patching can be done in a rolling fashion.
Same with the database in many cases (the patches can be rolled if permitted)
but in general, both will be the same.
the physical size/configuration of the boxes can be different.
grid control can do things with 8i and above.
Many Thanks.
Mathew Butler, April 12, 2006 - 8:59 am UTC
RAC Licensing
Yogesh, May 05, 2006 - 11:41 am UTC
We are in the process of finalizing the Sun hardware for RAC. Initially we decided to go for Sun Fire V890 Server with 8 physical CPU's. But recently somebody came up with a server Sun Fire T2000 Server, which has 8 cores, 1 CPU/chip, which is 10 times cheaper than V890.
As per oracle software price list RAC license will cost 20000$ per CPU.
In above scenario how much a licence for EE + RAC will cost in case of T2000?
As well if one is using T2000 servers, will oracle database software be able use "PARALLEL" functionalities?
May 05, 2006 - 2:55 pm UTC
I don't do pricing, that is for your sales guy.
The database does parallel on a single cpu, it does it on hyperthreaded cpus, it does it on dual cores, it does it on smp. so "yes" to the parallel thing
Dual core CPU's and performance
Yogesh, May 08, 2006 - 6:51 am UTC
Have you personally used these dual core, 8 core etc... CPU servers? Do they have same processing power as physical X number of cpu's? I know one need to benchmark these things... But please share your experience.
May 08, 2006 - 8:20 am UTC
multiple physical cpus are obviously better than dual core are obviously better than hyperthreaded.
Everything you read about them from the vendors will tell you this as well. It is why
a) 2 physical cpus are 2 physical cpus from a license perspective
b) a dual core machine is not licensed as 2 full cpus
c) a single hyperthreaded cpu - appearing as 2 cpus - is licensed as a single cpu
Not sure I'd agree
Howard Rogers, May 09, 2006 - 7:47 am UTC
Dual core means 2 independent CPUs on one chip. That's actually better than two physically separate chips connected by some externalised bus or other.
Hyperthreading is a completely different issue, and there'd be no complaint from this corner if you were to declare it to be a bit of a damp squib.
But to state that "multiple physical cpus are obviously better than dual core" is missing the point that dual core IS multiple physical CPUs.
May 09, 2006 - 8:30 am UTC
ok, mistated, but dual core has "not the performance" in a SMP box of "two physical chips".
But that's the point...
Howard Rogers, May 09, 2006 - 5:28 pm UTC
But that's the point. A dual core chip is *better* than two independent CPUs in an SMP box. The communication between the two CPUs takes place at the speed of the chip, not the speed of the communication bus between two single-core CPUs on a single motherboard.
Of course, the performance of a server depends on more than the CPU, and a properly-designed SMP box will out-perform a box designed for single CPUs which happens to have had a dual core chip fitted, because of those other design factors. But if you have a box designed for two-chips, and you instead fit a single dual-core chip, performance will be better, not worse. See for example: </code>
http://www.pcstats.com/cfarticleview.m?articleid=1797&page=5 <code>
And you might note that Intel's dual core chips have improved since that article was written: their latest dual core chips have independent/differentiated memory caches, too.
So it's a false dichotomy to talk about SMP and dual core as if they are different things. Dual core *is* symmetric multi-processing, and performs better that two single-core procesors, other things being equal.
OK, I'll shut up now.
Howard Rogers, May 09, 2006 - 5:36 pm UTC
I suppose it would help if I could read the articles I cite properly before quoting them. The last link I provided says precisely the opposite of what I thought it said, and supports your position, not mine.
However, </code>
http://www.pugetsystems.com/articles.php?id=23 supports my original argument (and is more up-to-date, too).
And a lot of the comments here
http://discuss.joelonsoftware.com/default.asp?joel.3.290648.9 <code>are similarly supportive.
But it sounds like it could become one of "those" questions, so I'll shut up about it now!
If it can't scale on one box, it won't scale in RAC
Charles Forbes, June 06, 2006 - 11:04 am UTC
You mentioned a couple of times that RAC doesn't do any magic as far as scalability goes, and that an un-scalable application in a 1-server environment won't scale in RAC either.
There was also a note, though, from a prior poster which mentioned that the Oracle RAC class teaches you how to develop an application that will scale on RAC. That sounds like, "An application that might scale on a single server might not scale on RAC."
Could you explain what Oracle was trying to get at by giving that advice?
Thanks,
Chuck
June 06, 2006 - 11:23 am UTC
point to it so we have "context" please.
RAC Testing
Matthew Lange, June 06, 2006 - 11:33 am UTC
Hi all, I was just perusing this thread, and thought I should point out the following OTN How-To: "Build Your Own Oracle RAC 10g Cluster". It's a great step-by-step made for those looking to make a simple, low-cost, TEST RAC for learning purposes.
</code>
http://www.oracle.com/technology/pub/articles/hunter_rac10gr2.html <code>
Cheers!
If it can't scale on one box, it won't scale in RAC ... cont
Charles Forbes, June 06, 2006 - 12:11 pm UTC
The post I was referring to from above:
Training is critical August 22, 2005
Reviewer: Shawn Brockway from Columbus, OH
"I left the class with a clear understanding of how RAC works and what RAC means to applications. We haven't implemented yet because upon returning from the class I evaluated our application and found many of the flaws that were suggested in the class to lead to failures with RAC. Our application won't scale the way it needs to with RAC so for us at the time RAC would have been a disaster."
Thanks,
Chuck
June 06, 2006 - 1:44 pm UTC
It is likely he is saying "our application would not scale, let alone not scale on RAC"
The class teaches you that RAC is not "scale as if by magic, fixes all broken implementations and makes them automagically scale"
If it can't scale on one box, it won't scale in RAC ... cont
Charles Forbes, June 06, 2006 - 1:00 pm UTC
June 06, 2006 - 1:54 pm UTC
There are not many "rac singularities", it is that things that don't scale well on a single box scale perhaps EVEN LESS well.
example: everyone must update the same single row (gap free sequence generator, hit counts, whatever). Doesn't scale well on a single instance, scales less well in a clustered environment.
example: hard parses - lots of latching in a single instance. Add another instance that needs to know about what the other instance is doing and you make it worse.
private interconnect btw nodes
Robert, June 06, 2006 - 4:19 pm UTC
>>
Followup:
You'll want a private interconnect between the two, the faster the better.
Separate from the public network.
<<
A bit off-topic but does this "private interconnect" requirement means the
boxes need to be directly cabled together ? meaning they have to be at least in the same building ?
June 06, 2006 - 4:35 pm UTC
Typically - RAC is "for a room". The speed of light starts coming into play here - the shorter the cables, the less the latency of communication between the instances.
There are stretch clusters for "longer distance", but RAC is not about disaster recovery - it is for continous availability in a room.
Dataguard is for disaster recovery, when the room goes away.
Things worse on RAC
Daniel, June 07, 2006 - 2:31 am UTC
As Tom said , a few things bad on single-instance Oracle get worse on RAC...
Here is an example of something which is actually perfectly OK on non-RAC but is really bad on RAC :
Imagine an OLTP application using an application server which pools Oracle connections. A business transaction performs several updates to the same row( such as customer account) , which are spread accross multiple service calls.
Each service call is an Oracle transaction ( i.e. does a commit), so there are no locking issues nor any need to use XA.
This works fine on a single instance .
On RAC the Oracle sessions working on the same business transaction may well be on different nodes , meaning lots of GC block requests and "pinging" , degrading the performance by an order of scale.
Of course this can be fixed by routing the service calls relating to the same business transaction to the same Oracle instance, but this is exactly the point I am trying to make -
the old OPS tuning mantra "partition the load" is (sadly) alive and well; not every application which scales on non-RAC will scale on RAC without (possibly extensive)RAC-specific tweaking.
June 07, 2006 - 6:58 am UTC
that doesn't work fine on a single instance, that has
a) data integrity issues
b) imposes log file sync waits on the application
A business transaction should not be spanning multiple Oracle transactions like that from an application server.
OK on no RAC , bad on RAC
Daniel, June 07, 2006 - 5:31 pm UTC
>a) data integrity issues
>A business transaction should not be spanning multiple >Oracle transactions like that from an application server.
I guess I should have been more carefull with my terminology. One could say it is not a "business transaction" but rather a "business process" . Different stages of it relate to exchanges with external devices and the functional requirment dictates a COMMIT at every stage.
>b) imposes log file sync waits on the application
True , but the scalability limitation imposed by this hits at much higher throughput rates than RAC "pinging" issues.
Re-visiting Original Question
James Valenti, July 24, 2006 - 6:33 pm UTC
So after considerable discussion here about RAC (thanks to all participants) I am convinced that scalability and availability are two possibly very good reasons to go with RAC. However, embedded somewhere in my original question, but not really discussed, I asked about the possibility of people using RAC to simply upgrade hardware. The current IS solution being a monolithic hardware platform that has to be ‘fork lifted’ to be replaced. Suppose I don’t really care about availability or scalability as much as all the work and money involved in replacing a large single platform system every five years or so. Are there IS shops using RAC to simplify hardware replacement by adding the latest server technology to the ‘head’ of the cluster and chopping off the obsolete ‘tail’ of the cluster? Is this a legitimate expectation, that in addition to availability and scalability, RAC provide the ability to ‘slide-in’ new hardware?
July 24, 2006 - 6:40 pm UTC
it is technically feasible IF the new hardware is the same architecture and same OS
a reader, July 25, 2006 - 4:00 am UTC
I have got an impression that Rac will perform in a most efficent way if all nodes in system are about same performance level.
Because there will memory references to other nodes in rac cluster (e.g. when it is found out which node is mastering blocks) then if one node is less efficient than others it will 'slow' other nodes. Faster nodes have to wait answers from slower nodes.
July 25, 2006 - 10:57 am UTC
it is "best practice" to use equally sized hardware - yes, otherwise you have to set some init.ora parameters to re-adjust the loading of the machines, something you really do not want to do.
however, if you just wanted to migrate to "new hardware", it would be technically possible to do it - add two new "bigger better machines" to a two node cluster, slide out the two "older pieces of hardware". temporarily unbalancing the processing power.
is Cache fusion is a costly operation..give some insight
Ajeet, July 25, 2006 - 6:51 am UTC
I am working a RAC project - in the benchmarking stage..one of our local expert says that cache fusion is very costly operation so instead of using it in RAC ,we should use manual load balancing such as divide the load using application - something like partitioning the input data etc etc - but I feel this way we will be using half of the so many features provided by Oracle RAC..but this discussion goes on - so my question is it any known issue with cache fusion .
Tom what is your view.if you provide a little bit of insight here that would help us understand this.
i have gone through the oracle documentations on cach fusion and to me it seems pretty cool and harmlesss.
I am going to start with let oracle do what it is suppose to do approach of your.
if you can give some useful pointers ,links that would also be very helpful.
Best regards
Ajeet
July 25, 2006 - 11:27 am UTC
there are "extreme cases" (eg: a small table, say on a single block, with a row every node in the cluster wants to update) whereby the cross instance transfers would be a real problem.
But, they would be a real problem without a cross instance transfer as well (if it doesn't scale - it won't scale even worse on RAC, that small hot table doesn't scale in the first place)
It would basically be almost impossible to partition based on input data when push comes to shove. Data models are generally far to complex to perfectly partition like that. Application level partitioning is pretty easy to accomplish (eg: run HR on nodes 1 and 2, run Financial on nodes 3 and 4 - if something fails you still have 3 nodes to run everything on).
However, I would suggest you benchmark - as you are doing now - with the 'simple' implementation (making sure it can scale IN THE FIRST PLACE! with just a single computer).
Don't engineer a solution for a hypothetical situation.
just a reader, July 25, 2006 - 8:01 am UTC
Using services in rac 10g systems is very powerfull feature and surely something that should be looked at.
( With it one could handle those problematic situations in rac (e.g. xa -transactions and transaction branches going into different nodes).
)
Scalability is maybe less, but availability is same as in a cluster without services.
If I have understood it correctly:
- using services will (if services are using separate database tables) reduce interconnect traffic because blocks will exist only in one machine
- however there will still be interconnect traffic because house keeping information of blocks may still be in different nodes than actual blocks.
July 25, 2006 - 11:42 am UTC
using services CAN do that if a service = a single instance. That is not what a service is in general however, a service is one or more instances in general.
ASM instance on cluster nodes
Yogesh, July 25, 2006 - 2:44 pm UTC
A small doubt. If we are using ASM as a volume manager for RAC, we will need ASM instance on both the nodes. I guess we'll need to use same set of LUN's (used on first node) at the time of creating second ASM instance. At this time will ASM complain that they are already being used?
July 25, 2006 - 2:47 pm UTC
No, it won't.
ASM (just like say veritas) would have to run on each node.
And just like veritas - it works just fine.
The metadata about the disk used by asm is stored in a fashion that all instances of ASM can get it.
ASM uses private or public interface
Yogesh, July 25, 2006 - 2:50 pm UTC
That was really a quick reply!! Amazing ....
So both the instances on nodes will communicate. Will they be using Private Interconnect for that? Just thinking of additional overhead.
July 25, 2006 - 2:54 pm UTC
they have this disk - metadata is on disk.
this is nothing that any other clustered file system thing would need to do.
Voting disk?
Yogesh, July 25, 2006 - 2:57 pm UTC
July 25, 2006 - 3:03 pm UTC
no, header segments. metadata stamped onto disks.
Thanks for this valuable information
Yogesh, July 25, 2006 - 3:04 pm UTC
Is RAC not good for Data Warehouse Applications
James Valenti, July 27, 2006 - 1:52 pm UTC
Tom,
An colleague of mine pointed me to an article on the web about RAC. The author presents arguments against using RAC in a data warehousing environment. (
</code>
http://www.dba-oracle.com/t_rac_clusters_data_warehouse.htm <code>)
It would be very interesting to hear comments on this article from you or others following this discussion.
Is RAC not a good solution for running full table scans in parallel against very large tables (50-75 GB)?
July 27, 2006 - 2:24 pm UTC
This one quote should throw up a sufficiently large red flag:
"I have never heard of any shop using RAC for a data warehouse, and I could imaging large parallel operations clobbering the interconnect as to make RAC somewhat sub-optimal for data warehouse processing".
That indidcates the authors don't have any experience with the technology in the area they are saying it is not applicable for!!
Think about the impact of that quote.
Premise: RAC is not for warehouses
Supporting information: We just think so, we have no proof points, we've never actually done it, we've never actually talked to anyone that has done it.
Just ponder the implications of that......
I'm not going to rip it apart point by point here. But don't you think the authors should have something a bit more concrete than "we haven't done it, so it is not doable"
We have many customers running warehouses with RAC.
file system on 10G RAC
Tony, July 27, 2006 - 4:26 pm UTC
Hi, Tom
I've read your review of file system on "expert oracle database architecture". It's brief and I would like to know what's best option for 10G RAC on Linux. What's the advantage ASM has over raw partition and OCFS? Thanks a lot.
July 27, 2006 - 4:38 pm UTC
RAW partitions are all or nothing - the partition is "the file". Your unit of access is that device.
ASM gives you a filesystem - consisting of many underlying raw devices.
I would likely not consider RAW and would use ASM over OCFS for database files, OCFS for trace/alert areas.
scaling rac
Ryan, July 27, 2006 - 4:41 pm UTC
We are considering moving to RAC in the future. We are currently doing design and development, but we are planning for a possible scale up to RAC with a later release due to the estimated volume. We want to plan for it now and get it into the architecture so we don't have to make significant changes in the future.
Our concern is the possible need to 'pin' datablocks to specific nodes. How important is it try to write code so that datablocks consistently get accessed by a specific node?
Right now we are trying to design our system around this for the possible future scale up. We are partition tables with the intention that specific partitions will be primarily accessed by specifc RAC nodes. This is easier for us because we have primarily back ground jobs and we can tell DBMS_SCHEDULER which node to use for each job.
It's not perfect. We will need to do some data sharing between nodes.
Sorry for the long lead up. Does Oracle have any benchmarks showing the costs and benefits of doing this? For example if we completely ignore pinging and let all our blocks move between nodes vs. trying to isolate blocks to specific nodes as much as possible.
July 27, 2006 - 5:41 pm UTC
there is no such possible 'need' to second guess the software. You won't need to 'pin' blocks to specific nodes.
RAC is designed to to data sharing between nodes. It is unavoidable.
What you need to do is build something that scales - period. On a single machine. RAC will amplify scaling issues. Worry about making it *scale* period.
is it optimized for it
Ryan, July 28, 2006 - 1:21 pm UTC
Is there any advantage to try to access certain datablocks on certain nodes?
I believe the docs state that:
Node A: changes a datablock in memory
Node B: requests it with a broadcast message
Node A: must flush it to disk
Node B: then queries it from disk.
So you are saying there are no advantages at all to building software for RAC that tries to keep the access of datablocks to specific nodes?
July 28, 2006 - 8:47 pm UTC
unless your data is "embarassingly partitionable" (meaning really really really partitionable all of the way up and down your relations), it just isn't something you can realistically achieve.
Think about DEPT -> EMP -> PROJECTS
how do you "partition" that data if.....
some applications use "where deptno=10" and join dept to emp to projects.
other applications use "where project_name = :x" and join projects to emps
It just isn't realistic.
the flushing to disk - that was OPS (Oracle Parallel Server), it is done via the high speed interconnect in RAC.
David Aldridge, July 28, 2006 - 3:23 pm UTC
sepearing on different nodes
Ryan, July 31, 2006 - 9:31 am UTC
If it is really, really partitionable... is there any value in doing it? Our project is all background jobs. We think we can seperate out the vast majority of them.
Is there any value in doing this?
Also, I thought the highspeed interconnect for RAC was only true for data blocks that were not changed. Its also true for datablocks that changed? They don't get flushed?
July 31, 2006 - 10:15 am UTC
Under OPS (ancient stuff, software from way back in the last century), we used the disk.
With the advent of RAC, the interconnect is used for all things, disk pinging is not there.
Is there any value? Doubtful - I say doubtful because I've never seen a schema for any real application of any real value that was perfectly partitionable. So, the value goes out the window.
If you are that rare thing that is perfectly partitionable, yes, you can reduce the amount of interconnect traffic this way.
It is most commonly achieved by running two separate applications on a two node cluster - app1 uses node1, app2 uses node2 and in the event of failure, they have a hot failover (virtually instantaneous failover) solution. That is "perfectly partitioned"
But a single application - the second you introduce parent/child/child - it becomes a real issue trying to partition that. You partitioned the parent by primary key suppose, the first child by foreign key, the next child??? it typically does not have the parents primary key (if you use surrogates like most people do)
RAC
A reader, August 01, 2006 - 10:35 am UTC
Tom,
I know that you always advise installing one database per server. In case of RAC, should I install one database per cluster?
I have a situation where client wants to install 15-18 databases on a single 3 node cluster. Is there any documentation from Oracle which says this is not such a good idea?
Thanks
August 01, 2006 - 6:42 pm UTC
one INSTANCE per machine.
maybe less than one database :)
4 nodes - 4 instances, 1 database in a RAC cluster...
Look, ask them "why are you buying RAC - is it for availability? Let me tell you why this is 'low availability' if you do this. Is it for scalability? Let me tell you how to scale this puppy..."
It comes down to *common sense* in this case.
It would be sort of like documentation from Sears stating that using your gas powered push mower as hedge trimmers would be a bad idea.
Sun's T2000, any luck?
A Reader, August 14, 2006 - 2:38 pm UTC
I hope I am not posting to a wrong thread. This is the only place I can find Sun's T2000 related topics. As a matter of factor, we are planning doing the RAC on T2000 server as what Yogesh is doing or has done. If Yohesh can provide more info on how to take advantage of the 8 cores, it would be great. Our system functions like a ONE CPU server.
Our primary test with a single server shows the database is not as fast (not fast at all, N times slower in load tests) as on other servers with multi-CPUs. We thought that could be because of the configuration things on both OS and DB levels but still no luck or not too much progress after many tries.
Does anyone reading this site have successful stories on how to configure Oracle 10g R2 on T2000 server? Any helpfull tips would be appreciated.
Thank you.
August 14, 2006 - 3:09 pm UTC
what does the OS report back as the number of cpu's?
true multi-chip will perform better
than multiple cores on a chip will perform better than
hyperthreading
there is nothing special about configuring oracle on a platform like that.
Sun's T2000
A Reader, August 14, 2006 - 3:55 pm UTC
When I run the following query:
select ksppinm name, ksppstvl value, ksppdesc description
from x$ksppi x, x$ksppcv y
where (x.indx = y.indx)
and ksppinm like '%cpu_count%'
order by name;
I got the following results:
NAME VALUE DESCRIPTION
--------------- -------------------- ----------------------------------------
cpu_count 32 number of CPUs for this instance
Oracle thinks the T2000 has 32 CPUs. I don't know how it's count. It should make it much faster not slower with that many "CPUs".
Thank you.
August 14, 2006 - 4:09 pm UTC
why would you do that query when a simple:
ops$tkyte%ORA10GR2> show parameter cpu_count
NAME TYPE VALUE
------------------------------------ ----------- ------------------------------
cpu_count integer 2
would do it???
why would it run "faster" with that many cpu's. for example, if your load is a "single path" load (not done in parallel), then I don't care if you have 1 or 1,000 cpus - you are only using one cpu (and likely bottlenecked NOT by cpu, but disk)
Now, again, how many cpu's does the OS report you having
And why do you think Oracle is not using them
RAC for scalability, other options...
A reader, September 01, 2006 - 11:43 am UTC
Tom,
We are looking at the option of implementing RAC, particularly for scalablity and then for high availability. I would like to know what other options are available to increase scalability other than RAC.
Thank you for your time
September 01, 2006 - 12:07 pm UTC
good design :)
that is the single biggest factor. Period...
whether it be rac, smp, whatever
without a good design, something built to scale in the first place, hardware isn't going to do a thing for you.
A reader, September 02, 2006 - 9:12 am UTC
I would like to add further to Toms recommendation, good understanding of RAC its feature and how they work otherwise
you will always be running into problems.
asm vs cluster file system
A reader, September 05, 2006 - 6:27 pm UTC
Hi tom
Can we use ASM in place of cluster file system.
September 06, 2006 - 7:45 am UTC
yes, ASM provides shared access to a filesystem across many nodes in a cluster. It is in fact a clustered filesystem.
asm and clusterware
A reader, November 13, 2006 - 6:41 am UTC
"yes, ASM provides shared access to a filesystem across many nodes in a cluster.
It is in fact a clustered filesystem."
.. assuming you are only working with oracle files =) ... other files can be stored if you put them inside the oracle database =)
dbms_scheduler and RAC
Greg, April 23, 2007 - 9:16 am UTC
Hi Tom,
Wasn't sure where else to ask this .. so I hope this is a quick enough question and not too far off-topic .. ;)
We are currently evaluating RAC, and we had a couple questions on how dbms_scheduler behaves with the nodes and such.
If we have 3 nodes (for example), and on Node 1 a job is running, that launches a dbms_scheduler.create_job command ...
1) Which node will the job launch? Node 1 (ie same as dbms_scheduler session) or a "random" node (meaning - whichever RAC decides to use - ie same as if another user logged in and ran it?
2) (this question may be moot depending on the answer to 1) Is there any way to control which node the dbms_scheduler launches on? (I understand we don't want to do this - I just want to understand *all* options ... good or bad ...)
3) Does it matter if the job is launched as a "run immediately" or "run at a certain time" ??
April 23, 2007 - 4:38 pm UTC
dbms_job/warehouse
Kscott, April 24, 2007 - 8:30 am UTC
We run a 2TB warehouse on RAC. Our batch process each night makes use of dbms_job to submit multiple ETL processes to the 2 nodes of the cluster. If an instance is not specified, RAC does a very good job of load balancing between the 2 nodes. In other words, if you have enough job processes, on the average, half the jobs will run on each node.
alternatives to rac
Ana, September 12, 2007 - 11:12 am UTC
I just got out of a meeting centered on high availability (we can tolerate upto a few minutes downtime) options for databases. I had always been under the impression that Oracle databases needed Oracle solutions - either RAC or dataguard depending on individual needs. The system admin threw in his opinion - he said he had set up Oracle products (even version 7) in clustered environments in his previous places of work - he had used OS clustering (veritas). I wonder how does Oracle work on just OS clustering (without database clustering). Does it mean I don't need RAC as long as I am okay with the active/passive failover that OS clustering allows?
Thanks
September 15, 2007 - 4:42 pm UTC
it is a cold failover - when machine A fails, the disks are mounted to machine B and you startup the database on machine B.
You have a machine sitting there doing not much, until you lose machine A in that scenario. Like having a spare computer in the closet.
RAC & Application Design
RAM, October 09, 2007 - 12:23 pm UTC
Hi tom,
Thanks for all your efforts.
We are in need of producing a High availability solution for a critical application, we did propose 10g RAC with DataGuard in place.The same was sent to Design Authority Team to validate the same.Despite of all the good efforts and our understanding about the High Avaliablity features below is the response we got from the Design Team.
"At the moment the only consideration for HA is the prospect of implementing a RAC based DB solution. However, this is only a light HA solution, and can actually increase down time. The application design and implementation of change must be geared to no or limited downtime. It is not acceptable to suffer multiple outages for maintenance/upgrade changes to the DB, that are not specific to the current platform. "
We are not sure what all things we need to take care on the application design front for High Avaliability.Please can you provide your valuable comments..Are we missing something serious?
Thanks
A reader, December 03, 2007 - 11:24 pm UTC
Tom,
We will be building a new project with 3 node cluster,Sol 10 and oracle 10203. I recommended going with 2 node cluster with enough/required processing/failover capacity instead of 3 node as cache fusion,resource mastering,hashing,etc, etc is much simplified in 2 node cluster compared to 3 node and I also recollect reading(not able to find the link) clusters should be scale/designed in range of 2 (2,4,6,8)instead of odd numbers.
Can you please provide your take on this whether my understanding is correct and if you could point to some links/docs would be greatly appreciated.
Thanks.
Moving RAC database
A reader, February 25, 2008 - 9:53 am UTC
Tom,
I recently ran into a situation where I had a cold backup of a 3-node RAC database. I found that it is not possible to restore this backup to a separate 2-node RAC. There are references to the 3rd instance which are impossible to get rid of. I have looked all over the documentation including duplicate database command of RMAN but there is no documented method to move a database from 3-node RAC to 2-node RAC using backup/restore/recovery route.
Can you please point me to some documentation which describes how to do this? I am in a situation where I have to move a database from original 3-node cluster to a separate more powerful 2-node cluster. Export/import is ruled out because of size of databases. Oracle 10gr2 using RAC and ASM.
Thanks
February 25, 2008 - 1:52 pm UTC
first, tell me what references you are talking about...
for you see, a 3 node rac cluster that suffers a failure of a node becomes a 2 node rac cluster all by itself - not sure what specific issues you hit?
Moving RAC database
A reader, February 27, 2008 - 9:18 am UTC
The issue is that I have a cold backup of a database which was running on 3-node RAC cluster. The original database and 3-node cluster are no longer accessible. I had to create a new database on a totally new 2-node RAC cluster by restoring this cold backup. It is not a node failure situation. It is like creating a copy of a database which was originally running on 3-node cluster. The copy is created from a cold backup on a 2-node cluster.
The issue I hit was that on restoring the 3-node RAC database backup to a 2-node RAC database, the restored database kept showing a 3rd redo thread belonging to the 3rd instance which is no longer existent. Oracle would not allow me to delete these redo log groups complaining that at least 2 redo log groups are required per instance.
Finally, I found a note on Metalink note (415579.1) which detailed converting a RAC database to single instance database. That document had steps on deleting the redundant redo log groups. There were 2 redo log groups (group 5 and 6) in thread 3. Here is what I did following the note:
alter database disable redo thread 3;
alter database clear unarchived logfile group 5;
alter database clear unarchived logfile group 6;
alter database drop logfile group 5;
alter database drop logfile group 6;
speed of RAC vs. non-RAC?
Reid, March 10, 2008 - 4:58 am UTC
We also are considering RAC for scalability and do not have high availability requirements. We will be using Oracle applications - E-Business and PeopleSoft.
Because of the high cost of RAC, we are considering a single SMP server. As part of this process, I am attempting to find benchmarks comparing RAC database peformance to non-RAC.
So far I have ascertained that: TPC has exactly one TPC-C RAC benchmark and it is 4 years old. SAP SD benchmark does not have any. Oracle application benchmark - see
http://www.oracle.com/apps_benchmark/index.html - says, "results comparison(s) between Single Instance versus RAC is also discouraged". Why?
And is there a way to get a recent comparison with a standard database application benchmark?
Thanks.
RAC in batch environment
Gurvijay, September 15, 2009 - 10:19 pm UTC
We are evaluating RAC on 10G R2.
Our application is mainly batch inserts after reading from XML files. Our application is multi-threaded so that it can process more than one XML data file at any one time. The XML files are of the order of 400 MB, and it takes a few hours to insert data from one XML file, as it has a deeply structured data. (A separate thought is required to make it more efficient, I will discuss it separately)
However, our application has very few reads.
We don't need HA.
We need it to be scalable, as more data providers will come on board, we will need more power to process.
We are not very much constrained with performance, as we have large window to process these files.
I can see some benefit for going RAC (scalability only).
Before we pursue this further, I wanted to know if:
RAC is suitable for our purpose. (as defined batch inserts with no HA, yes scalable, no performance)
many thanks in advance.
Regards
snip snip below:
PS: How will I know if I have to post a new question or you have accepted to answer (your pop up window)?
September 16, 2009 - 10:04 am UTC
if it takes a few HOURS to process such a tiny bit of data (400mb - that is TEENY TINY, I don't care how deeply nested something is) - you have issues way outside of the database.
You confuse me
"we are not very much constrained with peformance"
"we need it to be scalable"
Those are oxymoronic - I don't know how you can say the two together in the same thought like this?
can rac be used to spread your processing out ? Only if you are database intensive and it sounds like you are not, you have some relatively inefficient code and hardly do a thing to the database.
RAC in batch environment
Gurvijay, September 16, 2009 - 8:26 pm UTC
Thank you for your input.
RAC + ASM for HA
Victor, October 13, 2009 - 3:44 pm UTC
I guess if we have a RAC with ASM normal redundancy spread across 2 SANs (one being failgroup of the other) it could be considered a HA environment.
accessing different partition of data through different Node
S Roy, May 13, 2010 - 2:42 am UTC
Hi Tom,
Is it performance beneficiary to design our code in such a way that in a RAC environment if we access different partition of the table from different node?
Regards,
S Roy
Instance_Groups Vs Services in 10g RAC
Remo, August 28, 2010 - 12:06 am UTC
Tom,
I have a 3 node 10g RAC and trying to implement parallel query. Kindly clarify my doubts.
1. Why do we need INSTANCE_GROUPS parameter when we have SERVICES?
2. I have created a service with 1 preferred instance and 2 available instance.For this service, how many instance will be used for parallelism? Only preferred instance or both will be used?
3. If users directly connect to one instance without service, how many instance will be used for parallelism?
Thank you.
September 07, 2010 - 7:56 am UTC
1) services are used to connect to an instance. an instance may belong to many services, a service can map to many instances - but it is a method of locating a single instance to connect to.
instance groups tells the current instance you are connected to - what other nodes in the cluster may participate in a parallel query across instances with it. It tells the current instance "who else can help it" with a parallel query.
2) depends on how you set up the instance groups, as you already know, it isn't service driven.
3) see #2
Restore RMAN Disk backups of RAC Database to Single Instance On Another Node
asktom fan, May 11, 2012 - 8:56 am UTC
Hi Tom,
In metalink document 415579.1 (How To Restore RMAN Disk backups of RAC Database to Single Instance On Another Node), step 7 is to determine the recovery point:
<quote>
Check the last archive sequence for all redo threads and select the archive sequence having LEAST "Next SCN" among them. In our case sequence 58 of thread 1 has Next SCN of 233107 while sequence 3 of thread 2 has Next SCN of 233110. Since squence 58 of thread 1 has least Next SCN we will recover upto this point. (If you are keen to have recovery run until some specific time you can always give SET UNTIL TIME)
</quote>
My question is: why can't we select the archive sequence having GREATEST "Next SCN" among them?
Thanks!
May 11, 2012 - 11:49 am UTC
because given the set of archives you have - that is the last SCN you know for a fact you have all of the redo for. Each instance had their own redo threads and some may have archived way before the others.