StorageMojo keeps producing interesting material. This time it’s about FAB (Federated Array of Bricks). I wonder if this will make the same hype as RAC?
RAC - simple, cheap, effective, application transparent way to assemble rock solid highly available and scalable database environment out of cheap boxes (read bricks).
FAB principles quoted from page 1 (bold marks are mine):
A Federated Array of Bricks (FAB) is a logical
disk system that provides the reliability and performance
of enterprise-class disk arrays, at a fraction of the
cost and with better scalability.
…
A Federated Array of Bricks (FAB) is a low-cost alternative
to disk arrays, and is designed to be scalable from
very small to very large systems. FAB achieves this by
composing together storage bricks, where each brick is
a small rack-mounted storage appliance built from commodity
components including disks, a CPU, NVRAM, and
network cards. FAB systems cost much less than disk arrays
to manufacture and develop, due to the economies
of scale inherent in volume production, and because FAB
can replace entire array product lines (amortizing development
costs). Because of these factors, we anticipate that
a FAB system can be built for far less than the equivalent
high-end system. FAB provides comparable reliability,
achieved through replication: we store the same disk
block on multiple bricks, and we create redundant paths
between all components of the system. FAB performance
scales by completely distributing all functionality (no centralized
bottlenecks) across the set of available bricks.
I’m afraid the real life sounds more like:
- RAC — if application is able to partition workload well than it will “go RAC” well or with a bit of efforts can be distributed well without RAC.
- FAB — if application (database) can partition storage well than it would be able to “go FAB” nicely as well as without it.
Cost saving analogy - compare cost of a brick house vs. price of bricks. It turned out that monolith or prebuilt houses are much cheaper when technology reached certain level.
Dear Reader, what is your perspective?
Interesting results from Google.
The first one references 42. Well, nothing surprising.
Next comes UKOUG event “Applications Unlimited Tour”. Unfortunately, it’s not really my area. I will have to change my profile it seems. 
Oracle-L comes third. Again no surprise here.
Later we can see Doug’s flying Guinness story (not that you believe it, right?) as well as its replicas through feed aggregators.
So I find those results somewhat poor. I will have to make sure that more relevant contents is delivered.
Last weekend we went to the Canadian Museum of Civilization. We liked very much the Canada Hall. The Grand Hall and the First People Hall were not that interesting mainly because Alex Jr. was pretty much bored there. Obviously, he enjoyed more the Children’s Museum area.
Very impressive was IMAX theater where we watched the Deep Sea movie. It was really cool on the big round screen. Highly recommended!
I wrapped up some photos here. If flash is the problem than here.
REPORT SCHEMA is quite handy to get growth rate of your database/datafile/tablespace providing RMAN repository is used. I don’t remember what triggered that thought but I see it in my “to blog” list so here it goes.
Running several times REPORT SCHEMA AT TIME 'SYSDATE-n'; and processing the results will provide nice trend with granularity up to datafile. It can be processed and aggregated to tablespace or the whole database.
I don’t see that historical datafile information is stored in a repository so RMAN, probably, calculates or implies it from data about backups taken (BDF table?). That means that timeline precision depends in the backups frequency. I’ve run REPORT SCHEMA AT TIME with DEBUG ON quickly but it shows only calls to RMAN internal package and I didn’t have enough time to trace the queries.
In my UKOUG 06 presentation on block change tracking internals I assumed that “change tracking state change latch” is, probably, used by DBWR and CTWR to protect access to a buffer area in shared pool. I wanted to verify it and tried to trace this latch.
It turned out that it has nothing to do with what I thought. So far I found only few cases when “change tracking state change latch” is acquired:
- during instance startup (when opened) — CTWR process starts and takes the latch once.
- during instance shutdown — CTWR stops and takes the latch once.
- during enabling and disabling block change tracking while database is open — session’s shadow process acquires the latch twice and CTWR process takes it once when it starts or stops.
So it seems that this latch is used very rarely. Perhaps, there are other cases which I didn’t figure out but as I observed in several databases, the number of gets for this latch is 1 if block change tracking is left enabled from instance startup.
This leaves the open question — how does CTWR determine which blocks have been changed?
Kevin recently mentioned one very nice blog. I was going through some posts there and this entry reminded me one story. I’m sure many of you can recall similar cases.
I worked on one site for a while and during 2.5 years it didn’t face a single media corruption of Oracle datafiles. Not that it’s a low profile site - quite the opposite and storage infrastructure was setup very well there - no SPOF, mirrored inside SAN boxes and between boxes, redundant switches, HBAs, controllers, you name it - dream of a DBA. Even change management procedures were followed thoroughly.
But one day, my fellow DBA (who is usually extremely cautious and reviews his actions at least twice) overwrote a controlfile with some crap. Even the fact that controlfiles were on raw devices didn’t prevent this disaster from happening. Trivial error as we found out later - a DBA mistakenly swapped arguments of a tar command (like “tar cvf * file.tar” instead of “tar cvf file.tar *”) and tar happily used controlfile as a tape device. :) End result - 10 minutes outage while I was figuring out what happened, dd’ing controlfile image from another mirror and starting the instance. By the way, it was a RAC database and, of course, RAC didn’t help - surprisingly for some managers.
So they were kind of protected with multiplexed controlfiles even though recovery wasn’t transparent (wouldn’t it be nice if Oracle could survive loss of minority of multiplexed controlfiles - just like CRS with voting disks?). Interesting, that online redo logs were not multiplexed and recovery could have been a bit trickier should the current redo log be overwritten. The reason for that was that they had already quadruple mirroring and people were blindly ignoring human factor and Mr. Murphy - “it must be enough if we already mirrored it 4 times”.
What we see? Well implemented protection against one class of problems while ignoring obvious threats from another side. Perhaps, because of all kind of vendors making fuss about their technology and its importance, while nobody focusing attention on the areas that require low investments but as much important or even more.
In my experience human factor risk is one areas that is heavily underestimated most of the times.
So what are your stories?
Nice post on David Weiss’ Blog few days ago.
I believe that a good DBA should have well developed “peripheral vision”. It helps everywhere — troubleshooting production issues or performance problems, database/application design, system architecture, reading documentation, etc. I also notice that a newbie DBA with good peripheral vision catches up much faster than a strictly focused person.
On the other hand, in many situations a DBA should be able to narrow her/his focus on specific issues according to current priorities. For example, while investigating a certain performance problem, it might be challenging to keep track of the target. I have seen many situations that began with a single user complaint, and were “resolved” by, say, creating an index on the table that has nothing to do with the original complaint, only because the DBA got sidetracked from his or her target.
Or maybe it’s just something to do with keeping your priorities straight?
First of all, there is now FREE Solaris 10 media kit available to order by mail if you’re not fancy downloading several GB. By the way, it also includes Sun Studio (anyone using it?). I ordered mine and it should come soon.
Second, I found a cool article on Solaris BigAdmin portal - Deployment Guide for an Open Source Stack on the Solaris 10 OS also available as PDF. It shows how to install MySQL and PostgreSQL as well as Apache, Tomcat and PHP, and integrate them all together. Really simple guide so anyone can do it. Couple interesting bits I found there are tips on performance tuning (unfortunately, not much explained) and how to configure Solaris Zones (but it’s more for Apache installation I believe).
It will take me a while to follow up on all the presentations I’ve visited during UKOUG06 and photos I’ve taken. One of the best presentations was Inside RAC from Julian Dyke. You can find it here if you don’t have access to UKOUG06 archive.
However, the fun part you won’t get unless you’ve been on his presentation. It turned out that toilet paper is an integral part of a RAC cluster. Just go to give you an impression (more pictures here):
The performance with participation of few volunteer was great. Cones on the heads indicated the role of the nodes for a certain block during a particular operation. Square label with an SCN number of it is an image of the block. Toilet paper marked the path of the block.
Very quickly the scene became a mess of toilet paper which wasn’t of very high quality. This leads to a real life hint - make your interconnect rock solid and reliable.
Another issue is that some “nodes” were busy quite with several activities and blocks/cones/toilet paper ends were falling out of their hands. This is another lesson - make sure your nodes have enough CPU capacity and not totally occupied with something else (read short on CPU).
All in all it was a great performance. Artists?
Attendees of UKOUG 2006 can now download presentations from the agenda.
I received a few emails from people who don’t have access to the UKOUG 2006 files, asking whether my presentations are available online. They are available and should appear on the Pythian Papers page shortly.
Feel free to contact me with any questions/feedback.
Recent Comments