Skip to main content

What happened to the "future" institutional data center

Anonymously contributed:

In the past couple of years, one of LLNL's achievements was the build out of an
enterprise-class data center in B112 under the O& B PAD.
Fully redundant power, industry standard everything!
Only one side of it has been fully populated. When the time came to to populate the other side, ULM decided to do it on the cheap by ordering the undoing of the power redundancy from the populated side to accommodate the unpopulated side.

This is to allow more tenants to move in with little cost.

The result is that we will have a grade C data center.

Imagine telling doing an A+ job and being told that C+ would have been OK!

That is what happens when bureaucrats (instead of Managers) make decisions.

Comments

Anonymous said…
There's no money to build out the north part of B112 to the same level/standards as the south side...so the next best thing is to divert power. Having all of the south side on dual power was overkill...far outside any industry standard or norm, and far above and beyond what any LLNL data center had. So diverting some south power to north makes solid fiscal sense. There's truly no money to shake loose from this tree.
Anonymous said…
Overkill? The datacenter was based on
industry standards of major co-location and App hosting providers.

As a center of excellence, LLN will accept no less.
The only mistake was that
the south side was populated sparsely and that did not fool anyone in ULM!
Anonymous said…
Follow the leader; his name is Terascale Facility!
Anonymous said…
Let us face it: OPS & BUS management just follow orders! they dont question. If I were the person in charge of the data center, I will report the power diversion to DOE as a waste of resources and manpower.
It took resources to get the datacenter where it is today and it will take resources to divert power
from south to north side!
Anonymous said…
The Data center should have been based on the requirements of prospective customers. While it is a beautiful facility it is also poorly suited to many of the groups who would in theory need to move their equipment in. Which explains in part why it has been very slow to fill.

Talk to your customer, document and verify their needs, create solutions that meet those needs.

In private industry if you execute well you get to regularly return to step one.
Anonymous said…
Overkill is the appropriate word. DOE should be advised of the waste of resources that went into the original infrastructure costs that have been severely underutilized and mismanaged for the past several years of the data center operation.
Anonymous said…
Yet another fine Upper Level Management decision. They are really working hard to demonstrate their superior intellect on this one. Look at them go!!!
Anonymous said…
Upper management never asked a state of the art data center. Low level management had vision that were not proportionate with funding. The management responsible for the overkill were too greedy. Yeah! Bring in servers and rack them sparsely on one side of the data center and go ask for more money to build the other side.
I am glad upper management saw that.
Anonymous said…
The Laboratory is about doing great science and making sure that the support functions are managed efficiently. This is an example of not keeping the mission at the core of what matters. Let's focus on what is really important. Without great science, the Laboratory does not exist. This is a great message to all support functions. Find ways to be efficient and reduce unneccessary costs.
Anonymous said…
"This is a great message to all support functions. Find ways to be efficient and reduce unneccessary costs." - 8:53 PM


Are you kidding me? Support orgs and management help create greater job security and growth of their particular areas by more spending and greater inefficiency.

Those costs are then burdened onto the remaining scientific staff as larger tax and overhead rates for which they must seek additional funding. It's almost like a Ponzi scheme for those still trying to do science at the NNSA labs. Who will be the last fool left within the scientific ranks to help pay for this mess?
Anonymous said…
The people who pushed for a state of the art data center live in an ideal world! They built it for the ideal institution and with the assumption that funding will always be available.
To those people, I say: don't forget you are still "overhead" and as such
you better find cheaper ways to do things.
If you were the Terascale facility, that would be another story!
Anonymous said…
The idea of dual power is not overkill. This is especially true in the new era of not having hot work permit capability to work on panels. Unfortunately some of the client equipment did not have dual power supplies and thus the dual capability was useless.

I have seen a commercial data center that had redundant power, redundant cooling, motor generators that could keep the center running in case of full power outages. It was a beautiful setup. It was a phone data center. It wasn't to keep the phones working, it was to keep the billing system going.

That was probably the dream/goal of what was going to be put into place for B112. They fell far short of that goal.

For those that argue that such redundancy is not needed, remember that your capability to fill out your time card and GET PAID rests on the viability of that data center.

And for those who do have equipment housed in B112, chime in and tell the stories of how easy it is to get in there and work on your equipment. Especially on the off hours.
Anonymous said…
Yes, it is hard to work on your equipment. You must be out by 3PM.
The DC has to improve its relations with tenants and making decisions to benefit its tenants.
After all, the tenants are customers and in the private sector what happens when tenants are not happy?
Anonymous said…
Here is what will be done!!! If LLNL doesn’t do it then someone else will do it for us and we all lose our jobs.

First of all leave the power configuration alone. You don’t build data centers without power redundancy, it is one of the most important pieces of the foundation for which you will build upon. You certainly don’t change your power strategy especially after it has already been implemented. Its cost prohibitive and it’s a bad move. In addition I would imagine too many things are implemented by now to make that sort of change.

Virtualize physical hardware wherever possible! Rack all systems into the fewest amounts of racks and rows with respect to power, size, weight, and cooling capability. Go dense, go lean and fill out the south side. Demonstrate a well thought out solution with real ROI and then propose building out the north side of the data center.

One of the many goals amongst any modern day data center is to achieve the maximum amount of compute resource in the smallest amount of space possible.

Results:
$ Eliminate costly hardware
$ Preserve datacenter real estate for systems that cannot be virtualized
$ Save on power cost and infrastructure
$ Save on cooling cost and infrastructure
$ Do more with less
$ Keep everyone employed
$ Lets get real scientific about it too so DOE can lead by example and set standards based on LLNL’s success
$ ULM will be the victim of success and DOE will praise them for it. :-)

Lastly there are too many people in areas who are afraid to let go of their precious hardware. They cannot handle change and believe that they will lose their jobs if they can’t hold onto their closet data centers!!! We better change, work together and get it right...and real soon!!
Anonymous said…
"Lastly there are too many people in areas who are afraid to let go of their precious hardware."

Lord help you if you move your equipment into B112. Although you are a tenant, the actions of how they run the place gives the impression you can't be trusted. They had cutbacks and are not staffed on swing or owl. You can't get in on the off hours without calling them in (and incurring extra charges for doing so). I'd be concerned about my precious hardware if it were the prisoner of B112.
Anonymous said…
"Lord help you if you move your equipment into B112."

Your comment definitely holds merit, however if we don’t get it together and make some changes then at some point we all lose out. Things cannot continue the way they are for too much longer.

I too feel the trust issues and I struggle for access to the DC, it just needs to CHANGE as I mentioned. We all need to CHANGE!
Anonymous said…
We all need to CHANGE!

March 11, 2010 9:54 PM

From what, to what? Can you be more specific?
Anonymous said…
"We all need to change"

Show me the improved model and I'd consider the change.

Don't come with the "If we build it, they will come" mentality.

B112 just announced extended hours of coverage, from 7:00 am to 4:45 pm.

It was also just announced that all of the hardware signed a Memorandum of Understanding (MOU) stating that they will not fail except during the coverage hours. That should allay all fears for 24x7 systems.

Popular posts from this blog

Plutonium Shots on NIF.

Tri-Valley Cares needs to be on this if they aren't already. We need to make sure that NNSA and LLNL does not make good on promises to pursue such stupid ideas as doing Plutonium experiments on NIF. The stupidity arises from the fact that a huge population is placed at risk in the short and long term. Why do this kind of experiment in a heavily populated area? Only a moron would push that kind of imbecile area. Do it somewhere else in the god forsaken hills of Los Alamos. Why should the communities in the Bay Area be subjected to such increased risk just because the lab's NIF has failed twice and is trying the Hail Mary pass of doing an SNM experiment just to justify their existence? Those Laser EoS techniques and the people analyzing the raw data are all just BAD anyways. You know what comes next after they do the experiment. They'll figure out that they need larger samples. More risk for the local population. Stop this imbecilic pursuit. They wan...

Trump is to gut the labs.

The budget has a 20% decrease to DOE office of science, 20% cut to NIH. NASA also gets a cut. This will  have a huge negative effect on the lab. Crazy, juts crazy. He also wants to cut NEA and PBS, this may not seem like  a big deal but they get very little money and do great things.

LLNL un-diversity

Actual post from Dec. 15 from one of the streams. This is a real topic. As far as promoting women and minorities even if their qualifications are not as good as the white male scientists, I am all for it. We need diversity at the lab and if that is what it takes, so be it.  Quit your whining. Look around the lab, what do you see? White male geezers. How many African Americans do you see at the lab? Virtually none. LLNL is one of the MOST undiverse places you will see. Face it folks, LLNL is an institution of white male privilege and they don't want to give up their privileged positions. California, a state of majority Hispanics has the "crown jewel" LLNL nestled in the middle of it with very FEW Hispanics at all!