Fixing the National Weather Service's Computer Gap

To contact us Click HERE
In previous blogs, I have documented the profound inadequacy of the computational resources used for operational numerical weather prediction by the National Weather Service (NWS) and the serious implications this deficiency has for the quality of weather forecasts in the U.S.   I have described how the world-leading European Center for Medium Range Weather Forecasting (ECMWF) now has more than ten times the computer power as the U.S. Environmental Modeling Center (EMC), how U.S. skill in global prediction is in second or third place, and how the lack of computer resources is crippling the NWS's ability to move forward in probabilistic prediction, the next major area of development.
(Reminder:  EMC, part of the NWS, is the operational weather prediction entity of the U.S.)

I have talked to many people about my blogs and assessment, including meteorologists, both inside and outside the NWS, and highly placed managers and administrators in the NWS:  there is essentially no disagreement that we have a serious problem in numerical weather prediction, and that lack of computer power is a major cause but not the only one.

The new NOAA Fairmont Computer Center hss far more capability than EMC's computer center

It is time to fix the NWS's operational computer deficiency and this blog will describe how it can be done within a year using funds that are already appropriated.  But it will take leadership and a willingness to do things a different way.  And an end to highly disfunctional relationships in NOAA and the NWS.  This is going to be a very frank assessment of the current situation and will get somewhat technical in places...so please forgive me or skip this blog if you find it tedious.

The Problem is Worse Than I Thought

When U.S. Senator Maria Cantwell learned about the lack of computer power for U.S.  numerical weather prediction at a luncheon I attended, she asked an important question of the head of the NWS:  how can this be when Congress has appropriated large amounts of funds for weather and climate computers?  He did not answer, but the answer is clear: nearly all of these resources have been unavailable for weather prediction--most are used for climate studies.

But the problem is deeper and more disturbing than that... other groups in NOAA are securing bigger computers than the national operational center, EMC.   And some of these groups are actively working to acquire computer resources for themselves rather than EMC.   A good case is the NOAA Earth System Research Lab (ESRL) in  NOAA's Office of Atmospheric Research.  This lab is tasked with doing research to support operational numerical weather prediction (NWP) in the NWS, even though they are not in the NWS.  As noted in an earlier blog, this is a crazy organizational situation, with those running operational NWP in the U.S. unable to control the research that supports them.  ESRL has been able to find funds for very large supercomputers (the "jet" machines) that far eclipse what EMC has to work with.  ESRL has established essentially operational capabilities and wants to expand it further (they called it Regular Research).  Amazingly, two high administrators in OAR/ESRL told me that I should not be working to secure big computers for EMC but rather should get it for THEM!  I was really taken aback by their attitude.  And recently the Hurricane Forecast Improvement Program (HFIP) received  a large computer resource (placed at ESRL) and HFIP is using them for operational global and hurricane-scale simulations.

NOAA ESRL in Boulder houses supercomputers more capable than those used by the U.S.'s main numerical weather prediction facility

So we have the nutty situation in which operational NWP is starved for computer resources, undermining progress in weather prediction, while climate studies have massive supercomputers available and NOAA fosters active competitors in its organization that are doing essentially operational weather prediction with far greater resources than EMC, the U.S. operational center.   This screams about poor leadership and management in NOAA.

The other problem is that the NWS is wasting a substantial amount of the limited computer power it does have today.  The graphic below shows how the NWS is using their current computer.  A lot of it does not make sense.  Time is on the X axis (entire day) and the Y axis is the number of nodes (a node is a collection of processors) used.  The various colors represent different models or simulations run on this computer.

The red color on the lower portion, the largest use of the computer, is for the Climate Forecast System, in which they run seasonal forecasts.  But they run these forecasts FOUR TIMES A DAY, which makes no sense.  Why run a seasonal simulation that often?  In contrast, running the global model (the GFS), shown by the dirty green color, only takes a small part of the computer.  Furthermore, they run the GFS out FOUR TIMES A DAY to 384 h--why do they do that?  Most other big centers only find it is useful to run out twice a day.  I could find no objective proof in the literature or elsewhere why such frequent runs could be useful.   I can go into more detail, but the bottom line is that the use of EMC's computer is inefficient and not well thought out.  A lot is done for legacy reasons.  A rational evaluation of cost and benefit would clearly change allocations substantially.   But even if they used the current small computers rationally they don't have enough to do what needs to be done.

Production Schedule for EMC's Computer

What do they really need?

      For EMC to serve the nation in a reasonable way, I believe they need the computational resources to do the following:

(1) Run a global ensemble system at 12-15 km resolution (currently they are at roughly 50 km). (Remember ensembles is when you run a model many times with different starting points and model physics, this allows one to get at the uncertainties in a forecast).  This ensemble needs to be running the best physics possible, unlike the inferior physics used in the current U.S. global ensemble system.
(2)  Run convection resolving high-resolution ensembles over the U.S. (1-4 km resolution).  Currently, the U.S. ensemble system is at 16 km resolution.  Many of the runs of the current use inferior physics to save computer time.
(3)  Run a rapid-update system (like ESRL's HRRR) at 3 km resolution.   Eventually, (2) an (3) should be combined.
(4)  Lowest priority but useful.  Run a global model at 2-4 km resolution.

Doubling resolution takes about 8 times the computer power.    My back of the envelope calculation is that the above is doable if EMC had 5-10 petaflops of computer power (well within the range of recently acquired machines by others).  The plan below will give it to EMC for operational use and maintain high reliability.

How to Fix the Problem Quickly

First, EMC needs to get their house in order and reduce the waste in their current schedule, which I estimate is roughly 25% of their current computer.

EMC will get an upgrade this summer of their two .07 petaflop machines (the vendor is  IBM, one operational and one backup) to .2 petaflops.  This is helpful, but not nearly enough.  Congress just passed the Hurricane Sandy relief bill for roughly 50 billion dollars.   Within this bill is 25 million for enhanced hurricane weather prediction and data assimilation and 50 million for hurricane research...money that is going to NOAA.  One thing we learned is that good global weather prediction is the key for hurricane forecasting--that is why the European Center Global Model was the best during Sandy. So you want to help hurricane forecasting?  USE ALL OF THE 25 MILLION TO UPGRADE EMC's COMPUTER RESOURCES.

The German weather service just purchased a 23 million dollar CRAY supercomputer that dwarfs what the U.S. NWS now uses.

Use the 25 million in Sandy money to acquire (EMC likes to lease) ONE big machine, a computer with 1-3 petaflops or more.  My discussions with several computer vendors suggests that the NWS might be surprised about how much they could get for 25 million.  Perhaps as high as 5-10 petaflops if they play their cards right.  I believe this machine could possess at least 99% reliability and folks in the NWS computer hierarchy agree.  (Hell...I have a cluster I use for weather forecasting that maintains such reliability and I do it on a shoestring, surely they can as well!).  The recently acquired NOAA Fairmont machine can serve as backup for the new EMC computer, as well as being available for development and research. 

Thus, the operational load can be split between the current IBM system, which will increase in size again in 2015 to roughly one petaflop, and the new system purchased with Sandy money.  Using these new resources wisely, the NWS operational can jump to world leadership capability in numerical weather prediction and radically improve the products it provides to U.S. users.

Additional Fixes

There is little doubt EMC could quickly take advantage of the increased computer resources (I have confirmed this by talking with their leadership).  However, as noted earlier, the problems in U.S. numerical weather prediction are deeper than lack of supercomputers (although fixing that deficiency would be a good start).  Management and leadership failures have abounded.  To address these problems, immediate attention should be given to the following items:

1. Establish a numerical weather prediction advisory board for EMC that provides recommendations from experts in the entire community.  A big part of the problem is that the National Weather Service folks have been too isolated from the rest of the meteorological community.  They serve the nation but have generally been unenthusiastic and getting guidance and advice from their users, the private sector, and the research community.  This has led NWS EMC to second/third tier status and must change.  For years, U.S. National Academy committees and others have recommended that EMC establish a representative advisory committee that would act as an active partner.  NWS management has pushed back on this and have done nothing.  Enough is enough....this advisory committee should be established immediately and should serve as a sounding board for deciding on which models are run, how they are run, the computer resources, needed and more.

Japan's weather supercomputer (peak .85 petaflops) is roughly ten times larger then U.S. EMC.

2.  Restructure NWP research and development in NOAA/NWS.  The current separation of  weather prediction research from operations has been a continuing disaster and must end.  NOAA leadership finally must deal with this mess.   Moving EMC into NOAA and combining with OAR/ESRL under one manager might work.  Or move ESRL folks into the NWS under EMC. 

3.  Establish a comprehensive verification program for U.S. models.  To improve weather forecasting models you must know their strengths and weaknesses.   The NWS/EMC model verification program is very weak and superficial.  If you want to see how bad things are, check their very poor model verification web pages.  Ask a simple question:  how well the model's verify over the NW?  Better over the mountains or lowlands?  Or how has forecast skill over California changed during the past few decades? You will be disappointed I guarantee you. A lot of the statistics are monthly, making it impossible to determine the trends in model skill.

NOAA money supports the Developmental Testbed Center, which I know quite a bit about (I have been chair of their Science Advisory Board).  The dream was that folks could provide new research innovations that would be tested in an operational-like environment for a wide range of cases.  If successful, they would go into operations.  Sounds good?  After nearly a decade and millions of dollars, this is a dream that never seems to happen.  The DTC should take the testbed role seriously.  Now.

Taiwan's weather bureau has a computer twice as fast as EMCs and has purchased one over 15 times as powerful.

4.  Support a model improvement research program.   The U.S. has the largest meteorological research community in the world, with universities like the U.S. doing cutting edge research on numerical weather prediction and related topics.  NOAA/NWS have failed to take advantage of this huge community, maintaining a miniscule extramural research program.   Any new research funds goes right into NOAA coffers.  This must change.  Let's start with the 50 million in Sandy research money and use most of it for extramural, university-based research. NWS/NOAA extramural weather model research should be targeted to the most acute needs of the National Weather Service modeling efforts.  Trust me, money speaks in the research community.

5.  Create a strategic plan with community input and do it.   Currently, there is NO comprehensive and detailed strategic plan by the National Weather Service on the improvement of numerical weather prediction.   This contrasts with foreign meteorological services (such as our neighbors, the Canadians), who have laid out detailed and aggressive roadmaps of their future direction.   You can't go far without a map.  The NWS needs one and the community should be at the table when it is constructed.

5.  Provide decent documentation of what U.S. modeling centers are doing.

  Want to figure out the details of the models run by the U.S. ?  Good luck.  It is pretty much impossible to do so by going to EMC or its parent NCEP's web sites.  Scanty, out-of-date material is all you will find.  Amusingly, what you WILL find is their response to "certain blogs."  You can't imagine whose.

Let me be blunt: the state of operational U.S. numerical weather prediction is an embarrassment to the nation and it does not have to be this way.  Taiwan, Germany, England, the European Center, Canada, and other nations have more computer power for their weather prediction services.  Our nation has had inferior numerical weather prediction for too long.  New computers are an obvious and relatively easy first step, because they make everything possible.  For the price of a single warplane we could have greatly improved weather prediction that would save lives and property.   Congress and the American public should not accept delays in action.  If this issue was placed before a real leader like President Lincoln, asking him when we should act, I can imagine what he would say (click on the arrow at the bottom of the  picture to find out):

Second-Rate U.S. Numerical Weather Prediction: Why You Should Care

To contact us Click HERE
In several of my previous blogs I noted that U.S. numerical weather prediction islagging behind the European Center and others--a diagnosis pretty much universally accepted in my field. I listed some of the reasons: inferior computers, poor management, lack of effective leadership,inability to tap the large U.S. weather research community, and others.

But some highly placed folks in the National Weather Service(NWS) and elsewhere have argued that U.S. inferiority in numerical weatherprediction really doesn’t matter,since U.S. government forecasters have access to the superior forecasts of themodels of the European Center (EC), the UK Met Office, and others.   As a prime example, they noted that EC modelforecasts played a major role in NWS warnings in the week before HurricaneSandy made landfall.  Some European Center officials have said the same thing--the current situation is fine!  Weather prediction is a global entity, so why be concerned if they are ahead?
In this blog, I will take issue with these arguments andwill suggest that first-rate numerical weather prediction by the U.S. NationalWeather Service is crucial for the nation and of great benefit to the entireworld.  That it is one of thecost-effective investments our nation could make.

Let me make a few points.
Point 1:  U.S. has the potential to be far superior tothe European Center and others.
Forget the defeatist rhetoric.  The U.S. has by far the largestmeteorological research establishment in the world.  We spend more money on weather research thanany other nation.   Why should anyone believe that the best we can do is tofollow or equal the EC?  As anumerical modeler myself I firmly believe that our global model could far exceed theperformance of the rather conservative EC effort.   To go technical on you for a second, I believethat using ensemble-based data assimilation (in space and time), far bettermodel physics, higher resolution, and better use of observational assets, wecould produce vastly improvedforecasts, far superior to the EC, with huge economic and safety benefits tothe nation.
To put it another way. Think of the U.S. and the European Center weather prediction efforts as two cars.  One (the U.S.)  has a much bigger engine (knowledge andresearch base) than the other.   But thebig-engine car has a very inefficient transmission (and other deficits) and ends up going slower thanthe small-engined competition (EC).  Yes, wecan hitch a ride with Europeans, but our car could leave them in the dust, ifonly we had the will to do so.The U.S. governmental weather prediction effort looks like this...a very big research engine, without the ability to use U.S. massive research power to move our weather prediction forward rapidly.  The guy in the red pants is a NOAA bureaucrat.
Point 2:  U.S. national and regional prediction is ashadow of where it should be.
A lot of the discussion in this blog and elsewhere has been about the inferiority of the U.S.global model (the GFS) to global models of others (EC and UKMET).   But as important as the global models are,they are only part of the story.  The lack of computer power and poor coordination between research and operational weather communities in the U.S. has crippled our ability to move forward towards the high-resolution weather prediction capability that we know represents the future:  probabilistic prediction.  And remember the EC only does global modeling--they are not interested in high-resolution prediction over the U.S. 

It is clear that the future of weather prediction will be to forecast probabilistically for all parameters, with the essential infrastructure to do this being high-resolution ensembles (meaning we will run our forecast models MANY times --say 25-100 times--using different model starting points and model physics).  A number of National Academy study groups have recommended this approach (I have been a member of several of them!) and noted that such ensembles must be high enough resolution (2-4 km grid spacing) to  resolve convection (thunderstorms).   You want to predict major convective outbreaks, like the UNFORECAST derecho (strong convective system with powerful winds) that hit the northeast U.S. last June?  You need this capability.  But the NWS Environmental Modeling Center (EMC) does not have this critical capability because they don't have the computer resources, among other reasons.

In fact, they don't have the computer power to run even current generation weather technology.  For example, the NOAA Earth System Research Lab (ESRL) has developed a new high-resolution prediction system called HRRR (High Resolution Rapid Refresh) that was able to predict the powerful derecho hours before (see graphic).

In contrast, the NWS EMC model (NAM) failed (see graphic below).  The NWS EMC does not have the computer power today to run HRRR operationally.

The NWS NAM model missed this important event 12h before

Point 3:  U.S. modeling inferiority is costing the U.S. private sector big bucks and denying real-time access to the U.S. research community.

U.S. business and governmental interests, such as weather prediction firms and utilities, need the best forecasts, with even modest differences in skill having big financial implications.  Thus, a number of companies and U.S. entities are paying hundreds of the thousands of dollars EACH to get European Center model output.  Yes, we are talking about millions of dollars that is being used to support the EC modeling effort.  The Europeans have a different financial model than the U.S. National Weather Service, one patterned after ancient empires:  they require financial tribute from those wishing the best meteorological "protection The U.S., to its credit, provides model output for free, which not only fosters commerce, but assists nations and researchers all over the world.  But what we give away is clearly not as good as EC's global model, thus allowing the rest of the world to help support their modeling efforts.

Even the U.S. research community is required to pay for real-time access to EC grids.  A long time ago I was able to get EC grids because I was working with an EC researcher.  The EC bureaucrats caught this infraction, cut off the grids, and offered me a "deep discount" rate of $50, 000 a year!  My research using the superior EC grids (which required real-time access) was over.  And my situation is repeated many times over with the rest of the U.S. weather research community.

Epilogue

This weekend I went to a talk by a Stanford political scientist who noted that great nations generally don't fail from external threats--rather they weaken from within.  They get lazy, inefficient, lose their edge, and start making bad decisions.   U.S. operational numerical weather prediction is a prime example of a nation resting on it laurels and falling behind.  We invented numerical weather prediction.  Most scientific and technical advances  in weather prediction have occurred here and STILL DO,   Our research community is still largest and dominant. And with all of that we have lost leadership and have fallen well back into the pack.  And the cost is not just in prestige, but also a weakening of  our nation's economic prowess and needlessly jeopardizing life and limb.

As explained in my previous blogs the route to fixing U.S. numerical weather prediction is clear, including:

(1) Secure sufficient computer power, either by redirecting some of the huge computer resources acquired for climate prediction or using new funds (like money already appropriated in the Hurricane Sandy relief bill)

(2)  Integrate numerical weather prediction research and operations, including replacing the ineffective division between the NWS EMC and the NOAA labs.  NOAA must dedicate enough extramural research funds to entrain the U.S. research community.

(3) The need for effective leadership in NOAA, including a clear vision where weather prediction is going in the U.S.  

There is a real cost to inferior U.S. numerical weather prediction and I am confident the numbers are easily in the billions of dollars.  We have let a crucial piece of the technological infrastructure of our nation to weaken and atrophy.  We must take a new course.

Hike of the Week- Coplay Lake

To contact us Click HERE

Coplay Lake @ Dennis Long

Updated Tuesday 2-26-2013. This is on the edge of Mount Rainier Park the Carbon River side. This is the summer toad to Summmit Lake. Road is not gated so might have just a few trucks.
Check out Green Trails Maps for directions. 237 is the map for this hike. Click here for complete list of maps. 



Road walk @ Dennis Long

Open mountain views on the way up @ Michael Fagin

Road goes up to Summit Lake too far to do in winter  Michael Faign

At the edge of the lake @ Dennis Long

Driving Directions; From Enumclaw drive west on State Route 410 (Chinook Pass Highway) to SR 165. Proceed on SR 165 to the Carbon River Road/Mowich Lake Highway junction. Turn left onto Carbon River Road and follow it to Cayada Creek Road (Forest Road 7810), which is  just before the national park entrance. Turn left (north) and drive about 4 miles which is close  to the snow line about 2800 feet

Trail Stats: To lake is about 2.5 miles one way and 1100 feet gain. Depends where you park

Fixing the National Weather Service's Computer Gap

To contact us Click HERE
In previous blogs, I have documented the profound inadequacy of the computational resources used for operational numerical weather prediction by the National Weather Service (NWS) and the serious implications this deficiency has for the quality of weather forecasts in the U.S.   I have described how the world-leading European Center for Medium Range Weather Forecasting (ECMWF) now has more than ten times the computer power as the U.S. Environmental Modeling Center (EMC), how U.S. skill in global prediction is in second or third place, and how the lack of computer resources is crippling the NWS's ability to move forward in probabilistic prediction, the next major area of development.
(Reminder:  EMC, part of the NWS, is the operational weather prediction entity of the U.S.)

I have talked to many people about my blogs and assessment, including meteorologists, both inside and outside the NWS, and highly placed managers and administrators in the NWS:  there is essentially no disagreement that we have a serious problem in numerical weather prediction, and that lack of computer power is a major cause but not the only one.

The new NOAA Fairmont Computer Center hss far more capability than EMC's computer center

It is time to fix the NWS's operational computer deficiency and this blog will describe how it can be done within a year using funds that are already appropriated.  But it will take leadership and a willingness to do things a different way.  And an end to highly disfunctional relationships in NOAA and the NWS.  This is going to be a very frank assessment of the current situation and will get somewhat technical in places...so please forgive me or skip this blog if you find it tedious.

The Problem is Worse Than I Thought

When U.S. Senator Maria Cantwell learned about the lack of computer power for U.S.  numerical weather prediction at a luncheon I attended, she asked an important question of the head of the NWS:  how can this be when Congress has appropriated large amounts of funds for weather and climate computers?  He did not answer, but the answer is clear: nearly all of these resources have been unavailable for weather prediction--most are used for climate studies.

But the problem is deeper and more disturbing than that... other groups in NOAA are securing bigger computers than the national operational center, EMC.   And some of these groups are actively working to acquire computer resources for themselves rather than EMC.   A good case is the NOAA Earth System Research Lab (ESRL) in  NOAA's Office of Atmospheric Research.  This lab is tasked with doing research to support operational numerical weather prediction (NWP) in the NWS, even though they are not in the NWS.  As noted in an earlier blog, this is a crazy organizational situation, with those running operational NWP in the U.S. unable to control the research that supports them.  ESRL has been able to find funds for very large supercomputers (the "jet" machines) that far eclipse what EMC has to work with.  ESRL has established essentially operational capabilities and wants to expand it further (they called it Regular Research).  Amazingly, two high administrators in OAR/ESRL told me that I should not be working to secure big computers for EMC but rather should get it for THEM!  I was really taken aback by their attitude.  And recently the Hurricane Forecast Improvement Program (HFIP) received  a large computer resource (placed at ESRL) and HFIP is using them for operational global and hurricane-scale simulations.

NOAA ESRL in Boulder houses supercomputers more capable than those used by the U.S.'s main numerical weather prediction facility

So we have the nutty situation in which operational NWP is starved for computer resources, undermining progress in weather prediction, while climate studies have massive supercomputers available and NOAA fosters active competitors in its organization that are doing essentially operational weather prediction with far greater resources than EMC, the U.S. operational center.   This screams about poor leadership and management in NOAA.

The other problem is that the NWS is wasting a substantial amount of the limited computer power it does have today.  The graphic below shows how the NWS is using their current computer.  A lot of it does not make sense.  Time is on the X axis (entire day) and the Y axis is the number of nodes (a node is a collection of processors) used.  The various colors represent different models or simulations run on this computer.

The red color on the lower portion, the largest use of the computer, is for the Climate Forecast System, in which they run seasonal forecasts.  But they run these forecasts FOUR TIMES A DAY, which makes no sense.  Why run a seasonal simulation that often?  In contrast, running the global model (the GFS), shown by the dirty green color, only takes a small part of the computer.  Furthermore, they run the GFS out FOUR TIMES A DAY to 384 h--why do they do that?  Most other big centers only find it is useful to run out twice a day.  I could find no objective proof in the literature or elsewhere why such frequent runs could be useful.   I can go into more detail, but the bottom line is that the use of EMC's computer is inefficient and not well thought out.  A lot is done for legacy reasons.  A rational evaluation of cost and benefit would clearly change allocations substantially.   But even if they used the current small computers rationally they don't have enough to do what needs to be done.

Production Schedule for EMC's Computer

What do they really need?

      For EMC to serve the nation in a reasonable way, I believe they need the computational resources to do the following:

(1) Run a global ensemble system at 12-15 km resolution (currently they are at roughly 50 km). (Remember ensembles is when you run a model many times with different starting points and model physics, this allows one to get at the uncertainties in a forecast).  This ensemble needs to be running the best physics possible, unlike the inferior physics used in the current U.S. global ensemble system.
(2)  Run convection resolving high-resolution ensembles over the U.S. (1-4 km resolution).  Currently, the U.S. ensemble system is at 16 km resolution.  Many of the runs of the current use inferior physics to save computer time.
(3)  Run a rapid-update system (like ESRL's HRRR) at 3 km resolution.   Eventually, (2) an (3) should be combined.
(4)  Lowest priority but useful.  Run a global model at 2-4 km resolution.

Doubling resolution takes about 8 times the computer power.    My back of the envelope calculation is that the above is doable if EMC had 5-10 petaflops of computer power (well within the range of recently acquired machines by others).  The plan below will give it to EMC for operational use and maintain high reliability.

How to Fix the Problem Quickly

First, EMC needs to get their house in order and reduce the waste in their current schedule, which I estimate is roughly 25% of their current computer.

EMC will get an upgrade this summer of their two .07 petaflop machines (the vendor is  IBM, one operational and one backup) to .2 petaflops.  This is helpful, but not nearly enough.  Congress just passed the Hurricane Sandy relief bill for roughly 50 billion dollars.   Within this bill is 25 million for enhanced hurricane weather prediction and data assimilation and 50 million for hurricane research...money that is going to NOAA.  One thing we learned is that good global weather prediction is the key for hurricane forecasting--that is why the European Center Global Model was the best during Sandy. So you want to help hurricane forecasting?  USE ALL OF THE 25 MILLION TO UPGRADE EMC's COMPUTER RESOURCES.

The German weather service just purchased a 23 million dollar CRAY supercomputer that dwarfs what the U.S. NWS now uses.

Use the 25 million in Sandy money to acquire (EMC likes to lease) ONE big machine, a computer with 1-3 petaflops or more.  My discussions with several computer vendors suggests that the NWS might be surprised about how much they could get for 25 million.  Perhaps as high as 5-10 petaflops if they play their cards right.  I believe this machine could possess at least 99% reliability and folks in the NWS computer hierarchy agree.  (Hell...I have a cluster I use for weather forecasting that maintains such reliability and I do it on a shoestring, surely they can as well!).  The recently acquired NOAA Fairmont machine can serve as backup for the new EMC computer, as well as being available for development and research. 

Thus, the operational load can be split between the current IBM system, which will increase in size again in 2015 to roughly one petaflop, and the new system purchased with Sandy money.  Using these new resources wisely, the NWS operational can jump to world leadership capability in numerical weather prediction and radically improve the products it provides to U.S. users.

Additional Fixes

There is little doubt EMC could quickly take advantage of the increased computer resources (I have confirmed this by talking with their leadership).  However, as noted earlier, the problems in U.S. numerical weather prediction are deeper than lack of supercomputers (although fixing that deficiency would be a good start).  Management and leadership failures have abounded.  To address these problems, immediate attention should be given to the following items:

1. Establish a numerical weather prediction advisory board for EMC that provides recommendations from experts in the entire community.  A big part of the problem is that the National Weather Service folks have been too isolated from the rest of the meteorological community.  They serve the nation but have generally been unenthusiastic and getting guidance and advice from their users, the private sector, and the research community.  This has led NWS EMC to second/third tier status and must change.  For years, U.S. National Academy committees and others have recommended that EMC establish a representative advisory committee that would act as an active partner.  NWS management has pushed back on this and have done nothing.  Enough is enough....this advisory committee should be established immediately and should serve as a sounding board for deciding on which models are run, how they are run, the computer resources, needed and more.

Japan's weather supercomputer (peak .85 petaflops) is roughly ten times larger then U.S. EMC.

2.  Restructure NWP research and development in NOAA/NWS.  The current separation of  weather prediction research from operations has been a continuing disaster and must end.  NOAA leadership finally must deal with this mess.   Moving EMC into NOAA and combining with OAR/ESRL under one manager might work.  Or move ESRL folks into the NWS under EMC. 

3.  Establish a comprehensive verification program for U.S. models.  To improve weather forecasting models you must know their strengths and weaknesses.   The NWS/EMC model verification program is very weak and superficial.  If you want to see how bad things are, check their very poor model verification web pages.  Ask a simple question:  how well the model's verify over the NW?  Better over the mountains or lowlands?  Or how has forecast skill over California changed during the past few decades? You will be disappointed I guarantee you. A lot of the statistics are monthly, making it impossible to determine the trends in model skill.

NOAA money supports the Developmental Testbed Center, which I know quite a bit about (I have been chair of their Science Advisory Board).  The dream was that folks could provide new research innovations that would be tested in an operational-like environment for a wide range of cases.  If successful, they would go into operations.  Sounds good?  After nearly a decade and millions of dollars, this is a dream that never seems to happen.  The DTC should take the testbed role seriously.  Now.

Taiwan's weather bureau has a computer twice as fast as EMCs and has purchased one over 15 times as powerful.

4.  Support a model improvement research program.   The U.S. has the largest meteorological research community in the world, with universities like the U.S. doing cutting edge research on numerical weather prediction and related topics.  NOAA/NWS have failed to take advantage of this huge community, maintaining a miniscule extramural research program.   Any new research funds goes right into NOAA coffers.  This must change.  Let's start with the 50 million in Sandy research money and use most of it for extramural, university-based research. NWS/NOAA extramural weather model research should be targeted to the most acute needs of the National Weather Service modeling efforts.  Trust me, money speaks in the research community.

5.  Create a strategic plan with community input and do it.   Currently, there is NO comprehensive and detailed strategic plan by the National Weather Service on the improvement of numerical weather prediction.   This contrasts with foreign meteorological services (such as our neighbors, the Canadians), who have laid out detailed and aggressive roadmaps of their future direction.   You can't go far without a map.  The NWS needs one and the community should be at the table when it is constructed.

5.  Provide decent documentation of what U.S. modeling centers are doing.

  Want to figure out the details of the models run by the U.S. ?  Good luck.  It is pretty much impossible to do so by going to EMC or its parent NCEP's web sites.  Scanty, out-of-date material is all you will find.  Amusingly, what you WILL find is their response to "certain blogs."  You can't imagine whose.

Let me be blunt: the state of operational U.S. numerical weather prediction is an embarrassment to the nation and it does not have to be this way.  Taiwan, Germany, England, the European Center, Canada, and other nations have more computer power for their weather prediction services.  Our nation has had inferior numerical weather prediction for too long.  New computers are an obvious and relatively easy first step, because they make everything possible.  For the price of a single warplane we could have greatly improved weather prediction that would save lives and property.   Congress and the American public should not accept delays in action.  If this issue was placed before a real leader like President Lincoln, asking him when we should act, I can imagine what he would say (click on the arrow at the bottom of the  picture to find out):

Fixing the National Weather Service's Computer Gap

To contact us Click HERE
In previous blogs, I have documented the profound inadequacy of the computational resources used for operational numerical weather prediction by the National Weather Service (NWS) and the serious implications this deficiency has for the quality of weather forecasts in the U.S.   I have described how the world-leading European Center for Medium Range Weather Forecasting (ECMWF) now has more than ten times the computer power as the U.S. Environmental Modeling Center (EMC), how U.S. skill in global prediction is in second or third place, and how the lack of computer resources is crippling the NWS's ability to move forward in probabilistic prediction, the next major area of development.
(Reminder:  EMC, part of the NWS, is the operational weather prediction entity of the U.S.)

I have talked to many people about my blogs and assessment, including meteorologists, both inside and outside the NWS, and highly placed managers and administrators in the NWS:  there is essentially no disagreement that we have a serious problem in numerical weather prediction, and that lack of computer power is a major cause but not the only one.

The new NOAA Fairmont Computer Center hss far more capability than EMC's computer center

It is time to fix the NWS's operational computer deficiency and this blog will describe how it can be done within a year using funds that are already appropriated.  But it will take leadership and a willingness to do things a different way.  And an end to highly disfunctional relationships in NOAA and the NWS.  This is going to be a very frank assessment of the current situation and will get somewhat technical in places...so please forgive me or skip this blog if you find it tedious.

The Problem is Worse Than I Thought

When U.S. Senator Maria Cantwell learned about the lack of computer power for U.S.  numerical weather prediction at a luncheon I attended, she asked an important question of the head of the NWS:  how can this be when Congress has appropriated large amounts of funds for weather and climate computers?  He did not answer, but the answer is clear: nearly all of these resources have been unavailable for weather prediction--most are used for climate studies.

But the problem is deeper and more disturbing than that... other groups in NOAA are securing bigger computers than the national operational center, EMC.   And some of these groups are actively working to acquire computer resources for themselves rather than EMC.   A good case is the NOAA Earth System Research Lab (ESRL) in  NOAA's Office of Atmospheric Research.  This lab is tasked with doing research to support operational numerical weather prediction (NWP) in the NWS, even though they are not in the NWS.  As noted in an earlier blog, this is a crazy organizational situation, with those running operational NWP in the U.S. unable to control the research that supports them.  ESRL has been able to find funds for very large supercomputers (the "jet" machines) that far eclipse what EMC has to work with.  ESRL has established essentially operational capabilities and wants to expand it further (they called it Regular Research).  Amazingly, two high administrators in OAR/ESRL told me that I should not be working to secure big computers for EMC but rather should get it for THEM!  I was really taken aback by their attitude.  And recently the Hurricane Forecast Improvement Program (HFIP) received  a large computer resource (placed at ESRL) and HFIP is using them for operational global and hurricane-scale simulations.

NOAA ESRL in Boulder houses supercomputers more capable than those used by the U.S.'s main numerical weather prediction facility

So we have the nutty situation in which operational NWP is starved for computer resources, undermining progress in weather prediction, while climate studies have massive supercomputers available and NOAA fosters active competitors in its organization that are doing essentially operational weather prediction with far greater resources than EMC, the U.S. operational center.   This screams about poor leadership and management in NOAA.

The other problem is that the NWS is wasting a substantial amount of the limited computer power it does have today.  The graphic below shows how the NWS is using their current computer.  A lot of it does not make sense.  Time is on the X axis (entire day) and the Y axis is the number of nodes (a node is a collection of processors) used.  The various colors represent different models or simulations run on this computer.

The red color on the lower portion, the largest use of the computer, is for the Climate Forecast System, in which they run seasonal forecasts.  But they run these forecasts FOUR TIMES A DAY, which makes no sense.  Why run a seasonal simulation that often?  In contrast, running the global model (the GFS), shown by the dirty green color, only takes a small part of the computer.  Furthermore, they run the GFS out FOUR TIMES A DAY to 384 h--why do they do that?  Most other big centers only find it is useful to run out twice a day.  I could find no objective proof in the literature or elsewhere why such frequent runs could be useful.   I can go into more detail, but the bottom line is that the use of EMC's computer is inefficient and not well thought out.  A lot is done for legacy reasons.  A rational evaluation of cost and benefit would clearly change allocations substantially.   But even if they used the current small computers rationally they don't have enough to do what needs to be done.

Production Schedule for EMC's Computer

What do they really need?

      For EMC to serve the nation in a reasonable way, I believe they need the computational resources to do the following:

(1) Run a global ensemble system at 12-15 km resolution (currently they are at roughly 50 km). (Remember ensembles is when you run a model many times with different starting points and model physics, this allows one to get at the uncertainties in a forecast).  This ensemble needs to be running the best physics possible, unlike the inferior physics used in the current U.S. global ensemble system.
(2)  Run convection resolving high-resolution ensembles over the U.S. (1-4 km resolution).  Currently, the U.S. ensemble system is at 16 km resolution.  Many of the runs of the current use inferior physics to save computer time.
(3)  Run a rapid-update system (like ESRL's HRRR) at 3 km resolution.   Eventually, (2) an (3) should be combined.
(4)  Lowest priority but useful.  Run a global model at 2-4 km resolution.

Doubling resolution takes about 8 times the computer power.    My back of the envelope calculation is that the above is doable if EMC had 5-10 petaflops of computer power (well within the range of recently acquired machines by others).  The plan below will give it to EMC for operational use and maintain high reliability.

How to Fix the Problem Quickly

First, EMC needs to get their house in order and reduce the waste in their current schedule, which I estimate is roughly 25% of their current computer.

EMC will get an upgrade this summer of their two .07 petaflop machines (the vendor is  IBM, one operational and one backup) to .2 petaflops.  This is helpful, but not nearly enough.  Congress just passed the Hurricane Sandy relief bill for roughly 50 billion dollars.   Within this bill is 25 million for enhanced hurricane weather prediction and data assimilation and 50 million for hurricane research...money that is going to NOAA.  One thing we learned is that good global weather prediction is the key for hurricane forecasting--that is why the European Center Global Model was the best during Sandy. So you want to help hurricane forecasting?  USE ALL OF THE 25 MILLION TO UPGRADE EMC's COMPUTER RESOURCES.

The German weather service just purchased a 23 million dollar CRAY supercomputer that dwarfs what the U.S. NWS now uses.

Use the 25 million in Sandy money to acquire (EMC likes to lease) ONE big machine, a computer with 1-3 petaflops or more.  My discussions with several computer vendors suggests that the NWS might be surprised about how much they could get for 25 million.  Perhaps as high as 5-10 petaflops if they play their cards right.  I believe this machine could possess at least 99% reliability and folks in the NWS computer hierarchy agree.  (Hell...I have a cluster I use for weather forecasting that maintains such reliability and I do it on a shoestring, surely they can as well!).  The recently acquired NOAA Fairmont machine can serve as backup for the new EMC computer, as well as being available for development and research. 

Thus, the operational load can be split between the current IBM system, which will increase in size again in 2015 to roughly one petaflop, and the new system purchased with Sandy money.  Using these new resources wisely, the NWS operational can jump to world leadership capability in numerical weather prediction and radically improve the products it provides to U.S. users.

Additional Fixes

There is little doubt EMC could quickly take advantage of the increased computer resources (I have confirmed this by talking with their leadership).  However, as noted earlier, the problems in U.S. numerical weather prediction are deeper than lack of supercomputers (although fixing that deficiency would be a good start).  Management and leadership failures have abounded.  To address these problems, immediate attention should be given to the following items:

1. Establish a numerical weather prediction advisory board for EMC that provides recommendations from experts in the entire community.  A big part of the problem is that the National Weather Service folks have been too isolated from the rest of the meteorological community.  They serve the nation but have generally been unenthusiastic and getting guidance and advice from their users, the private sector, and the research community.  This has led NWS EMC to second/third tier status and must change.  For years, U.S. National Academy committees and others have recommended that EMC establish a representative advisory committee that would act as an active partner.  NWS management has pushed back on this and have done nothing.  Enough is enough....this advisory committee should be established immediately and should serve as a sounding board for deciding on which models are run, how they are run, the computer resources, needed and more.

Japan's weather supercomputer (peak .85 petaflops) is roughly ten times larger then U.S. EMC.

2.  Restructure NWP research and development in NOAA/NWS.  The current separation of  weather prediction research from operations has been a continuing disaster and must end.  NOAA leadership finally must deal with this mess.   Moving EMC into NOAA and combining with OAR/ESRL under one manager might work.  Or move ESRL folks into the NWS under EMC. 

3.  Establish a comprehensive verification program for U.S. models.  To improve weather forecasting models you must know their strengths and weaknesses.   The NWS/EMC model verification program is very weak and superficial.  If you want to see how bad things are, check their very poor model verification web pages.  Ask a simple question:  how well the model's verify over the NW?  Better over the mountains or lowlands?  Or how has forecast skill over California changed during the past few decades? You will be disappointed I guarantee you. A lot of the statistics are monthly, making it impossible to determine the trends in model skill.

NOAA money supports the Developmental Testbed Center, which I know quite a bit about (I have been chair of their Science Advisory Board).  The dream was that folks could provide new research innovations that would be tested in an operational-like environment for a wide range of cases.  If successful, they would go into operations.  Sounds good?  After nearly a decade and millions of dollars, this is a dream that never seems to happen.  The DTC should take the testbed role seriously.  Now.

Taiwan's weather bureau has a computer twice as fast as EMCs and has purchased one over 15 times as powerful.

4.  Support a model improvement research program.   The U.S. has the largest meteorological research community in the world, with universities like the U.S. doing cutting edge research on numerical weather prediction and related topics.  NOAA/NWS have failed to take advantage of this huge community, maintaining a miniscule extramural research program.   Any new research funds goes right into NOAA coffers.  This must change.  Let's start with the 50 million in Sandy research money and use most of it for extramural, university-based research. NWS/NOAA extramural weather model research should be targeted to the most acute needs of the National Weather Service modeling efforts.  Trust me, money speaks in the research community.

5.  Create a strategic plan with community input and do it.   Currently, there is NO comprehensive and detailed strategic plan by the National Weather Service on the improvement of numerical weather prediction.   This contrasts with foreign meteorological services (such as our neighbors, the Canadians), who have laid out detailed and aggressive roadmaps of their future direction.   You can't go far without a map.  The NWS needs one and the community should be at the table when it is constructed.

5.  Provide decent documentation of what U.S. modeling centers are doing.

  Want to figure out the details of the models run by the U.S. ?  Good luck.  It is pretty much impossible to do so by going to EMC or its parent NCEP's web sites.  Scanty, out-of-date material is all you will find.  Amusingly, what you WILL find is their response to "certain blogs."  You can't imagine whose.

Let me be blunt: the state of operational U.S. numerical weather prediction is an embarrassment to the nation and it does not have to be this way.  Taiwan, Germany, England, the European Center, Canada, and other nations have more computer power for their weather prediction services.  Our nation has had inferior numerical weather prediction for too long.  New computers are an obvious and relatively easy first step, because they make everything possible.  For the price of a single warplane we could have greatly improved weather prediction that would save lives and property.   Congress and the American public should not accept delays in action.  If this issue was placed before a real leader like President Lincoln, asking him when we should act, I can imagine what he would say (click on the arrow at the bottom of the  picture to find out):

Radar, Wind, and Snow

To contact us Click HERE
The front has gone by, the winds have lessened, and heavy snow is falling on the western slopes of the Cascades.    But first, let me answer a frequently asked question:  what is the persistent echo seen just offshore of Hoquiam in many images from the Langley Hill radar? (see below).   This pattern is seen persistently in the lower radar scans and looks like a half circle.    The answer: the lower portion of the radar beam is hitting the surface and reflecting back to the radar, which is located a few miles northwest of Hoquiam.   Such a phenomenon is known as ocean or sea clutter.   The image below is from the half-degree elevation angle scan of the Langley radar, which means the center of the beam is a half-degree above the horizontal.  The width of the beam is roughly 1 degree, so the lower portion of the beam skims the surface in the vicinity of the radar.
Some of you might ask, doesn't the radar also hit the ground and terrain?  Why don't we see a return there as well?  The answer is that you would except for the fact the radar has ground clutter suppression software that subtracts that out.   Land and mountains stay put and so the radar can be trained to ignore returns from them. Not so easy with an ocean surface with waves and swell.  We knew we would get some sea clutter with the new radar, but it is a small price to pay for the capability to see farther out into the Pacific.  In fact, due to the intercession of Senator Cantwell, our radar has the capability to scan lower than any other NWS radar in the country:  .15 degrees.  Yes, more ground clutter, but more offshore range.

Let's compare the radar imagery at .15, .5, and 1.5 degree elevation angles this morning (7:02 AM).  First .15 degrees....lots of sea clutter, but you can see the shallow convective showers way offshore.



.5 degrees--the range pulls in a bit.

1.5 degrees:  no sea clutter, but the horizontal range is much less.

With taller targets, the .15 degree elevation angle can see 300 km offshore or more.

This radar image also shows weak convective showers over the ocean in the cold, unstable air and substantial enhancement of the precipitation as the air is forced to rise by Olympics and coastal mountains.

Finally, the winds.   The vigorous cold front and the strong northwesterlies than followed produced some strong winds over the region--but nothing truly damaging or exciting.  Here are the max winds (mph)during the past 24 hr.  Gusts above 50 mph in and downwind of the Strait and some 40s over the Sound.  Notice how quickly the winds weakened over land.


 And yes, snow.  This kind of cool, unstable, northwest flow pattern is great for snow:  expect snow totals of 1-2 feet above 3500 feet.  The mountains needed some fresh snow.   Skiers will be happy.

Reminder:  if any of you want to attend the Northwest Weather Workshop on March 1-2 (next weekend) in Seattle, you can get more information and register here.

Fixing the National Weather Service's Computer Gap

To contact us Click HERE
In previous blogs, I have documented the profound inadequacy of the computational resources used for operational numerical weather prediction by the National Weather Service (NWS) and the serious implications this deficiency has for the quality of weather forecasts in the U.S.   I have described how the world-leading European Center for Medium Range Weather Forecasting (ECMWF) now has more than ten times the computer power as the U.S. Environmental Modeling Center (EMC), how U.S. skill in global prediction is in second or third place, and how the lack of computer resources is crippling the NWS's ability to move forward in probabilistic prediction, the next major area of development.
(Reminder:  EMC, part of the NWS, is the operational weather prediction entity of the U.S.)

I have talked to many people about my blogs and assessment, including meteorologists, both inside and outside the NWS, and highly placed managers and administrators in the NWS:  there is essentially no disagreement that we have a serious problem in numerical weather prediction, and that lack of computer power is a major cause but not the only one.

The new NOAA Fairmont Computer Center hss far more capability than EMC's computer center

It is time to fix the NWS's operational computer deficiency and this blog will describe how it can be done within a year using funds that are already appropriated.  But it will take leadership and a willingness to do things a different way.  And an end to highly disfunctional relationships in NOAA and the NWS.  This is going to be a very frank assessment of the current situation and will get somewhat technical in places...so please forgive me or skip this blog if you find it tedious.

The Problem is Worse Than I Thought

When U.S. Senator Maria Cantwell learned about the lack of computer power for U.S.  numerical weather prediction at a luncheon I attended, she asked an important question of the head of the NWS:  how can this be when Congress has appropriated large amounts of funds for weather and climate computers?  He did not answer, but the answer is clear: nearly all of these resources have been unavailable for weather prediction--most are used for climate studies.

But the problem is deeper and more disturbing than that... other groups in NOAA are securing bigger computers than the national operational center, EMC.   And some of these groups are actively working to acquire computer resources for themselves rather than EMC.   A good case is the NOAA Earth System Research Lab (ESRL) in  NOAA's Office of Atmospheric Research.  This lab is tasked with doing research to support operational numerical weather prediction (NWP) in the NWS, even though they are not in the NWS.  As noted in an earlier blog, this is a crazy organizational situation, with those running operational NWP in the U.S. unable to control the research that supports them.  ESRL has been able to find funds for very large supercomputers (the "jet" machines) that far eclipse what EMC has to work with.  ESRL has established essentially operational capabilities and wants to expand it further (they called it Regular Research).  Amazingly, two high administrators in OAR/ESRL told me that I should not be working to secure big computers for EMC but rather should get it for THEM!  I was really taken aback by their attitude.  And recently the Hurricane Forecast Improvement Program (HFIP) received  a large computer resource (placed at ESRL) and HFIP is using them for operational global and hurricane-scale simulations.

NOAA ESRL in Boulder houses supercomputers more capable than those used by the U.S.'s main numerical weather prediction facility

So we have the nutty situation in which operational NWP is starved for computer resources, undermining progress in weather prediction, while climate studies have massive supercomputers available and NOAA fosters active competitors in its organization that are doing essentially operational weather prediction with far greater resources than EMC, the U.S. operational center.   This screams about poor leadership and management in NOAA.

The other problem is that the NWS is wasting a substantial amount of the limited computer power it does have today.  The graphic below shows how the NWS is using their current computer.  A lot of it does not make sense.  Time is on the X axis (entire day) and the Y axis is the number of nodes (a node is a collection of processors) used.  The various colors represent different models or simulations run on this computer.

The red color on the lower portion, the largest use of the computer, is for the Climate Forecast System, in which they run seasonal forecasts.  But they run these forecasts FOUR TIMES A DAY, which makes no sense.  Why run a seasonal simulation that often?  In contrast, running the global model (the GFS), shown by the dirty green color, only takes a small part of the computer.  Furthermore, they run the GFS out FOUR TIMES A DAY to 384 h--why do they do that?  Most other big centers only find it is useful to run out twice a day.  I could find no objective proof in the literature or elsewhere why such frequent runs could be useful.   I can go into more detail, but the bottom line is that the use of EMC's computer is inefficient and not well thought out.  A lot is done for legacy reasons.  A rational evaluation of cost and benefit would clearly change allocations substantially.   But even if they used the current small computers rationally they don't have enough to do what needs to be done.

Production Schedule for EMC's Computer

What do they really need?

      For EMC to serve the nation in a reasonable way, I believe they need the computational resources to do the following:

(1) Run a global ensemble system at 12-15 km resolution (currently they are at roughly 50 km). (Remember ensembles is when you run a model many times with different starting points and model physics, this allows one to get at the uncertainties in a forecast).  This ensemble needs to be running the best physics possible, unlike the inferior physics used in the current U.S. global ensemble system.
(2)  Run convection resolving high-resolution ensembles over the U.S. (1-4 km resolution).  Currently, the U.S. ensemble system is at 16 km resolution.  Many of the runs of the current use inferior physics to save computer time.
(3)  Run a rapid-update system (like ESRL's HRRR) at 3 km resolution.   Eventually, (2) an (3) should be combined.
(4)  Lowest priority but useful.  Run a global model at 2-4 km resolution.

Doubling resolution takes about 8 times the computer power.    My back of the envelope calculation is that the above is doable if EMC had 5-10 petaflops of computer power (well within the range of recently acquired machines by others).  The plan below will give it to EMC for operational use and maintain high reliability.

How to Fix the Problem Quickly

First, EMC needs to get their house in order and reduce the waste in their current schedule, which I estimate is roughly 25% of their current computer.

EMC will get an upgrade this summer of their two .07 petaflop machines (the vendor is  IBM, one operational and one backup) to .2 petaflops.  This is helpful, but not nearly enough.  Congress just passed the Hurricane Sandy relief bill for roughly 50 billion dollars.   Within this bill is 25 million for enhanced hurricane weather prediction and data assimilation and 50 million for hurricane research...money that is going to NOAA.  One thing we learned is that good global weather prediction is the key for hurricane forecasting--that is why the European Center Global Model was the best during Sandy. So you want to help hurricane forecasting?  USE ALL OF THE 25 MILLION TO UPGRADE EMC's COMPUTER RESOURCES.

The German weather service just purchased a 23 million dollar CRAY supercomputer that dwarfs what the U.S. NWS now uses.

Use the 25 million in Sandy money to acquire (EMC likes to lease) ONE big machine, a computer with 1-3 petaflops or more.  My discussions with several computer vendors suggests that the NWS might be surprised about how much they could get for 25 million.  Perhaps as high as 5-10 petaflops if they play their cards right.  I believe this machine could possess at least 99% reliability and folks in the NWS computer hierarchy agree.  (Hell...I have a cluster I use for weather forecasting that maintains such reliability and I do it on a shoestring, surely they can as well!).  The recently acquired NOAA Fairmont machine can serve as backup for the new EMC computer, as well as being available for development and research. 

Thus, the operational load can be split between the current IBM system, which will increase in size again in 2015 to roughly one petaflop, and the new system purchased with Sandy money.  Using these new resources wisely, the NWS operational can jump to world leadership capability in numerical weather prediction and radically improve the products it provides to U.S. users.

Additional Fixes

There is little doubt EMC could quickly take advantage of the increased computer resources (I have confirmed this by talking with their leadership).  However, as noted earlier, the problems in U.S. numerical weather prediction are deeper than lack of supercomputers (although fixing that deficiency would be a good start).  Management and leadership failures have abounded.  To address these problems, immediate attention should be given to the following items:

1. Establish a numerical weather prediction advisory board for EMC that provides recommendations from experts in the entire community.  A big part of the problem is that the National Weather Service folks have been too isolated from the rest of the meteorological community.  They serve the nation but have generally been unenthusiastic and getting guidance and advice from their users, the private sector, and the research community.  This has led NWS EMC to second/third tier status and must change.  For years, U.S. National Academy committees and others have recommended that EMC establish a representative advisory committee that would act as an active partner.  NWS management has pushed back on this and have done nothing.  Enough is enough....this advisory committee should be established immediately and should serve as a sounding board for deciding on which models are run, how they are run, the computer resources, needed and more.

Japan's weather supercomputer (peak .85 petaflops) is roughly ten times larger then U.S. EMC.

2.  Restructure NWP research and development in NOAA/NWS.  The current separation of  weather prediction research from operations has been a continuing disaster and must end.  NOAA leadership finally must deal with this mess.   Moving EMC into NOAA and combining with OAR/ESRL under one manager might work.  Or move ESRL folks into the NWS under EMC. 

3.  Establish a comprehensive verification program for U.S. models.  To improve weather forecasting models you must know their strengths and weaknesses.   The NWS/EMC model verification program is very weak and superficial.  If you want to see how bad things are, check their very poor model verification web pages.  Ask a simple question:  how well the model's verify over the NW?  Better over the mountains or lowlands?  Or how has forecast skill over California changed during the past few decades? You will be disappointed I guarantee you. A lot of the statistics are monthly, making it impossible to determine the trends in model skill.

NOAA money supports the Developmental Testbed Center, which I know quite a bit about (I have been chair of their Science Advisory Board).  The dream was that folks could provide new research innovations that would be tested in an operational-like environment for a wide range of cases.  If successful, they would go into operations.  Sounds good?  After nearly a decade and millions of dollars, this is a dream that never seems to happen.  The DTC should take the testbed role seriously.  Now.

Taiwan's weather bureau has a computer twice as fast as EMCs and has purchased one over 15 times as powerful.

4.  Support a model improvement research program.   The U.S. has the largest meteorological research community in the world, with universities like the U.S. doing cutting edge research on numerical weather prediction and related topics.  NOAA/NWS have failed to take advantage of this huge community, maintaining a miniscule extramural research program.   Any new research funds goes right into NOAA coffers.  This must change.  Let's start with the 50 million in Sandy research money and use most of it for extramural, university-based research. NWS/NOAA extramural weather model research should be targeted to the most acute needs of the National Weather Service modeling efforts.  Trust me, money speaks in the research community.

5.  Create a strategic plan with community input and do it.   Currently, there is NO comprehensive and detailed strategic plan by the National Weather Service on the improvement of numerical weather prediction.   This contrasts with foreign meteorological services (such as our neighbors, the Canadians), who have laid out detailed and aggressive roadmaps of their future direction.   You can't go far without a map.  The NWS needs one and the community should be at the table when it is constructed.

5.  Provide decent documentation of what U.S. modeling centers are doing.

  Want to figure out the details of the models run by the U.S. ?  Good luck.  It is pretty much impossible to do so by going to EMC or its parent NCEP's web sites.  Scanty, out-of-date material is all you will find.  Amusingly, what you WILL find is their response to "certain blogs."  You can't imagine whose.

Let me be blunt: the state of operational U.S. numerical weather prediction is an embarrassment to the nation and it does not have to be this way.  Taiwan, Germany, England, the European Center, Canada, and other nations have more computer power for their weather prediction services.  Our nation has had inferior numerical weather prediction for too long.  New computers are an obvious and relatively easy first step, because they make everything possible.  For the price of a single warplane we could have greatly improved weather prediction that would save lives and property.   Congress and the American public should not accept delays in action.  If this issue was placed before a real leader like President Lincoln, asking him when we should act, I can imagine what he would say (click on the arrow at the bottom of the  picture to find out):