New Release Of GHCN Code

Download link

I have made quite a few updates to my GHCN code, which is primarily targeted at the US HCN database – though it will work with any GHCN data set. The release includes seven files, compressed into file ghcn.tgz

getdaily  GHCN.cpp  GHCN.h  ghcnd-stations.txt  Main.cpp  Makefile  stations.txt

getdaily is a tcsh script for downloading and organizing the latest HCN daily temperature database from NOAA. It takes about an hour to run over a very fast link. It generates 49 .txt files – one for the US and one for each of Obama’s 57 states.

The code is C++ and consists of three files : Main.cpp GHCN.cpp  GHCN.h. It should  compile without warnings on g++ or Visual Studio compilers. The Makefile works on any gnu system, like Linux. On Cygwin, the suffix “.exe” gets added. To use Visual Studio, you will have to create a project and import the three code files.

There are a lot of new options

ScreenHunter_171 Apr. 13 09.03

stations.txt must be in the same directory as the executable. Typical usage would be

./ghcn.exe NY.txt > NY.csv

That reads all NY records and outputs the results to a NY.csv file

The rest comes in the category of “either you can figure it out now, or you need two years of education to figure it out

My critics will enjoy looking through this and seeing that (unlike them) I am not doing anything to bias the output data one way or another.  That is because (unlike them) I am an actual scientist – and an honest person.

About stevengoddard

Just having fun
This entry was posted in Uncategorized. Bookmark the permalink.

52 Responses to New Release Of GHCN Code

  1. Eric Barnes says:

    Thanks!

  2. Password protected says:

    I’ll bite.
    Obama’s 57 states…?
    I hope Canada isn’t one of them.

  3. R2Dtoo says:

    And I thank you for all you do.

  4. bubbagyro says:

    It does not matter how right you are, if your are a heretic. Like the witches of bygone, either you drown or float when thrown in the pond.

  5. Owen says:

    The Climate Liars will never reveal their coding, data or anything else. They are sociopathic scum pretending to do science in order to inflict their sick, twisted political agenda on the world. These people are very dangerous. They are lunatics.

  6. I can read everything fine in Visual Studio 2008, and it says build successful, but I don’t see what happens after that. I have the 2 .cpp and 1 .h visible, and I tell it to build it. Any tips on how I can turn your code into an executable, like a list of commands I need to do from VS? Could you go ahead and make the .exe and post it? (I would prefer to build myself in case I get smart enough to edit your code).

    I have a linux laptop I could use after that (it won’t run Windows anymore, hardware problem).

    I don’t know C++ (obviously), but a nice visual interface for this would be fun to work on. If it was easy enough, the world would be doing a whole lot more auditing and asking some pretty difficult questions.

    Is there a way to make your program work from a DOS prompt, even just to pull in the daily data (run everything on a windows machine?)

    I tried doing something like the getdaily in excel, and it would bomb after a few hours, but would successfully read and output the text files for many stations before it died (I was only using it as a VBA interface, not storing any data. I had to do some text translation too so Windows could work with it). It was totally unpredictable how many hours it would run before excel would crash. Longest I got was about 7 hours and it was maybe 1/3 of the way done. So it was really slow too, even on a 50 mbs connection..

    I’d really like to get this to work if you can give a few steps to a C++ rookie. It might help others too if you had a step by step. This is very important work so the more participants the better.

    • On Linux, all you have to do is cd into the directory and type “make”
      The executable will be named “ghcn”
      You can run that from from the bash prompt. For Texas it would be

      Download the files
      ./getdaily
      Run the executable
      ./ghcn TX.txt > Tx.csv

    • The Makefile contains the magic, I guess. Just manage to find a system where:
      > c++ -o ghcn Main.cpp GHCN.cpp
      makes sense.

      On FreeBSD I simply run
      > clang++ -O3 -march=k8-sse3 -fomit-frame-pointer -ffast-math -fno-strict-aliasing -o ghcn_clang Main.cpp GHCN.cpp

        • -g?
          ls -l ./ghcn*
          -rwxr-xr-x 1 bark fart 179208 Apr 14 22:51 ghcn_clang3.5.bin
          -rwxr-xr-x 1 cat meat 1421832 Apr 14 23:03 ghcn_clang3.5.bin-g

          That’s an order of magnitude to handle a bunch of unusable debugging symbols. I dunno about the speed, but I’m sure it’s not _faster_.

        • g++ is just the GNU c++ compiler. clang++ works (I just tried it) & “c++” is the canonical name for whatever c++ compiler you happen to have on your system (& which here is linked to clang++, not g++):

          sha256 `which c++ clang++`
          SHA256 (/usr/bin/c++) = 78560db4cf51747ea9fb2de7ef591e9bc1a9b195e91d28c8352034a7c967c7ea
          SHA256 (/usr/bin/clang++) = 78560db4cf51747ea9fb2de7ef591e9bc1a9b195e91d28c8352034a7c967c7ea

          QED 😉

      • The makefile says g++, not c++
        Your original comment made no sense

        “Just manage to find a system where:
        > c++ -o ghcn Main.cpp GHCN.cpp
        makes sense.”

    • the -o argument is the name of the output file (so, if you need spaces in the name, you would put “-o Harry\ Harryhausen\ something” because the blackslash allows a space to go in the filename.

      Parenthetically, spaces are delimiters, so don’t use them.

      So, here’s what it is:

      c++ is the compiler
      -o whatever is the file that it creates
      -f & -m & -O arguments are compiler specific & create a faster, or smaller executable.
      & all the stuff at the end is the .cpp files you are trying to compile.
      There are easier ways to do it, but it works.

  7. -=NikFromNYC=- says:

    Epic. The script kiddies of social media marketing have *only* mere marketing under their belt and lack all substance. But they are mere men made into girls by college LIBERAL Arts requirements and mandatory visits to the college Psyco-The-Rapist (psychotherapist) if they ever own their own balls. It used to be, guys balanced out girls, nation-wide. Then Society actually offered not only non-land owners the democratic (with a big D always) vote, but also hormonally insane *girls*! They immediately instituted Prohibition which by ironic effect created gangsterism and 3000 speakeasys in Washington, D.C. alone, and politics has never been again the same game of socialist workers party Democratic KKK party members against simple hands-off libertarians who enjoyed also the vice of hellfire Religion. Now they, on the Progresive Left, and the retrograde Right want to put the Internet viral Tea Party back in the box. But as late R.I.P. climate skeptical progressive lefty icon Alexander “Hot Cock” Cockburn exasperated in a video interview: “they have all the guns.”

    Donald Trump 2016.

    -=NYC=-

    P.S. Judith Curry isn’t a passive aggressive bitch anymore, she’s Joan of Arc. And Lord Mockton isn’t a mere puzzle game bumbling old folk’s home retrograde emotional throwback, but a competent real world scientist. These recent changes are also epic.

    • That doesn’t look like C++. What language was your post written in?

      • Brian G Valentine says:

        I think you need to take DOM or Mescaline to decipher it but I am too busy right now to experiment with that hypothesis

      • -=NikFromNYC=- says:

        It’s called English, bad ass flyboy.

        • Brian G Valentine says:

          Is there a translation guide somewhere for the language, “English bad ass flyboy”

          On second thought I have no need to translate it

    • Gail Combs says:

      Watch it Nik, not all of us girls vote for the idiotic progressives. Think about ‘dave’ and ‘steve’ and the cross dresser ‘Snow White’. they aren’t girls but I am sure they are Progressives.

      • -=NikFromNYC=- says:

        There is the Internet, the *only* culture left for dispersed populated rural West or East countries. Then there is something else entirely. There is real world high population density cities where different *types* of people get along just fine, no cultural turning into real culture war required. The cities represent peace in our time. Steve *pretends* that laypeople are NAZIs, as *if* the type of socially disconnected pressures of 1930s Germany exist now top in our day, laughably. But…Culture War. Yeah, whatever. Normal people have taken over Szep-i-Zizm, Iron Man and Ancient Gods idiot trivial fractal pattern matcher, and Steve Godwin’s desperate desire to fight a good war against real NAZIs comes out as total fodder and PR disaster idiocy. EVEN TONY WATTS DIRECTLY DISSES HIS MAIN THESIS AS ODD CHERRY PICKING. I didn’t do that. Tony did that.

        • Armies always consist of people who want to end the war, and others who want to keep it going.

        • -=NikFromNYC=- says:

          Thanks for thinking.

        • Gail Combs says:

          Steve,
          The people who ARE CANNON FODDER or whose sons are cannon fodder want to end the war.

          The Sociopaths sitting behind the lines making money hand over fist from the war want the war to continue.

          Notice how the USA has been almost continuously in wars since the Federal Reserve Act of 1913?

  8. Steven, I altered your code to add .5 degrees every 5 years since 1979 so your data could be accurate… for data prior to 1979 I deducted .3 degrees per decade going back to 1910 ..

    Finally your data matches the models.

    • Oh dear, now you’ve done it Philippe,

      Didn’t anyone tell you, DON’T FIX THE MODELS!!!

      If you fix the models, they will look just like the observations, and the whole Climate Industrial Complex will come crashing down. $21,000,000,000 worth of alarmists out of work. Harrison Ford and James Cameron, out of work. Bill McKibben, out of work. Joe Romm, out of work. Windmills spinning to the ground. Coal plants sprouting up all over.

      Where will it all end?

  9. tom0mason says:

    Thanks Steve,
    It will give me something to think about as the thaw (hopefully) sets in.

  10. & yeah, it compiles pretty cleanly:

    Main.cpp:319:28: warning: ‘&&’ within ‘||’ [-Wlogical-op-parentheses]
    if (check_ghcn && record_char_string[0] ‘Z’ )
    ~~~~~~~~~~~^~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ ~~
    Main.cpp:319:28: note: place parentheses around the ‘&&’ expression to silence this warning
    if (check_ghcn && record_char_string[0] ‘Z’ )
    ^
    ( )

  11. Steve, here is what I’m getting using ubuntu and tcsh (keep in mind I have spent 10 minutes using unix type stuff in the last 20 years):
    Reading state information… Done
    The following NEW packages will be installed:
    tcsh
    0 upgraded, 1 newly installed, 0 to remove and 4 not upgraded.
    Need to get 521 kB of archives.
    After this operation, 1,402 kB of additional disk space will be used.
    Get:1 http://us.archive.ubuntu.com/ubuntu/ precise/universe tcsh i386 6.17.06-2 [521 kB]
    Fetched 521 kB in 0s (693 kB/s)
    Selecting previously unselected package tcsh.
    (Reading database … 159907 files and directories currently installed.)
    Unpacking tcsh (from …/tcsh_6.17.06-2_i386.deb) …
    Processing triggers for man-db …
    Setting up tcsh (6.17.06-2) …
    update-alternatives: using /bin/tcsh to provide /bin/csh (csh) in auto mode.
    michael@michael-Inspiron-6000:~/Documents/AGW/release$ tcsh
    michael-Inspiron-6000:~/Documents/AGW/release> getdaily
    getdaily: Command not found.
    michael-Inspiron-6000:~/Documents/AGW/release> exit
    exit
    michael@michael-Inspiron-6000:~/Documents/AGW/release$ tcsh
    michael-Inspiron-6000:~/Documents/AGW/release> ./getdaily
    ./getdaily: Command not found.
    michael-Inspiron-6000:~/Documents/AGW/release>

    More:

    michael@michael-Inspiron-6000:~/Documents/AGW/release$ ./getdaily
    bash: ./getdaily: /usr/bin/tcsh^M: bad interpreter: No such file or directory
    michael@michael-Inspiron-6000:~/Documents/AGW/release$ getdaily
    getdaily: command not found
    michael@michael-Inspiron-6000:~/Documents/AGW/release$ ./ getdaily
    bash: ./: Is a directory
    michael@michael-Inspiron-6000:~/Documents/AGW/release$ ./getdaily
    bash: ./getdaily: /usr/bin/tcsh^M: bad interpreter: No such file or directory
    michael@michael-Inspiron-6000:~/Documents/AGW/release$ dir
    getdaily ghcnd-stations.txt Main.cpp stations.txt
    GHCN.cpp GHCN.h Makefile
    michael@michael-Inspiron-6000:~/Documents/AGW/release$

    Any suggestions? I thought the problem was in /usr/bin, but tcsh is there now that I installed it. Is there a way I can invite you to take over my machine?

    • Type “which tcsh” (without the quotes) to find out where it is located on your computer, and replace the path to tcsh in file getdaily, with the one on your computer

      • It says \usr\bin\tcsh which is where ‘getdaily’ wants it to be, and I can see tsch there. I wonder if the problem is the next line get_daily.tcsh . Where is that file?

        • That line should have a # comment at the front of it. Just delete the line.

          # get_daily.tcsh

        • Got a little further. From bash:
          michael@michael-Inspiron-6000:~/Documents/AGW/release$ /usr/bin/tcsh getdaily
          : Command not found.
          : Command not found.
          rm: No match.
          : Command not found.
          foreach: Words not parenthesized.
          michael@michael-Inspiron-6000:~/Documents/AGW/release$

          from tsch:
          michael-Inspiron-6000:~/Documents/AGW/release> /usr/bin/tcsh getdaily
          : Command not found.
          : Command not found.
          rm: No match.
          : Command not found.
          foreach: Words not parenthesized.
          michael-Inspiron-6000:~/Documents/AGW/release>

          Sorry if I’m being too dense> 🙂

        • Try running dos2unix on the file. It may have windows end of line characters

          > dos2unix getdaily

        • HA! It is running now. Thanks Steve. Might have more questions after getdaily runs. Thanks so much.

        • It ran and ended, but it only took maybe an hour. I have a total of 1196 files, but there are 22206 stations in stations.txt. So I should have 22206 .dly files + original + 49 state & USA files, correct? I do have the state files, so maybe the server just doesn’t have anything for some of the .dly files? Looks like the state.txt files are good. Why so few files?

        • Should be 1,140 stations in stations.txt

        • Ah, yes, I was reading ghcnd-stations.txt. So the stations.txt only contains 1140 (of the 22208 in ghcnd), probably only the ones with full records going back a long time? What selection criteria did you use to make stations.txt?

        • Those are the HCN stations used by USHCN and NCDC

      • So getdaily is done, now I get:
        michael@michael-Inspiron-6000:~/Documents/AGW/release$ make
        g++ -O3 -o ghcn.exe Main.cpp GHCN.cpp
        make: g++: Command not found
        make: *** [ghcn.exe] Error 127
        michael@michael-Inspiron-6000:~/Documents/AGW/release$

        Thanks

Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out /  Change )

Google photo

You are commenting using your Google account. Log Out /  Change )

Twitter picture

You are commenting using your Twitter account. Log Out /  Change )

Facebook photo

You are commenting using your Facebook account. Log Out /  Change )

Connecting to %s