GFDL - Geophysical Fluid Dynamics Laboratory

Skip to main content

Welcome to the FRE4 User Documentation

This document describes version 4 of the Flexible Modeling System Runtime Environment (FRE).The latest FRE release is Arkansas-10.

Compiled by: Tara McQueen

Table of Contents

What is FRE?

The FMS Runtime Environment (FRE) is a toolset for managing experiments from start to finish. This includes tasks such as acquiring source code, compilation, launching jobs to run models, and post-processing the output.

FRE consists of:

  • An XML model description file which contains all experiment-specific variables such as the commands used to check out the code, the path of the initial conditions file, namelists, runtime specifications, and instructions for output data post-processing. The user needs to edit this file in order to customize the model configuration.

  • CSH script templates for compiling and running models. (The user does not need to make any changes to the FRE-generated scripts.)

  • Several conventions upon which defaults are based, such as directory structure and naming.

  • Command line, Perl scripted, utilities that read the XML files and perform a specific set of functions.
    • fremake- checks out the model's code, creates/submits compiled scripts
    • frerun- creates and submits runscripts
    • frepp- creates and submits postprocessing scripts
    • frecheck- compare regression test runs
    • frelist- lists existing experiments and details in your XML file
    • frestatus- shows status of batch compiles and runs
    • frepriority- provides job and queue control for long-running jobs
    • frescrub- deletes duplicated postprocessing files
    • freconvert- convert an XML to a newer version
    • freppcheck- reports missing postprocessing files

Setup

In order to use FRE via system modules, you need to execute the commands:

For HPCS 
module use -a /home/fms/local/modulefiles 
module load fre

For Gaea 
module use -a /ncrc/home2/fms/local/modulefiles 
module load fre

This needs to be done in the shell where you will run the FRE commands, and it needs to be in the setup/platform/csh section of your XML file, so that FRE scripts can locate FRE tools. These commands are the same whether you run them on the workstation or the HPCS.

To acquire a set of XML files to use with FRE, do

For HPCS 
setenv CVSROOT /home/fms/cvs 
cvs co -r arkansas-10 xml

For Gaea 
(the CVSROOT is set by "module load fre")
cvs co -r arkansas-10 xml

What if you want to make sure you're getting a specific version of FRE?

The "fre" module will always point to the latest official release of FRE. That default will change over time. To set a specific version, you can load it with: 

module load fre/[version]

as in 

module load fre/arkansas-10

FRE releases will be named after rivers, with a number at the end for intermediate releases and bugfix releases. You can see the available versions and which one is default with the module avail command.

What if you need more fine-grained control over the versions of FRE tools?

The "fre" module actually loads a number of other system modules. You can load them individually, for example, in order to use a new version of the NetCDF utilities but stay with your old version of frerun and frepp. The following modules are available:

  • The fre-commands module contains ardiff, diag_table_chk, frecheck, freindent, frelist, fremake, fremetar, frepp, freppcheck, frepriority, frerun, frescrub, frestatus, frusavehist, list_files_with_tag, list_paths, mkmf, prepare_dir.csh, refrepp, time_stamp.csh, and versions.
  • The fre-transfer module contains lamal, hsmget, and hsmput.
  • The fre-nctools module contains combine-ncc, decompress-ncc, fregrid, fregrid_parallel, list_ncvars.csh, mppnccombine, nccmp, ncexists, plevel.sh, split_ncvars.csh, stacksize.csh, timavg.csh, varlist.csh, container.csh, list_months.csh, and split_into_months.csh.
  • The fre-curator module contains frecanon, freconvert, fredb, fredb-pp and frematar tools.

Compiling an Experiment

Once you have an XML file of the model with the experiments you wish to run, you can begin by using the FRE tool fremake, to compile. If you are unsure what experiments are in a specific xml file, use the frelist tool.

frelist -x xml_file

To compile an experiment, execute the following. Run from your xml/ directory, or provide the full path to the xml file.

fremake -x xml_file experiment_name

At the end of the generated output will be a command to submit the script to run in batch mode. It will resemble:

TO SUBMIT => qsub /home/$USER/.../experiment_name/.../exec/compile_experiment_name.csh

You can compile interactively by executing the script directly on ic0, without the "qsub."

To compile in batch mode, copy the qsub command and execute. After that, a job number will appear. The experiment is now in the process of compiling. To check the status of this batch job you may do:

qa -u $USER

Each job is identified by its provided JOBID number. Example output:

JOBID USER PROJ JOBNAME PE CPUS CPUMIN HOST STATE
459685 tlm - compile_[experiment name] ic.a 45/0 ic10 r

You may also use frestatus to print the status of batch compiles once they have started running. frestatus parses the batch stdout file, so it does not print information about jobs in the batch queue which have not yet started to run.

frestatus -x xml_file experiment_name

Running an Experiment

After your compilation has finished successfully, the next step is to submit the run scripts using frerun. Two types of runs are available for most experiments, depending on the xml tags within the experiment's <runtime> tag: regression runs and production runs. Regression runs are short runs typically executed for testing purposes. Production runs are full-length model runs that have the ability to continue over multiple batch jobs.

Execute the following for a regression run. The string "basic" indicates that a model run will be created for each of the <run> tags inside the <regression> tag with the attribute name="basic".

frerun -r basic -x xml_file experiment_name

Execute the following for a production run:

frerun -x xml_file experiment_name

Like in step 1, a submit command will appear at the end of the frerun command's output.

TO SUBMIT => qsub /home/$USER/.../experiment_name/.../scripts/run/experiment_name

As with fremake, you can compile interactively by running the script directly on ic0, without the "qsub." Do not do this unless the model run will require few resources: less than 40 processors and less than 20 minutes to run. Otherwise, to run in batch mode, copy the qsub command and execute it. After that, a job number will appear. The experiment is now in the process of running. Again, to check the status of this job you may use qa or frecheck.

Note that frecheck only reports the results of regression runs, not production runs. You can check the status of a run by looking at the batch script's stdout. To locate the batch stdout file, execute:

frelist --dir -x xml_file experiment_name

FRE Tools

fremake : Compile an Executable

fremake checks for the existence of the source code directory for the experiment to be compiled, and if it is not found, executes commands to acquire the source code from revision control systems as defined in the <source> tags in the XML file. It then creates a compile script based on the target specified on the command line and information in the <compile> tags in the XML file.

To view available fremake options run:

fremake -h

Execute fremake:

fremake [-options:] xml_file experiment_name

frerun : Run a Model

frerun creates and submits runscripts based on a shell script template and information from an XML file. To perform regression test runs, use the '-r [regression label]' argument, otherwise frerun will create a runscript for a production run based on information in the <production> tag for the experiment. If "-r suite" is specified, then all scripts for all regression runs labelled basic, scaling and restarts will be created. Regression tests are shorter than production runs and have different runtime specifications according to the XML tags.

To view available run options:

frerun -h

Execute frerun for production run:

frerun [-options:] xml_file experiment_name

Execute frerun from regression run:

frerun -r [basic,scaling or restarts] xml_file experiment_name

To run all currently available regression test, replace [basic, scaling or restarts] with suite.

frepp : Post-process Model Output

frepp is a FRE utility that creates and submits postprocessing scripts.

To view available options:

frepp -h

Execute frepp:

frepp [-options:] xml_file experiment_name

frecheck : Verify the Results of Regression Tests

frecheck runs the ardiff command to bitwise compare restart files on the output produced by frerun-generated runs and prints a report. It also generates a table listing of the runtime for different processor counts by parsing the fms.out files.

To view available options :

frecheck -h

Execute frecheck:

frecheck [-options:] xml_file experiment_name

frelist : List Information in an XML File

frelist lists the experiments in a specified XML file.

To view available options:

frelist -h

Execute frelist:

frelist [-options:] xml_file

frestatus : Check the Status of Batch Compiles and Runs

frestatus shows status and progress of batch compiles and runs. It parses batch scripts stdout to relay the status of your rtsmake and rtsrun shell scripts. Note that if you ran the scripts interactively, no status information will be found since there will be no batch script stdout files.

To view available options:

frestatus -h

Executre frestatus:

/home/fms/bin/frestatus [-options:] xml_file experiment_name

frepriority : Manage the Queues for an Experiment

frepriority implements job control for production runs. It changes batch queue information for an experiment. Without arguments, it will report job state, priority and remaining queue allocations for the experiment.

To view available options:

frepriority -h

Execute frepriority:

frepriority [-options:] xml_file experiment_name

frescrub : Delete Unnecessary Post-processing Files

frescrub deletes duplicated postprocessing files generated by FRE. frescrub will not delete any NetCDF files if any errors were reported for their association cpio files.

To view available opitions:

frescrub -h

Execute frescrub:

frescrub [-options:] xml_file experiment_name

freppcheck : Checks Post-processing files

freppcheck checks and reports missing postprocessing files.

To view available options:

freppcheck -h

Execute freppcheck:

freppcheck [-options:] xml_file experiment_name

freindent : Indent XML Tags

This is a utility to indent and align the tags in your XML file.

It will insert CDATA tags around your csh for safety. It will also substitute in entity values to your XML. If this is not desired, please try 

xmllint --format in.xml > out.xml

frerts : Streamlined Regression Testing

The frerts utility creates a shell script to launch fremake, frerun and frecheck in sequence. All the compilable experiments in the XML file will be compiled, and the suite of regression tests (all basic, restarts, and scaling regression runs) will be run. Then frecheck will be called, and the frecheck output will be emailed to the user.

The frerts utility adjusts the XML so that source, executables, stdout, and the results of the frecheck will be written to $HOME/autoRTS/date-string/. Model output will be written to $ARCHIVE/autoRTS/date-string/.

This tool does not take an experiment name. It is designed to be streamlined with very few options. The only frerts-specific option is to specify your own string you'd like to use to label the output instead of an automatically generated unique date string.

An email containing the results of the test will be sent to the user when a frerts run is finished.

The compile and run stages are done in separate, dependent jobs when you use "frerts -s".

For system testing, it is possible to run frerts without having your own XML files. Simply do: 

module load fre
frerts -s -x CM2.1U.xml

If the XML file specified doesn't exist locally, it will use the one provided in the "fre" system module.

frusavehist : Data Staging Tool

Somewhat old documentation is available at http://cobweb.gfdl.noaa.gov/~bnd/phpwiki/index.php/HistoryDataStaging

diag_table_chk : Check Diag Table Syntax

Check for errors in diagnostic tables.

FRE Curator

The Fre Curator tools allow for data publishing on the data portal. Publishing consists of converting metadata standards, maintaining quality assurance control, recording published metadata in database, searching, discovering, navigating and understanding the data in the data portal. These tools are set up to work in a chain like manor.

The Curator Database contains metadata regarding all aspects of modeling and publishing processes. This includes model configurations and runs, model output data, observations, TDS aggregations metadata, LAS configurations, project specifics, data portal statistics and user, group, data access attributes and other admidinstrative information. Project specifics include IPCC standard variable tables.

Users can log on to the Model Development Web Interface at http://cobweb.gfdl.noaa.gov:8080/extmdb/, using their GFDL log in. This will provide a means for users to see what is in the Curator Database.

Model Development Web Interface features:

The site offers a navigation panel that allows you to browse all the experiments. Furthermore, you can view corresponding experiment information. There is also a filter which allows the panel to only display experiments with specified components. Users can also compare experiments, generate analysis figures, generate RTS XML and check job status. Please view the help tag for further details and information.

There are 5 ways to access data on Data1.

OPeNDAP offers the most benefits to users by allowing them to access data anywhere on the Internet and offering a wide variety of programs.

LAS is a configured web server that provides access to data using an easy to navigate user interface.

FreCurator tools interact and operate in the following manor.

fredb-pp : Data Publishing

fredb-pp is at the highest level of the Fre Curator chain. Users can start here with only an XML experiment and arbitrary directory of model output. This tool will establish default values for lower-level tools, call the lower level tools accordingly and override default values by accepting command-line parameters. fredb-pp internally calls fredb in order to map an XML file to the Curator database. It will generate and store a checksum of the XML file to prevent redundant calls to the fredb script. The output from fredb call will be passed into a fremetar call.

To view available options:

fredb-pp -h

Execute fredb-pp:

fredb-pp [-options:] -x xml_file -n experiment_name -d model output directory

-w will call fremetar after mapping experiment to database to rewrite output metadata.

fredb : Data Publishing Manually

fredb is the next entry point into the Fre Curator chain. It is suggested that only power-users or developers use this instead of fredb-pp to manually map an experiment. It produces the experiment id, realize id and run id (triple ID) that is needed for fremetar to access the metadata. fredb is intended to only be called once for a single experiment. It will internally call freconvert and frecanon in order to ensure that the xml syntax has been updated and that the xml is compatible for mapping into the Curator database. A successful fredb call is mandatory for fremetar to be initialized.

To view available options:

fredb -h

Execute fredb (maually):

fredb [-options:] -x xml_file experiment_name

freconvert : Update XML Syntax

freconvert coverts old experiment XML files into the latest version.

To view available options:

freconvert -h

Execute freconvert (manually):

freconvert [-options:] -i input_file output_file

frecanon : Prepare XML for Input into Curator Database

frecanon dereferences all inherited XML fields and external links to prepare an XML file for entrance into the Curator Database. It will flatten the inheritance tree of experiment configuration files into a single, independent experiment. It is a bridge between FRE tools and FREratorMap. This process makes the XML self sufficient. It can be called independently by advanced users or called automatically by fredb.

To view available options:

frecanon -h

Execute frecanon (manually):

frecanon [-options] -x input_file -o output_file  experiment_name

FREMetar : Standardize Post-processing NetCDF Metadata

FREMetar was developed to eliminate the CMORization step after postprocessing. It adds or corrects NetCDF metadata using CMIPstandard metadata specifications in Curator. It works on global, axes and variable metadata. In addition, FREMetar renames NetCDF files to adhere to CMIP specifications. It is the final entry point to the Fre Curator chain and is primarily for developers. It requires the database triple ID for an experiment that was mapped to the database. FREMetar will be invoked by fredb-pp unless specified otherwise while running manually.

To view available options:

fremetar -h

Execute fremetar (manually):

fremetar -d dir -e experid -z realizid -r runid [-options:]

(must provide dir, exper_id, realiz_id and run_id) note- this triple ID produced by fredb and can be found in the database.

FREMapService / FREMapClient : Quality Assurance Control

These Java applications ensure database integrity and control write access to the Curator database. When a fredb call is made, the FREMapClient application communicates with the FREMapService and passes it the user's runtime environment information. FREMapService will then either call delete.csh, for removing an experiment from the database, or mapping.csh, for adding one. mapping.csh is responsible for calling FREratorMap . This high level structure controls the queue of experiment mappings in order to prevent processing two XML's at the same time and defends data integrity. FREMapService and FREratorMap are running on the cobweb machine where the curator DB is also located.

FREratorMap

FREratorMap is a Java application that takes an XML file and maps it's elements to fields in a database. The approximate time for this process is 5-15 minutes. After it is completed, a log file matching the date and time of the run will be found in /home/pcmdi/fre4/log. Within that log file will be experiment name, xml file name, user name, the unique triple ID and occurring errors. The triple ID can be used to obtain information from searching in the database. The existence of errors will ratify whether or not the file was run successfully. Experiments are stored in the DB in a manner where redundency is completely excluded. Any element of stored experiments are used in later mappings. This leads to comprehensive checking existing in DB elements during experiment loading in order to reuse them.

Automated Implementation

Data publishing can be further automated starting within the XML scripts. Users can put an analysis tag that creates a datapublisher template. This datapublisher will later call fredb-pp after calling frepp.

XML Files

XML (Extensible Markup Language) is a markup language similar to HTML, but with customizable sets of tags. We have defined tags appropriate for the information FRE tools need about climate models. FRE XML files contain all experiment-specific variables. Users may edit these XML files in order to customize the model configuration.

Editing XML Files

XML files can be edited with your favorite text editor, however, some editors provide special features useful for editing XML. The suggested XML editor is SciTE. SciTE is a general text editor that provides syntax highlighting similar to nedit, but SciTe also includes folding, which is useful for organizing your view of the xml. SciTe documentation is available online. VIM, emacs and nedit can perform syntax highlighting automatically. VIM and emacs can be configured to do folding and tag completion. To find out how to set up VIM or emacs to do these tasks, see the editor links on http://cobweb.gfdl.noaa.gov/~bnd/phpwiki(page only available from inside GFDL.)

Sample FRE XML Files

A sample XML file documents all the tags which are available for FRE version 4 xml, but needs to be updated for FRE version 4. It is available syntax-highlighted in HTML format at XML Tag Descriptions. Also, if you use Mozilla, you can browse the tree structure of the XML at this link: http://cobweb.gfdl.noaa.gov/fms/fre/example/rts.xml.

Variables in FRE XML files

Predefined Placeholders

Predefined placeholders can be used anywhere in your XML file: 
$(rootDir)        - the value of your <directory type="root"> </directory>
$(root)           - same as $(rootDir), also can be used without parentheses (for compatibility), might be deprecated in the future
$(siteDir)        - the directory with site-dependent configuration files
$(suite)          - the name of your XML file without path and extension (convenient to organize directories)
$(platform)       - value of --platform option (default is "gfdl.default" @ GFDL and "ncrc.default" @ Gaea)
$(target)         - value of --target option (default is "prod")
$(name)           - experiment name, also can be used without parentheses (for compatibility)
$(srcDir)         - the value of your <directory type="src"> </directory>
$(execDir)        - the value of your <directory type="exec"> </directory>
$(scriptsDir)     - the value of your <directory type="scripts"> </directory>
$(stdoutDir)      - the value of your <directory type="stdout"> </directory>
$(workDir)        - the value of your <directory type="work"> </directory>
$(archiveDir)     - the value of your <directory type="archive"> </directory>
$(postProcessDir) - the value of your <directory type="postProcess"> </directory>
$(analysisDir)    - the value of your <directory type="analysis"> </directory>

Please note that the parentheses aren't optional here, they are required. This is done to distinguish placeholders, which are to be expanded by FRE tools, from variables that will be expanded by c-shell in the scripts.

Predefined placeholders can be used inside the XML file to refer to two classes of data items: (1) options and arguments you supplied when you called a FRE tool (suite, name, platform, target), and (2) various directories which are defined for the selected combination of platform and target.

Rules to use directory placeholders in the <setup> section. The root directory is intended to be a reference location which many other directories are based on - so it must be defined before the directories which are dependent upon it. This means that you can define source directory using the $(rootDir), but not vice versa. Also all the directories have the specification order as above. This means you can define postProcess directory using $(archiveDir), but not vice versa, because the postProcess directory is listed after the archive directory. The $(analysisDir) is the last one, so it can't be used to define other directories.

User-defined Placeholders

You can define your own placeholder variables with properties. 

<property name = "foo" value = "bar"/>
..$(foo)..

Global properties are the immediate children of the experiment suite. Platform-dependent properties are immediate children of the platform. Please use <property> rather than XML entities when possible.

Environment Variables Expanded by FRE

These shell environment variables will be expanded by FRE: 
$HOME
$USER
$ARCHIVE

Directories

You can customize directories where various files will be located by editing the directory specification tags at the top of your xml file.

Available Directory XML Tags 

<setup>
   <platform name="hpcs.hpcs">
   <directory>
      <root> a reference location </root>
      <src> for FMS source code </src>
      <exec> for object files and executables </exec>
      <scripts> for run and postprocess scripts </scripts>
      <stdout> for run and postprocess stdout </stdout>
      <work> a scratch dir where executable is run </work>
      <archive> for model output data </archive>
      <postProcess> for postprocessing output </postProcess>
      <analysis> for analysis figures </analysis>
   </platform>
   </directory>
</setup>

Directory Defaults 

*root:* $(suite)/xml
*src:* $(rootDir)/$(name)/src
*exec:* $(rootDir)/$(name)/$(platform)-$(target)/exec
*scripts:* $(rootDir)/$(name)/$(platform)-$(target)/scripts
*stdout:* $(rootDir)/$(name)/$(platform)-$(target)/stdout
*work:* $TMPDIR/$(name)/$(platform)-$(target)
*archive:* /archive/$USER/$(name)
*postProcess:* /archive/$USER/$(name)/pp
*analysis:* /net2/$USER/analysis/$(name)

frelist can show the directories for an experiment: 
frelist -dir -x xml_file
frelist -dir -x xml_file experiment_name

Platforms and Sites

The site prefix in the platform name must be "hpcs" or "doe". Example: 

hpcs.hpcs
hpcs.ifort10
doe.pathscale

At GFDL, the default site is "hpcs". Therefore, platform name="hpcs" is equivalent to platform name="hpcs.hpcs".

At DOE, the default site is "doe".

Please refrain from using other dots in your platform name.

Compilation Targets

FRE provides three compilation targets you can specify with the --target command line option:

  • prod : high levels of code optimization, for production runs
  • repro : compiler flags (such as -fltconsistency) to facilitate reproducible answers across different processor counts
  • debug : debug options, low optimization, high verbosity

If you do not specify a "--target" option to fremake, frerun, frestatus, etc, you will get the "production" compilation configuration.

There are predefined target extensions for other compilation needs:

  • hdf5 : add directives to generate HDF5-based NetCDF files
  • openmp : add directives to compile and link with OpenMP

You can use these in conjunction with the main targets, as in "--target repro,hdf5".

If the predefined options are not sufficient, you can either amend the predefined targets in your XML file, or you can define your own targets. Enclose your directives into a node, and place it into a node with a "target" attribute: 

    <component name="XYZ" ... >
      ...
      <compile target="TargetName">
        ...
        <cppDefs> compiler/linker directives </cppDefs>
        ...
      </compile>
      ...
    </component>

More documentation on this is available at http://cobweb.gfdl.noaa.gov/~afy/fre_targets.html.

Make templates and Targets

Users do not need to specify make template files in the xml unless they have special needs that the basic options do not cover. The default make templates are written so that there is only one make template per platform. They have various sets of compile options defined. FRE will set these automatically based on the target: 
    REPRO=true if the target contains the "repro" substring 
    DEBUG=true if the target contains the "debug" substring
    NETCDF=4   if the target contains the "hdf5" substring and platform/csh contains a system 
               module which loads "netcdf-4.0.1" or newer (otherwise it will default to 3)
    OPENMP=on  if the target contains the "openmp" substring

FRE version 4 is backwards compatible with FRE version 2 and version 3 files, so leaving old settings should work and give the same results as before.

SrcList - Compiling extra source code

The <srcList> tag provides a way to specify extra source code files to include in the compilation.

Inheritance

It is possible for an experiment to inherit parameters from another experiment in the same XML file. A sample XML illustrating inheritance is shown at http://cobweb.gfdl.noaa.gov/fms/fre/example. Aside from a few special cases, the rules governing inheritance are as follows.

  • fremake and frerun will look inside the experiments for the data.
  • If no data is found, fremake and frerun will look for inherit in the attribute tag. If an inherit attribute is found, fremake and frerun will look inside the given experiment for the data. It will recurse in this manner until the data is found or until there are no more experiments from which to inherit.
  • If the data is not found in the inheritance tree, the value will be empty. If the value was required, an error message will be printed. If the value was optional, a warning message will be printed if you use the -v option on fremake and frerun.

If you want to run several configurations of the same model (using the same executable but different namelists or other input files) you don't need to check out and compile the same code multiple times. Compile once and use the 'inherit' attribute on the remaining experiments.

Special Case: Namelists

Namelists will be parsed and given priority as follows:

1. Namelists directly in the XML of the experiment 
<experiment name="mom4_test1_static10" inherit="mom4_test1">
   <input>
      <namelist name="ocean_model_nml">
      baroclinic_split = 4
      surface_height_split = 1
      barotropic_split = 60
      energy_diag_freq = 12
      debug=.false.
      acor=0.50
      robert=0.05
      layout=2,5
      </namelist>
   </input>
</experiment>

2. Namelists in files, in the order the files are given in the XML 
<experiment name="mom4_test1_static10" inherit="mom4_test1">
   <input>
      <namelist file="/home/user/file1"/>
      <namelist file="/home/user/file2"/>
   </input>
</experiment>

3. Namelists that are directly in the XML of the parent experiment.

4. Namelists files from the parent experiment.

A warning will be printed if you attempt to specify a namelist more than once in the inheritance tree, and an error will be printed if you try to give the same namelist twice directly in XML tags for an experiment. It is currently not possible to inherit only some values from a given namelist.

Special Case: Field Tables

Field tables are currently not parsed and are inherited on the basis of their file names. If you specify at least one field table, no field tables will be inherited from the parent experiment.

Special Case: Executable

The name of the executable has a default value if not specified anywhere. The default location is $root/$name/exec/fms_$name.x. The FRE scripts decide whether a child experiment should have its own executable based on whether you have specified new data in the or sections of the XML for your child experiment. If you intend for your experiment to inherit an executable, you should not re-specify anything in the sections of your child experiment, because then fremake will think you wanted to recompile with the new data. Actually, you do not need to run fremake on experiments which inherit an executable since the purpose of fremake is to create an executable.

Special Case: mkmf template

The location of the mkmf template is a special case because by default it is controlled by FRE. The default can be overridden by specifying the mkmf template in the setup section at the top of your XML file: 
<setup>
   <mkmfTemplate file="/path/to/template/file"/>
</setup>

Or the mkmf template can be specified in the XML directly: 
<mkmfTemplate>
 FC = f90
 CPPFLAGS = -macro_expand
 etc
</mkmfTemplate>

That in turn can be overridden on an experiment-by-experiment basis. The location of the mkmf template can be specified for an individual experiment in the compile section of the XML. 
<experiment name="mom4_test1_static10" inherit="mom4_test1">
   <compile>
      <mkmfTemplate file="/path/to/template/file"/>
   </compile>
</experiment>

Extending a production run

After a run has finished, take the following steps:

1. Increase the length of the run in the XML to the new full length of the run. (Update the <production> tag for the experiment.)

2. Use frerun -ext to generate and submit a new runscript.

3. Use frepriority to increase the number of queue allocations.

During a run,

1. Increase the length of the run in the XML to the new full length of the run. (Update the <production> tag for the experiment.)

2. Use frerun -ext to generate a runscript. frerun will overwrite the existing runscript. Do not use the -s option or submit the script again. The next time the script reloads, it will pick up the new version of the runscript.

3. Use frepriority to increase the number of queue allocations.

NetCDF Library Specification

For Riga and later FMS code and FRE tools, the default is to load the latest NetCDF library (module load netcdf-4.0.1) but create NetCDF classic format output files with 64 bit support. FRE will adjust the compiler options and cppDefs; you do not need to have -Duse_netCDF3 or -Duse_LARGEFILE in your XML file.

If you do want to output the new NetCDF file format, use the target "hdf5". You can specify this along with the predefined targets listed above, ie, "--target debug,hdf5".

Using TotalView with FRE

TotalView is GUI-based debugger. This analysis tool provides control over processes and thread execution with program state and variable visibility. This tool is ideal for reproducing and troubleshooting errors that can occur in programs.

To initiate TotalView during a fre experiment run, follow these steps:

fremake -t debug -x xml_file experiment_name

Submit job interactively (do not use qsub), for example:

/home/tlm/riga_201006/SM2.1U_Control-1990_lm3v_pot_A1/debug/scripts/run/SM2.1U_Control-1990_lm3v_pot_A1_1x0m8d_30pe*

After fremake is finished, do a frerun:

frerun -t debug [options] -x xml_file experiment_name

When that completes, the path to an executable with appear. This runscript must be edited for totalview to work. There are two steps to editing this file. First, you need to add a module load in the "global environment settings" section. There will be a list of module loads there. One needs to be added for totalview. *Note the example shows one edition of totalview, use the edition specific for you. To see available editions, do a module avail and select from there.

module load totalview.8.8.0-0

Second, you need to edit one line in the runscript. The line will look as follows:

/usr/bin/time -p mpirun -np $npes $executable:t |& tee fms.out

*Note that mpirun is used on hpcs and apirun is used for doe. Use which one applies to you. Change line to:

totalview mpirun -a -np $npes ./$executable:t

Now you can save the runscript and execute it. A few minutes into the run, Totalview will appear as 3 windows.

Click on the TotalView link for user documentation.

Compiling with Libraries

FRE can use pre-compiled libraries (.a files) at the link time if their paths are provided in a <library> subsection inside a <:component> section as shown in the example below: 

<component name="fms"  paths="shared" includeDir="/PATH_TO_src_DIR/shared/include" >
       <library path="/PATH_TO_LIBS_DIR/libfms.a"  headerDir="/PATH_TO_LIBS_DIR"/>
</component>

Note that "headerDir" is where the .mod files for that library are located and they are needed to be present and uptodate at the link time. The "make" utility decides whether the library is up-to-date and recompiles it if needed.

Also note that the <library> subsection replaces both the <source> and the <compile> subsections in the <component>.

A working example of using precompiled libraries is given below. In this example the libraries are already compiled in the parent experiment "mom4p1_coupled" and only one component ("mom4p1") needs to be compiled. 

<property name="MY_LIBS"         value="$root/mom4p1_coupled/$(platform)-$(target)/exec"/>

<experiment name="mom4p1_coupled_10by5" inherit="mom4p1_coupled">
  <description>
Same mom4p1_coupled  but with static memory allocation.
  </description>
   <component includeDir="$root/mom4p1_coupled/src/shared/include" paths="shared" name="fms" >
     <library path="$(MY_LIBS)/libfms.a"
         headerDir="$(MY_LIBS)"/>       </component>

   <component paths="ocean_shared" requires="fms" name="ocean_shared" >
     <library path="$(MY_LIBS)/libocean_shared.a"
         headerDir="$(MY_LIBS)"/>       </component>

   <component paths="atmos_null" requires="fms" name="atmos_null" >
     <library path="$(MY_LIBS)/libatmos_null.a"
         headerDir="$(MY_LIBS)"/>       </component>

   <component paths="ice_sis ice_param" requires="fms" name="ice_sis" >
     <library path="$(MY_LIBS)/libice_sis.a"
         headerDir="$(MY_LIBS)"/>       </component>

   <component paths="land_null" requires="fms" name="land_null" >
     <library path="$(MY_LIBS)/libland_null.a"
         headerDir="$(MY_LIBS)"/>       </component>

  <component name="coupler" paths="coupler" requires="fms land_null atmos_null ice_sis mom4p1_static_10by5">
     <library path="$(MY_LIBS)/libcoupler.a"
         headerDir="$(MY_LIBS)"/>      </component>

   <component paths="mom4p1" requires="fms ocean_shared" name="mom4p1_static_10by5" >
     <source root="/home/fms/cvs" versionControl="cvs" >
       <codeBase version="$(MOM_CVS_TAG)" > mom4p1_coupled </codeBase>
     </source>

     <compile>
       <cppDefs>"-DUSE_OCEAN_BGC -DENABLE_ODA
    -DMOM4_STATIC_ARRAYS -DNI_=360 -DNJ_=200 -DNK_=50 -DNI_LOCAL_=36 -DNJ_LOCAL_=40"
       </cppDefs>
     </compile>
   </component>

</experiment>

Development Runs that Do Not Write to Archive

With the new "frerun --noarchive" feature, you can run regression tests interactively without saving your output data to /archive. No cpio's will be created, which means things should go faster for you when you're trying to run short tests as part of a development cycle. To use it:

frerun --noarchive -r basic -x xml_file expt

Things to keep in mind:

1. Note that frecheck currently will not find output data that has not been archived. Support for this will be added to frecheck in a future release.

2. Your output data will be saved in the "ptmp" location. The default for non-IPCC users is 
<ptmp>$TMPDIR/ptmp</ptmp>

Thus the /ptmp directory will be local to $TMPDIR, so you will only be able to see your output data if you run interactively. (If you run in batch, it'll get deleted at the end of the job.)

Your output data will be in a path like

$TMPDIR/ptmp/archive/$USER/path-to-expt/exptname/...

also known as

$TMPDIR/ptmp/$(archiveDir)

3. If you do a run interactively with riga xml settings, you'll end up with 
$TMPDIR/
  ptmp/
  work/

You might be tempted to "wipeftmp" between runs, but you may want to be more specific than that -- Keeping the input data in the $TMPDIR/ptmp directory will help speed data movement along.

4. If you run the same runscript multiple times during debugging, be careful of pre-existing output data in your ptmp directory.

Data Staging

Staging the experiment's history output helps use computing resources more efficiently, because main jobs do not need to wait for mppnccombine of history data. A separate job combines data, and calls frepp. It is the recommended way to run production models.

Specify it with: 
 <postProcess combine=staged">

Data staging includes these features:

  • automatic catchup of postprocessing
  • automatic repair of history on first segment of model year
  • support for using /ptmp to stage history data
  • will not repeat completed steps after system crash

Information in stdout matches that in master log file.

To repair staging after a problem has occurred, look in /qinfo4fre/.../expt/. There are stdout and log files useful for troubleshooting. There are two kinds of files in the "qsync" subdirectory:

  • s: staging jobs not complete yet
  • p: postprocessing jobs not submitted yet

If a problem occured during staging job, the execute bit on the pp script will not be set. You should get an email saying you need to fix the history file before the postprocessing can be run. For riga staging script, when you have verified that the history file is correct, set the execute bit on the pp file, the staging script will take care of the rest the next time it is launched for the experiment. For pre-riga versions, you must call frepp -p and delete N pp files from the qsync directory.

Post Processing

One can create time series and climatological averages with the utility frepp. This can be called from the runscript as the model runs, or offline. Please note that in order to take advantage of the postprocessing, your runs should start on January 1, otherwise monthly and seasonal averages will not be able to be properly calculated. If you want to start your model run on another date, please run from that date to January 1, then start a new run including the postprocessing. Also, the postprocessing can handle segment lengths of 1 month, 6 months, or 1 year. It does not currently support other segment lengths.

Check for missing postprocessing files

The freppcheck utility checks for missing postprocessing files. It will ask for the start year and end year you want to check. Call as follows:

freppcheck -x xml_file experiment_name

It is suggested to run freppcheck with the -A option to check all variables and cpio files before running frescrub, which will delete intermediate files.

freppcheck -A -x xml_file experiment_name

Interpolation to different vertical levels

Several options are available for interpolating your data to different atmospheric pressure levels.

Keyword Description
ncep 100000 92500 85000 70000 60000 50000 40000 30000 25000 20000 15000 10000 7000 5000 3000 2000 1000
era40 100000 92500 85000 77500 70000 60000 50000 40000 30000 25000 20000 15000 10000 7000 5000 3000 2000 1000 700 500 300 200 100
hs20 2500 7500 12500 17500 22500 27500 32500 37500 42500 47500 52500 57500 62500 67500 72500 77500 82500 87500 92500 97500
am3 100000 92500 85000 70000 60000 50000 40000 30000 25000 20000 15000 10000 7000 5000 3000 2000 1000 500 300 200 100
ar5daily 100000 85000 70000 50000 25000 10000 5000 1000

To interpolate native model verical levels to these vertical levels, add the attribute zInterp to the appropriate component node of your XML as follows: 

<component type="atmos" source="atmos_month" zInterp="ncep">

If you want to get data both on the original model levels and interpolate to new vertical leaves, you can use more than one component, for example: 

<component type ="atmos" source="atmos_month">
...
<component type ="atmos_era40" source="atmos_month" zInterp="era40">
...

This will give you the post-processing directory called atmos with data on the original model levels, and a directory called atmos_era40 with data that has been interpolated to era40 levels.

Creating postprocessing data or figures offline

It is fairly straightforward to submit the postprocessing offline if you know which arguments are needed. Here are some useful options: 
          -t year      = the (first) year of data to process
          -p num       = "plus num years": additional years to process after the first year
          -d dir       = path to history directory [default $archive/$name/history]
          -A           = generate analysis figures only, based on existing pp data

Generate postprocessing data or figures from someone else's experiment

  1. Copy their XML file to a directory you own. This will be the root location for postprocessing scripts.
  2. Edit the section at the top of the XML file. The root directory should be the directory where you have placed the XML file, the archive directory should be the root location where you want postprocessing data to be placed, and the analysis directory should be the root location where you want analysis figures to be placed. Inside of the archive and analysis directories, a directory level with the experiment name will be created, and the pp directory will be created inside that.
  3. Call frepp, making sure to use the -d option to specify the location of the original history files you want to process, ie, -d /archive/user/cm2/cm2o_cmip/history. For example, you might use a command like frepp -d /archive/user/cm2/cm2o_cmip/history -t 0001 -p19 -x CM2.xml cm2o_cmip to create the postprocessing data and figures for years 1-20 for experiment cm2o_cmip.

Automated creation of diagnostic figures

A list of analysis scripts currently available for use with FRE is available at /home/fms/analysis/.

These analysis tags should be put inside of the tags for the data that the analysis with FRE, you need to convert your driver script to a template script by inserting the following lines and the beginning of your driver script. You only need to insert the lines your script needs. 

#---- VARIABLES SET BY FREPP ----#
set in_data_dir    #pp directory containing files to be analyzed (.../pp/atmos/ts/daily/5yr)
set in_data_file   #list or csh expression for all filenames to be analyzed
set descriptor     #experiment name
set out_dir        #directory to write output files
set WORKDIR        #working directory for script execution
set hist_dir       #directory containing history files
set frexml         #path to xml file
 
# plotting years
set yr1            #first year to analyze, from -Y
set yr2            #second year to analyze, from -Z
set specify_year   #single year to analyze
 
# data years, only used for making description file, only apply to ferret scripts using
# time series as input
set databegyr      #same as yr1 but adjusted if necessary to match timestamp of earliest data needed for analysis
set dataendyr      #same as yr2 but adjusted if necessary to match timestamp of last data needed for analysis
set datachunk      #the chunkLength of the timeSeries in years
set MODEL_start_yr #the beginning of the model simulation
set freq           #the frequency of the time series
 
# Specify batch mode "batch" or interactive mode "interactive" set mode
# Specify mom's version, either om2 or om3 because some files depend on mom's grid set mom_version
# used as mask file set gridspecfile
# used as mask file set staticfile

Then put the template script in the analysis tag in an XML file like this:

<analysis script="/home/user/bin/my_analysis_script.csh"/>

Then frepp will read your template script, specify the variables, create complete scripts, and submit them if the mode is set to batch. Here is a listing of the available attributes for the analysis tag. 

<analysis switch="on" mode="batch" momGrid="om3" specify_year="4-digit year"
          startYear="4-digit year" endYear="4-digit year"
          outdir="where you want to save your figure and text outputs"
          script="your template script with full path"
          cumulative="yes"/>

Only the script attribute is required; all others are optional. Here is a description of available attributes. The first value shown for the attribute is the default setting, followed by other valid options in descending priority.
  • switch="on|off" signals frepp to run|not run the analysis
  • mode="batch|interactive" signals frepp to submit|not submit a script automatically
  • momGrid="om3|om2_173jrows|om2_174jrows" i specifies the mom grid if necessary,
  • specify_year="XXXX" specifies a single year required by user's scripts, usually for processing daily data.
  • startYear="XXXX" specifies a specific year to start producing figures. The default value is the first year for which postprocessing data is available.
  • endYear="XXXX" specifies a specific year to stop producing figures. The default value is the last year for which postprocessing data is available.
  • outdir="" specifies the output directory where you want to save your figure and text output. You can also specify a directory with the -O command line argument to frepp, or with the attribute in the setup section of your xml file. If none of the above methods are used, the figures will be placed in /net2/user/analysis/$expt
  • script="" specifies the template script that will be read by frepp.
  • cumulative="yes|no" specifies whether the analysis will be cumulative (from beginning of run to latest available data) or just for the date range yr1-yr2.

If your analysis scripts work with more than one experiment, then you need to tell FRE what those additional experiments are with the tag as follows. 

<analysis>
    <addexpt xml="your-xml-file" name="your-expt-name">
</analysis>

Here is an example: 

<analysis script="/home/fjz/rtspp_analysis/templates/atw_atmos_obsmodmod_ts_mon.csh">
   <addexpt xmlfile="/home/ccsp/fjz/ipcc_ar4/CM2.1U_Control-1990_E1.xml name="CM2.1U_Control-1990_E1"/>
   <addexpt xmlfile="/home/ccsp/fjz/ipcc_ar4/CM2.1U_Control-1860_D4.xml name="CM2.1U_Control-1860_D4"/>
</analysis>

In your analysis scripts, insert these lines at the beginning to refer to the additional experiments. 
#---- VARIABLES SET BY FREPP ----#
set descriptor
set in_data_dir
set in_data_file
set out_dir
 
set descriptor_2
set in_data_dir_2
set in_data_file_2
set out_dir_2
 
set descriptor_3
set in_data_dir_3
set in_data_file_3
set out_dir_3

Creating monthly data by averaging daily data

To average daily data into monthly data, you need to add a special monthly timeSeries with the averageOf attribute to your XML: 

<component type="atmos" zInterp="ncep" start="0001" source="atmos_month">
   <timeSeries freq="daily" source="atmos_daily" chunkLength="5yr"/>
   <timeSeries freq="monthly" chunkLength="5yr"/>
   ...
   <timeSeries freq="monthly" averageOf="daily" chunkLength="5yr">
      <variables> t_ref_min t_ref_max </variables>
   </timeSeries>
</component>

There are several things to note about his functionality:

  1. You can either calculate this in the same postprocessing job with the or do it anytime after the daily timeSeries has been created. (ie, you can run just this piece of postprocessing offline.)
  2. The chunkLength requested for the averaged file must be the same as the chunkLength of the daily timeSeries file, ie, 5yrs. Longer chunks of pp should pick up the new variables and create 20yr timeSeries from the 5yr ones.
  3. This is specifically designed with the noleap calendar in mind and currently does not check the model calendar.
  4. The tag is required for this type of timeSeries calculation.

Calculating long timeseries from existing timeseries

The timeSeries and timeAverage attribute 'from' can override what chunklength or interval pp you want to calculate a timeSeries or timeAverage from, ie 

<timeSeries freq="monthly" chunckLength="200yr" from="100yr"/>

So if you just want to create a 200yr timeSeries from existing 100yr timeSeries, your XML file should contain ONLY the timeSeries line above, with no other tags. Simply adding the line above to your existing XML file in this situation is a bad idea, because you will be wasting cpu cycles. The frepp utility will calculate each you list in your XML file, so by adding the 200yr timeSeries line above to an existing XML file, your frepp call will duplicate all of the other work that has already been done before creating the 200yr timeSeries.

Naming Conventions

Source Code and Compilation

The program fremake creates a CVS checkout script in the $(src) directory. See the "Predefined Placeholders" and "Directories" sections of this document for more information on how the directory locations are defined.

The compilation script is then created in the $(exec) directory. The executable created by the script will be in that directory as well.

Runscript

The program frerun will create one or more runscripts. There are two methods of determining the name for a runscript based on whether you are running regression tests ( frerun is invoked with the -r regression_name argument) or a production run ( frerun is invoked without the -r regression_name argument).

For production runs, frerun will create the runscript as $(scripts)/run/$name, deriving the runtime information from the <production> element in your XML file. An example element is shown here. 

<runtime>
   <production simTime="8" units="years" npes="45">
      <segment simTime="1" units="months" runTime="00:44:00"/>
   </production>
</runtime>

The production runscript will run the full simulation time of 8 years in 1 month segments as denoted above, restarting itself as needed every 8 hours of run time.

For regression tests, frerun will derive the runtime information from the <regression> element(s) in your XML file. Example elements are shown here. 

   <runtime>
      <regression name="basic">
         <run days="8" npes="15" runTimePerJob="00:30:00"/>
      </regression>
      <regression name="restarts">
         <run days="4 4" npes="15" runTimePerJob="00:20:00"/>
         <run days="2 2 2 2" npes="15" runTimePerJob="00:20:00"/>
      </regression>
      <regression name="scaling">
         <run days="8" npes="1" atmos_layout="1,0" ice_layout="1,0" runTimePerJob="04:00:00"/>
         <run days="8" npes="3" runTimePerJob="02:00:00"/>
         <run days="8" npes="45" runTimePerJob="00:20:00"/>
         <run days="8" npes="60" runTimePerJob="00:20:00"/>
      </regression>
   </runtime>

To run a regression test with the information in the regression element labeled "basic" above, use frerun -r basic experiment name. Then a runscript will be created at $(scripts)/run/$name_$runparams, where $runparams is a string determined by the length of the run, the number of times the executable is called within the script, and the number of processors used. In the "basic" example above, $runparams would be 1x0m8d_15pe, which translates to "one times zero months, eight days on 15 processors".

A single frerun command may create more than one runscript for a given experiment. The runscripts will have different $runparams strings. With the example XML above, frerun -r restarts $name would create two runscripts, and frerun -r scaling $name would create four runscripts. The program frerun also recognizes the keyword suite, which would create runscripts from each of the three regression elements basic, restarts and scaling.

Runscript Output

Output directories are placed in the $archive/$name directory. If you specify the following: 
<directory type="archive">/archive/$user/fre/$name</directory>

Then production output would be as follows for experiment am2p10: 

/archive/$USER/fre/am2p10/
|-- ascii - directory containing text output files
|-- history - directory containing output as specified by the diagnostics table
|-- restart - directory containing datasets which can be used as initial conditions for future runs
|-- pp - directory containing post-processed history output

Regression test output utilizes another output directory level named for the runtime information as described above. The example regression elements shown in the previous section would produce the following output structure: 
/archive/fms/fre/am2p10/
|-- 1x0m8d_15pe
|   |-- ascii
|   |-- history
|   `-- restart
|-- 1x0m8d_1pe
|   |-- ascii
|   |-- history
|   `-- restart
|-- 1x0m8d_3pe
|   |-- ascii
|   |-- history
|   `-- restart
|-- 1x0m8d_45pe
|   |-- ascii
|   |-- history
|   `-- restart
|-- 1x0m8d_60pe
|   |-- ascii
|   |-- history
|   `-- restart
|-- 1x1m0d_45pe
|   |-- ascii
|   |-- history
|   `-- restart
|-- 2x0m4d_15pe
|   |-- ascii
|   |-- history
|   `-- restart
`-- 4x0m2d_15pe
    |-- ascii
    |-- history
    `-- restart

Porting (under construction)

The FRE installation at GFDL

Fre depends on several different packages which are all open source. These packages are located in /home/fms/local/opt/. These packages can be further divided into 4 categories, Perl, Python, Java and General. Below is a a table depicting what packages are related to what language.

Environmental Modules
Perl Python Java General
XML-Dumper lxml apache-log4j gd
XML-LibXML   java-getopt nccmp
XML-NamespaceSupport   jdk nco
XML-SAX   mysql-connector-java  
GD   poi-bin  
GDGraph   netcdfALL  
GDTextUtil      

FRE assumes that NetCDF library exists in the system.

In /home/fms/cvs/local/install, there is a list of installation scripts. There are proper lib tar files (file extension .gz) to match what is in this list.

FRE itself contains 6 sub packages. They are found at /home/fms/cvs/fre/. 
|-- fre-commands
|   |-- bin
|   |-- lib
|   |-- etc
|   |-- site
|       |-- hpcs-------> fre.properties, runscripts and .mk files
|       |-- doe-------> fre.properties, runscripts and .mk files
|       |-- ncrc-------> fre.properties, runscripts and .mk files
|-- fre-transfer
|   |--
|-- fre-nctools
|   |-- 
|-- fre-curator
|   |--
|-- fre-analysis
|   |-- 
|-- fre-xml
|   |--

The site/ stores the settings for the current environment. It contains files for specific systems such as hpcs and doe. All new machines will have an added directory here. Site( ) is a function used in applications found with in the bin/ directory. When called, it will return the prefix of the domain name. For example, if you are ncrc.gov then the site returned from Site() is ncrc. This will be used as a sub directory name for configuration files as is hpcs and doe are. Within that directory resides fre properties, runscripts and .mk files.

Tips for porting FRE to external sites

Prerequisites

External sites will need to have certain prerequisites in order to use FRE. FRE assumes that NetCDF library exists in the system. This requires you to install (download and compile) !NetCDF 4.0.1.

Below is a chart of Environmental Modules that need to be installed onto the external site. These packages are divided into 3 categories, Perl, Python and General.

Environmental Modules
Perl Python General
XML-Dumper lxml gd
XML-LibXML   nccmp
XML-NamespaceSupport   nco
XML-SAX    
GD    
GDGraph    
GDTextUtil    

Here is a list of prerequisites with version specification. They should be installed in this order (dependencies are listed together) along with the Environmental Modules listed above.

Porting Setup

There will be a list of installation scripts that match the prerequisites listed above. Within these scripts, code lines need to be edited in order to unpack, call configure, call make and call make install. There are if-else statements that check the platform in which the program is running on. You will need to add another else statement to match the domain of the new system. You will also need to alter the source and destination paths to fit your requirements within your new system.

FRE itself will contains 6 sub packages. 
|-- fre-commands
|   |-- bin
|   |-- lib
|   |-- etc
|   |-- site
|-- fre-transfer
|   |--
|-- fre-nctools
|   |-- 
|-- fre-curator
|   |--
|-- fre-analysis
|   |-- 
|-- fre-xml
|   |--

The fre-commands/site/ stores the settings for the current environment. All new machines will have an added directory here. Site( ) is a function used in applications found with in the bin/ directory. When called, it will return the prefix of the domain name. For example, if you are ncrc.gov then the site returned from Site( ) is ncrc. This will be used as a sub directory name for configuration files. Within that directory resides fre properties, runscripts and .mk files.

Withing the fre.properties script, you will need to change all directories to what is relative to your platform. For example the line FRE.directory.work.default=/path/ should be directed to a fast scratch drive within your platform and the line FRE.directory.archive.default=/path/$(name)/$(target) should point to a tape for long term storage. These are things that will vary between platforms and must be configured properly.

The hsm*.mk files sit in fre-transfer/site/ but the hsm tools expect them in fre-transfer/site . This currently needs to be linked by hand.

The next file found within your new domain directory is a runscript. This is a template used by all models. frerun takes this template and combines it with data from the XML file and generates a runscript. This file could be very different on a range of platforms.

The .mk files are for compilers on a system. They contain the settings used by fremake to build a model.

User contributed scripts/modifications

Any additional features, scripts or modifications produced by FRE users will be added here. They will be organized by topic. The user who contributed these additional features will be labeled with parenthesis.

freconvert

There is a modification of freconvert that prints all the dataSource files in the xml that is being converted to fre4. (Niki Zadeh) 
cvs up -r listDataSourceFiles_nnz freconvert

Warnings and caveats

  • Please do not rely on $FREROOT in your XML files. It is deprecated because system modules now control the root location of the FRE installation. Use $(rootDir) or another appropriate directory path.
  • Do cd $work/INPUT at the beginning of the csh sections in the xml file if the commands need to operate in the INPUT directory.
  • Note that the utility FRE uses to copy data (hsmget) will make a directory for the contents of an archive file, stripping off the extensions: tar, cpio, nc. Therefore, this utility may have undesirable results if files called "foo.tar", "foo.cpio" and "foo.nc.cpio" are in the same directory. If your data follows this pattern, please rename your files to avoid the conflict. Please note that the order of resolving archives from high to low priority is: tar, nc.tar, cpio, nc.cpio.
  • Some csh sections may need to be modified if they attempt to modify input data files in INPUT. This happens because these files may be a unix hard link to what should be a fixed copy of the input data. If your csh fits this scenario, please modify it to operate on a copy of the input data file.
  • qa -n 30 will give a wider column for the experiment name.
  • Check your csh sections for full paths to tools like "combine-ncc". In order to get the latest version provided by the FRE environment modules, do not use a full path for the following tools: combine-ncc, decompress-ncc, fregrid, fregrid_parallel, list_ncvars.csh, mppnccombine, nccmp, ncexists, plevel.sh, split_ncvars.csh, stacksize.csh, timavg.csh, varlist.csh, container.csh, list_months.csh, and split_into_months.csh.

If fremake -t debug gives errors

Errors look like: 
ld: fms.x: short data segment overflowed (0x5a10f8 >= 0x400000)
ld: can't relax section: No such file or directory
make: *** [fms_c48L48_am3p9_riga_201006.x] Error 1

Compiling without the debug flag works, but when using the debug mode fremake -t debug, it gives errors as shown above.

This is a memory overflow problem. A work around is to reduce the complexity of the compiler options you are using to compile.
  • 1) Go to the directory where your debug executable is.
  • 2) Locate the modules that you load in the compile script.
  • 3) Load these modules.
  • 4) do an "ls -lSr" in the directory and find the largest .o file.* Typically this will be mpp_domains.o
  • 5) rm that file
  • 6) run the compile script and when it starts to compile that file, type Ctrl-C.
  • 7) Cut and paste the command that the script gives you as a compile line and remove "-check -check noarg_temp_created -check nopointer " from the line. Hit enter.
  • 8) You may want to do this with a few (up to 5) of the largest .o files.
  • 9) Run the compile script.
  • 10) If it fails then return to step 5 and repeat as needed.