Mesh Refinement

ERF also allows both dynamic and static mesh refinement with sub-cycling in time at finer levels of refinement. Arbitrary integer refinement ratios are allowed although typically ratios of 2, 3 or 4 are used; refinement can also be anisotropic, allowing refinement in one coordinate direction but not another.

We utilize two-way coupling, in which the coarse solution is used to provide boundary conditions for the fine solution and the fine solution is averaged down onto the coarser level. In addition, we reflux all advected scalars to ensure conservation. For coarse-to-fine communication we provide “ghost cell” data for cell-centered data and tangential momentum components to the fine level by interpolating in space and time outside the region covered by the fine level. We also interpolate the normal momentum from coarse to fine on the coarse-fine interface itself; this ensures mass conservation since the normal momentum is in fact the flux for the density field. In order to ensure that the fine momentum on the coarse-fine boundary stays consistent with the interpolated coarse values throughout a fine timestep, we also interpolate the source term for the normal momentum on the coarse-fine interface.

With acoustic substepping, interpolating linearly in time between the coarse level’s old and new momenta is not enough: over a time step the density flux through a face is the time integral of the momentum on that face over the acoustic substeps, and the coarse level’s integral uses what the momentum actually did, which acoustic waves crossing the face make far from linear. So, without terrain or with terrain that does not move, the coarse level records the flux through each face under the fine level’s coarse-fine faces over each acoustic substep of its last RK stage. On its coarse-fine faces the fine level then uses the average of those fluxes over each of its own substeps, interpolated conservatively in space, and in the vertical sets the normal momentum (with terrain, the contravariant flux Omega) at the end of each substep from the same record. The fine fluxes through the interface then add up, over the coarse step, to the coarse ones, and the total mass is conserved to round-off. Setting erf.cf_flux_history = false (the default is true) goes back to interpolating the coarse momentum linearly in time on these faces; it is not done with moving terrain. This is done under one-way coupling as well as two-way, so that the fine level sees the same fluxes through its coarse-fine faces whichever coupling is used.

When using the anelastic approximation, this ensures that the computation of the updates to the fine momentum do not use any values of the perturbational pressure from the coarser level since the perturbational pressure is not synchronized between levels.

Note that any tagged region will be covered by one or more boxes. The user may specify the refinement criteria and/or region to be covered, but not the decomposition of the region into individual grids.

See the Gridding section of the AMReX documentation for details of how individual grids are created.

By default the boxes covering a tagged region span the full vertical extent of that region, i.e. the grids are not decomposed in the z direction; see Avoiding Decomposition in the Vertical Direction for the parameters that control this.

Note

Three different things are easily conflated when talking about the vertical direction, and they are controlled by three different sets of parameters.

Refining in z means changing the grid spacing: the finer level has \(dz_{fine} = dz_{coarse} / r_z\), where \(r_z\) is the refinement ratio in the z direction. Creating a level greater than 0 over some region does not by itself imply that dz changes there. We nevertheless say that such a region is “refined”, so the word is used for both ideas; whether the vertical spacing actually changes is a separate question, decided only by the refinement ratio. amr.ref_ratio sets one ratio per level which is then applied in all three directions, so using it always refines in z as well. To create finer levels without changing dz, use amr.ref_ratio_vect with 1 in the third slot, for example

amr.max_level = 1
amr.ref_ratio_vect = 2 2 1

which refines by a factor of 2 horizontally while leaving the vertical grid spacing of the fine level equal to that of the coarse level. Initialization from WRF input files supports either choice: a wrfinput file for a nest carries the same eta levels as its parent, so if the refinement ratio in z is greater than 1 then ERF interpolates the file onto the finer vertical grid as it reads it. See Nested Initialization From WRF Input Files.

The vertical extent of the refined region – how much of the depth of the domain the finer level covers – is a property of the region being tagged, not of the refinement ratio. It is set by the refinement box for static refinement (see Refining the Full Depth of the Domain) and by the tagging criterion, optionally buffered in z, for dynamic refinement (see Full-Depth Refinement with Dynamic Tagging). It can also be set once and for all with amr.refine_whole_domain_dir, which makes every level greater than 0 cover the entire domain in a chosen direction regardless of how the cells were tagged; see Refining the Whole Domain in One Direction.

The vertical decomposition of that region into individual grids – whether the region is chopped in z into several boxes so that they can be distributed across processors – is yet another question, controlled by amr.no_box_split_dir, which ERF sets to 2 by default so that no such chopping occurs, and, when that is disabled, by amr.max_grid_size_z and amr.refine_grid_layout_z; see Avoiding Decomposition in the Vertical Direction.

A level can therefore be created with the same dz as its parent, spanning the full depth of the domain, in boxes that are not split vertically – and each of those three statements is arranged by a different parameter.

Static Mesh Refinement

For static refinement, we can control the placement of grids by specifying the low and high extents (in physical space or index space) of each box.

The following example demonstrates how to tag regions for static refinement. In this first example, all cells in the region ((1200,1400,0.)(3000,2400,2048.)) and in the region ((3200,3400,0.)(4000,4000,2048.)) are tagged for one level of refinement.

amr.max_level = 1
amr.ref_ratio = 2

erf.refinement_indicators = box1 box2

erf.box1.in_box_lo = 1200 1400 0
erf.box1.in_box_hi = 3000 2400 2048
erf.box1.max_level = 1

erf.box2.in_box_lo = 3200 3400 0
erf.box2.in_box_hi = 4000 4000 2048
erf.box2.max_level = 1

It is possible to refine the box by multiple levels, and to have different levels of refinement for different boxes. For example, if we set

amr.max_level = 2
amr.ref_ratio = 2 2

erf.refinement_indicators = box1 box2

erf.box1.in_box_lo = 1200 1400 0
erf.box1.in_box_hi = 3000 2400 2048
erf.box1.max_level = 1

erf.box2.in_box_lo = 3200 3400 0
erf.box2.in_box_hi = 4000 4000 2048
erf.box2.max_level = 2

box1 is refined up to level 1, but box2 is refined up to level 2. If no max_level is specified for a refinement indicator, the region is refined up to the maximum level specified in amr.max_level. Error messages are generated to help users adjust the refinements to create a valid box at each level.

We note that instead of specifying the physical extent enclosed, we can instead specify the indices of the bounding box of the refined region in the index space of that fine level. To do this we use in_box_lo_indices and in_box_hi_indices instead of in_box_lo and in_box_hi. If we want to refine the inner region (spanning half the width in each direction) by one level of factor 2 refinement, and the domain has 32x64x8 cells at level 0 covering the domain, then we would set

amr.max_level = 1
amr.ref_ratio = 2

erf.refinement_indicators = box1

erf.box1.in_box_lo_indices = 16 32  4
erf.box1.in_box_hi_indices = 47 95 11
erf.box1.max_level = 1

There is also an option to specify the indices of the bounding box of the refined region in the index space of the coarser level, using in_box_lo_indices_crse and in_box_hi_indices_crse. This is useful when the user has a particular region in mind that they want to refine, and they know the indices of that region on the coarser level but not on the finer level. In this case, the code will automatically adjust the indices to create a valid box at the finer level.

amr.max_level = 1
amr.ref_ratio = 2

erf.refinement_indicators = box1

erf.box1.in_box_lo_indices_crse = 16 32  4
erf.box1.in_box_hi_indices_crse = 47 95 11
erf.box1.max_level = 1

The lo_indices should be divisible by the refinement ratio, and the hi_indices should be one less than a number divisible by the refinement ratio. There are no such requirements for the coarse level indices, since the code will adjust them as needed to create a valid box at the finer level.

Refining the Full Depth of the Domain

It is very common to want a refined region that covers the full depth of the domain, either because a physics package needs entire columns or simply because there is no reason to limit the refinement vertically. Rather than spelling out the vertical extent, the user may specify only the x and y values for any of the three forms of the box specification, in which case ERF fills in the vertical extent with the full extent of the domain:

for in_box_lo / in_box_hi, the vertical extent is set to geometry.prob_lo and geometry.prob_hi in z
for in_box_lo_indices / in_box_hi_indices, it is set to the full range of cells in z in the domain at the level being created
for in_box_lo_indices_crse / in_box_hi_indices_crse, it is set to the full range of cells in z in the domain at the next coarser level

So the first example above could equivalently have been written as

amr.max_level = 1
amr.ref_ratio = 2

erf.refinement_indicators = box1 box2

erf.box1.in_box_lo = 1200 1400
erf.box1.in_box_hi = 3000 2400
erf.box1.max_level = 1

erf.box2.in_box_lo = 3200 3400
erf.box2.in_box_hi = 4000 4000
erf.box2.max_level = 1

if the domain runs from 0 to 2048 in z. This form has the advantage that the inputs file does not have to be edited if the vertical extent of the domain or the number of cells in z is changed. Each of in_box_lo and in_box_hi (and their index-space counterparts) should be given either two or three values, and the lo and hi specifications for a given indicator must have the same number of values.

amr.refine_whole_domain_dir = 2 achieves the same thing from the other end: it makes every fine level span the full depth no matter how the boxes are specified. See Refining the Whole Domain in One Direction.

Some options require a refinement region that spans the full depth of the domain. The PBL models – MYJ, MYNN2.5, MYNN-EDMF, YSU and MRF – compute the height of the boundary layer by working up each column, so every box must go from the bottom of the domain to the top. If a refinement box is specified that does not, ERF prints a message telling the user how to correct the inputs file, then aborts. Using the two-value form above is the simplest way to guarantee this. Similarly, the SHOC PBL model requires that no box be split in the vertical direction; see Avoiding Decomposition in the Vertical Direction.

A refined region that deliberately covers only part of the depth is common in terrain runs; see Refining Only Part of the Depth with Terrain.

The two-value form applies to statically specified boxes. For dynamically created grids there is a separate, blunter mechanism based on the vertical tagging buffer; see Full-Depth Refinement with Dynamic Tagging.

If the vertical extent is given explicitly, note that the number of cells in the z direction should be divisible by the refinement ratio in the vertical direction, and that the specified indices are snapped outward to the nearest indices aligned with the refinement ratio (a message is printed whenever this snapping changes the box).

Refining Only Part of the Depth with Terrain

A nested LES region rarely needs to be refined all the way to the model top, so a refinement box whose vertical extent stops well below geometry.prob_hi in z is a natural thing to want. This is supported with terrain-fitted coordinates for both of the initialization pathways that build a stratified base state:

erf.init_type = input_sounding, with the terrain read from a text file via erf.terrain_file_name; and
erf.init_type = WRFInput, whether each level has its own wrfinput file or only level 0 does.

For example, with a domain 17500 m deep:

amr.max_level      = 1
amr.ref_ratio_vect = 3 3 1

erf.refinement_indicators = box1
erf.box1.max_level = 1
erf.box1.in_box_lo =  70000.   90000.     0.
erf.box1.in_box_hi = 110000.  150000.  6000.

Two things to watch, both specific to how the box is specified:

All three components must be given. The two-value form described above fills the vertical extent with the full depth of the domain, which is the opposite of what is wanted here.
amr.refine_whole_domain_dir must not be set to 2, since that overrides the vertical extent of every refinement box and gives the full depth of the domain; see Refining the Whole Domain in One Direction.
The z values are located on the nominal (undeformed) vertical grid – that is, on zlevels_stag – and not on the terrain-following heights, so the box may not reach the height you expect. This matters most with erf.init_type = WRFInput, where the nominal grid is uniform while the heights the wrfinput file supplies are strongly compressed near the surface. Check the box ERF reports (Saving in 'boxes at level') against the z_phys field in the plotfile.

With erf.init_type = WRFInput, if a file is given for the finer level then the box must be contained in the region that file covers; ERF checks this and aborts with both boxes printed if it is not.

The base state is built independently on each level, against that level’s own heights, and its vertical integration is causal upward – so a level that stops partway up gets exactly the base state it would have had if it had been refined to the model top. See The base state on a refined mesh for that, and Nested Initialization From WRF Input Files for the wrfinput-specific details.

The restrictions are the same for both pathways: the PBL models and the column-integral derived quantities need entire columns and cannot be used on a level whose grids do not span the domain in z. See Nested Initialization From WRF Input Files.

Anelastic runs. A level that stops inside the domain in z has coarse/fine faces at its top and bottom, and the anelastic projection treats them like its side faces: it applies no correction through them, so the flux through them has to come from the coarse level and add up to no net inflow. On a terrain-fitted mesh the flux through a z-face is the contravariant \(\Omega = \rho_0 w - \rho_0 u\, \partial z/\partial x - \rho_0 v\, \partial z/\partial y\). ERF therefore

fills the horizontal momenta just below and above the level from the coarse level, since \(\Omega\) on the level’s lowest and highest faces reads them;
sets \(\Omega\) on the level’s top and bottom faces from the coarse level’s \(\Omega\), interpolated so the fine faces over each coarse face average to it, both in the projection and in the fine-level fills the time step uses, so that theta and the other scalars are carried by the same face fluxes the projection balanced (the coarse \(\rho w\) alone would leave the net inflow in the level as a uniform divergence);
passes gmres_fft only the level’s own layers of \(\Delta z\) for its preconditioner.

All three terrain solvers (erf.terrain_poisson_solver) run such a level. The wall distance erf.wall_dist_type = terrain_height takes the ground heights a level does not hold from the coarser level. With one-way coupling this is all the level needs; two-way coupling of such a level over terrain has not been tested yet.

Refining the Whole Domain in One Direction

The mechanisms above shape the refined region one indicator at a time. There is also a single switch, amr.refine_whole_domain_dir, which requires every level greater than 0 to cover the entire domain in one coordinate direction, no matter where the cells are tagged. It takes the index of that direction – 0 for x, 1 for y, 2 for z – and defaults to -1, which disables it. The setting of interest to almost every ERF workflow is

amr.refine_whole_domain_dir = 2

With this set, tagging a cell behaves as if the entire column of cells through it had been tagged, so the finer level runs from the bottom of the domain to the top wherever it appears. This holds for both ways of tagging:

with a static refinement box, the box still selects the region in x and y, but its vertical extent no longer matters – the fine grids span the full depth even if in_box_lo / in_box_hi asked for only part of it;
with dynamic tagging, the fine grids span the full depth wherever the criterion fires, with no need to know in advance where that will be.

Because the clustering is then carried out in one fewer dimension, amr.grid_eff refers to the fraction of tagged cells in the plane perpendicular to that direction – here, the fraction of tagged columns in the x-y plane.

If splitting in z is allowed – that is, if amr.no_box_split_dir is set to -1 rather than left at its ERF default of 2, which ERF permits only when no level uses implicit acoustic substepping – then the region covering the domain in z may still be chopped in z into several boxes by amr.max_grid_size_z and amr.refine_grid_layout_z: their union always covers the full depth, but an individual box then does not. With the ERF defaults for those parameters (see Avoiding Decomposition in the Vertical Direction) no such chopping occurs, so each box by itself reaches from the bottom of the domain to the top, which is what the PBL models and the column-integral diagnostics require.

A few things to know:

The option applies only to levels greater than 0, i.e. the levels created by tagging; level 0 covers the domain in all three directions in any case.
It is a global setting: it applies to every refinement indicator and every level, so it cannot give one nest the full depth and another only part of it. In particular, do not set it when a partial-depth nest is wanted (see Refining Only Part of the Depth with Terrain).
The check that aborts when a PBL model is used with a refinement box that does not span the domain in z (see Refining the Full Depth of the Domain) tests the box as specified in the inputs file, and does not know that amr.refine_whole_domain_dir would have extended the grids. For the same reason, initialization of a nest from a wrfinput file fills the box registered by the indicator (the one ERF echoes as Saving in 'boxes at level'), not the grids that tagging finally produced. In both cases the box itself should still be given the full depth, which the two-value form of Refining the Full Depth of the Domain does automatically.
On each level, the number of cells in that direction must be divisible by the next finer level’s amr.blocking_factor in that direction divided by the refinement ratio, since that is the factor by which the tags are coarsened before they are clustered. The ERF default amr.blocking_factor of 1 makes this automatic; AMReX otherwise aborts with Domain size not divisible by blocking_factor/ref_ratio in refine_whole_domain_dir.
The option is not supported in builds that use bittree.

Mesh Refinement

Static Mesh Refinement

Refining the Full Depth of the Domain

Refining Only Part of the Depth with Terrain

Moving Refinement Regions

Dynamic Mesh Refinement

Full-Depth Refinement with Dynamic Tagging

Refining the Whole Domain in One Direction