The Paramesh Grid Implementation in flash3 Klaus Weide May 2006 Preliminaries


Download 445 b.
Sana24.06.2017
Hajmi445 b.


The Paramesh Grid Implementation in FLASH3 Klaus Weide May 2006


Preliminaries

  • Scope:

    • The PARAMESH3 (actually now: PARAMESH 4.0) implementation of the Grid unit in FLASH3
    • Ignoring particles (a related but separate topic)
    • Ignoring NO_PERMANENT_GUARDCELLS etc.
    • What users* of FLASH3 need to know about this Grid.
      • *In particular, writers of non-infrastructure code that will work with FLASH3.


Preliminaries

  • A Warning that Shall Be Repeated:

    • This presentation includes discussion of internal structures of the Grid implementation, in order to explain the function of the Grid.
    • Please do not access these structures in code outside the Grid unit. You may get away with it for a while, but we don't want to support it!
    • If you break it, you get to keep both parts.
    • Please do use the official interfaces. For example, use Grid_getBlkIndexLimits instead of constants NXB, NYB, etc.
      • This will keep your code working if the Grid implementation changes!


Basic Computational Unit : Block

  • The grid is composed of blocks

  • Blocks are composed of interior cells and surrounding guard cells

  • Paramesh: All blocks have the same size (number of cells), can have different resolution

  • PM in Flash3: Block size is determined at setup time.



First Look at Paramesh Grid

    • Purpose of the Grid: represent data
      • more on UNK variables etc. below
    • Each block resides on exactly one processor (at a given point in time)
    • Limitations imposed by Paramesh:
      • Same number of cells, guard cell layers
      • Resolution (“Delta”) of a block changes by multiples of 2
      • Resolution of neighbors differs at most by factor of 2


How Blocks are Identified

    • At a given time, a block is globally uniquely identified by a pair (PE, BlockID), where
      • 0 < PE < numprocs
      • 1 < BlockID < MAXBLOCKS
    • Locally, BlockID is sufficient to specify a block
      • User code can't directly access remote blocks anyway
    • Morton Numbers provide another way to identify blocks globally (more later)


How Blocks are Stored

      • Solution data,
      • per-block meta data,
      • tree information (for local blocks!)
  • are stored in F90 arrays declared like this:

    • real, dimension(,,,,MAXBLOCKS) :: UNK
    • real, dimension(,MAXBLOCKS) :: bnd_box
    • integer, dimension(,MAXBLOCKS) :: parent
  • MAXBLOCKS is a hardwired constant (from setup time)

  • “Inactive” (non-leaf) blocks also use storage

  • These structures are internal to the Grid unit and should not be accessed directly by other code.

  • Use the appropriate Grid_something subroutine calls instead!



How Blocks are Woven Together I

    • The Tree (1d: binary tree; 3d: “oct” tree)
      • parent/child relationships
    • Neighbors
      • geometric relationships
    • Morton Ordering


Connecting Blocks: The Tree

    • The Tree (1d: binary tree; 3d: “oct” tree)
      • parent/child relationships
      • three types of blocks:
      • Not all blocks have a parent – there is at least one root node, there may be many!
      • Each block has a refinement level, 1 ≤ lrefine(blockID) ≤ lrefine_max


Connecting Blocks: Neighbors

    • Neighbor relationships
      • neigh array – neighbors across faces
      • surr_blks – surrounding blocks incl. diagonal
        • regenerated by PARAMESH3 from other information
      • “wrap around” at periodic boundaries
    • Combined Tree and Neighbor info
      • gr_gid array
      • combines parent/child/neigh info
      • This is how linkage info is stored in checkpoint files


Connecting Blocks: Morton Order

    • Morton Ordering
      • a way to serialize blocks
      • Morton Function maps blocks to integers, '<' relation between integers => morton order of blocks
      • determines
        • how blocks are ordered in memory in each processor AND
        • how they are distributed across processors
      • Morton Curve: an illustration of the Morton order


How Blocks are Woven Together II

    • Warnings
      • The structures linking blocks should not be accessed directly by user code
      • parent, neigh, etc. may point to remote blocks
      • The implementation of Morton function is internal magic of Paramesh. You don't need to understand it to use FLASH3/Paramesh.
      • User code generally does not need to know where a block's neighbor is. It can rely on Paramesh to fill guard cells from neighbors correctly.


Per-Block Meta-Information

  • Paramesh keeps information associated with each local block:

  • parent, child, surr_blks, neigh – block linkage

  • lrefine, nodetype – tree-related information

  • coord, bnd_box, bsize – physical coordinates

  • There are public accessor interfaces Grid_getThisOrThat to get the parts of this information that may be useful to user code. Use them!



Block Data: Fluid Variables I

  • FLASH3 provides the following kinds of variables:

  • UNK variables – stored in adjacent slots in UNK array for each cell

    • cell-centered (or 'volume-centered')
    • basic hydrodynamic and other physical continuous variables, aka. solution variables, aka. "unknowns".
    • data maintained by PARAMESH
    • UNK contains three kinds of variables, from different Config declarations:
      • General solution variables
      • VARIABLE name #e.g., VARIABLE dens -> DENS_VAR
      • Species mass fractions
      • SPECIES NAME #e.g., SPECIES NI56
      • For technical reasons there is always at least MFRAC_SPEC
      • "mass scalars" - additional passiveley advected variables
      • MASS_SCALAR NAME #e.g., MASS_SCALAR YE


Block Data: Fluid Variables II

  • Additional kinds of variables:

  • GRIDVAR aka. SCRATCH variables

      • GRIDVAR name #e.g., GRIDVAR otmp
    • cell-centered (or face-centered)
    • for temporary use, data is not meant to survive Grid changes
    • data maintained by FLASH3
  • Face variables

      • FACEVAR name #e.g., FACEVAR Mag
    • face-centered
    • currently only used by MHD
    • data maintained by PARAMESH
  • work space – mentioned for completeness

    • maintained by PARAMESH, not currently used by FLASH3
  • Fluxes – mentioned for completeness

      • FLUX name #e.g., FLUX rho
    • maintained by PARAMESH/FLASH3, used in Hydro


Applying Boundary Conditions

  • Specify type of Boundaries in flash.par

    • Inner boundaries are now also possible
    • Initialization from checkpoint file if restart is TRUE
  • Non-default boundary condition handling:

    • physics units access and modify block data
    • Usually, provide custom Grid_applyBCEdge
    • provide custom Grid_applyBCEdgeAllUnkVars when needed
    • Not recommended: provide custom amr_1blk_bcset
    • NoDriver unit call Grid_updateRefinement after each time step
  • Hydrostatic boundary conditions:

    • work in progress
    • Implementation as in FLASH2 default is now available


Life of The Mesh I

  • What happens to the Grid, from start to finish of a calculation

  • Initialization

    • Initialization from scratch if restart is FALSE
    • Initialization from checkpoint file if restart is TRUE
  • Changes during Evolution

    • physics units access and modify block data
    • physics units call Grid_fillGuardCells when they need it
    • Driver unit call Grid_updateRefinement after each time step
  • Data and meta info are periodically stored in checkpoint files.



Life of The Mesh – Initialization from scratch

  • How is the Grid constructed initially?

  • One or more initial blocks are created, to become root block(s) of tree(s) – see runtime parameters nblockx, nblocky, nblockz

  • These are distributed by FLASH3 across processors from the very beginning (“parallel divide domain”), as evenly as possible

  • Some of the initial blocks can now be turned into internal boundary blocks (obstacles), see Simulation_defineDomain

  • PARAMESH magic machinery is let loose on the remaining initial block(s) repeatedly until desired lrefine_max can be reached. It takes care of

    • creating new (finer) blocks as children of coarser parents
    • redistributing blocks across processors (rebalancing)
    • User can influence by
    • setting runtime parameters (lrefine_max, refine_var_n, etc.)
    • supplying custom Grid_markRefineDerefine implementation


Life of The Mesh – Initialization from checkpoint

  • How is the Grid reconstructed ?

  • Checkpoint contains Grid blocks in original (PE, blockID) order (Morton Order)

  • This logically one-dimensional sequence of blocks is chopped up and distributed evenly across available processors.

  • The same applies to meta info (e.g., gr_gid for tree info)

  • Each processor reads and stores only the chunk assigned to it. No processor stores all the global tree data.

  • PARAMESH3 is called to initialize additional data structures.



Life of The Mesh – Evolution

  • How does the Grid change during a simulation ?

  • Physics units access and modify block data

  • Grid_fillGuardCells is called when needed

  • Driver unit calls Grid_updateRefinement periodically

  • PARAMESH magic machinery is let loose on the blocks. It takes care of

    • creating new (finer) blocks as children of coarser parents
    • deleting leaf blocks where reolution is too high
    • redistributing blocks across processors (rebalancing)
    • User can influence by
    • setting runtime parameters (lrefine_max, refine_var_n, etc.)
    • supplying custom Grid_markRefineDerefine implementation


Life of The Mesh – The End

  • How does the Grid save data ?

  • The state of the blocks and meta-info is saved to checkpoint files

  • Blocks end up stored sequentially in global (PE,blockID) order, i.e., Morton Order.

  • Some more on this in IO part of tutorial.




Do'stlaringiz bilan baham:


Ma'lumotlar bazasi mualliflik huquqi bilan himoyalangan ©fayllar.org 2017
ma'muriyatiga murojaat qiling