MS RFC 17: Dynamic Allocation of layers, styles, classes and symbols

Date:2006/05/12 Author:Frank Warmerdam, with edits from Daniel Morissette and Umberto Nicoletti Contact:warmerdam at pobox.com, dmorissette at mapgears.com, umberto.nicoletti at gmail.com Last Edited:2007/07/18 Status:Adopted (2007/07/18) - Implementation completed (2007/07/23) Version:MapServer 5.0 Id:$Id: ms-rfc-17.txt 8278 2008-12-23 21:34:31Z hobu $

Purpose

Modify the MapServer core libraries so that lists of layers, classes and styles are dynamic, not fixed to compile limits MS_MAXCLASSES, MS_MAXSTYLES, MS_MAXLAYERS and MS_MAXSYMBOLS.

MS_MAXSYMBOLS

Change symbolSetObj so that this:

int numsymbols;
symbolObj symbol[MS_MAXSYMBOLS];

becomes:

int numsymbols;
int maxsymbols;
symbolObj **symbol;

Add the following function to ensure there is at least one free entry in the symbol array in the symbolSetObj. This function will only grow the allocated array of pointers if needed (if maxsymbols == numsymbols) and will then allocate a new symbolObj if symbol[numsymbols] == NULL. The new entries in the array will be set to NULL and the new symbolObj[numsymbols] will be all set to zero bytes.

symbolObj *msGrowSymbolSet( symbolSetObj * );

• Modify all places that add new symbols to call msGrowSymbolsSet() to ensure there is space. These locations can be fairly easily identified by greping on MS_MAXSYMBOLS.
• Modify all places that access symbols by index to use the proper way to reference symbols in the new array, possibly using a GET_SYMBOL() macro similar to the GET_LAYER() added in MS-RFC-24 for layers.
• mapsymbol.c: Modify msInitSymbolSet() to initially setup the symbol set with an array of just one symbol.
• There should be no swig MapScript changes required as it already uses msAppendSymbol().

MS_MAXLAYERS

• map.h: Since MS RFC 24: Mapscript memory management already converted the array of layers in a mapObj to an array of pointers, we only need to add “int maxlayers;” to mapObj to indicate the current allocation size of layers array. Note that this also determines the size of the mapObj layerorder array.
• mapdraw.c: Change msDrawMap() to use map->numlayers in place of current MS_MAXLAYERS for asOWSReqInfo array.
• mapobject.c: Add msGrowMapLayers(mapObj*) function to ensure there is room for at least one more layer on the map. This grows layers, and layerorder arrays.
• mapobject.c: modify msInsertLayer() and MapScript layer constructor code (see last item) to use msGrowMapLayers().
• mapfile.c: set maxlayers to MS_MAXLAYERS and allocate layers accordingly in initMap(). Use msGrowMapLayers() before calling loadLayer() and loadLayerValue() when parsing.
• mappluginlayer.c: make dynamic. I’m not exactly sure what there is a static factory with entries dimensioned by the number of layers.
• It looks like swig MapScript already has an mapObj.insertLayer() method using msInsertLayer() which should be safe. Do MapScript applications sometimes just update the layers and numlayers directly? Answer: all layer manipulation is done by the layer constructor (when a not null mapObj is passed as only argument) or by msInsertLayer. Must modify the layer constructor as well.

MS_MAXCLASSES

• map.h: Add maxclasses field in layerObj (RFC-24 already converted the array of classes in a layerObj to an array of pointers).
• layerobject.c: Modify msInsertClass() and MapScript constructor code (like for layers) to use msGrowLayerClasses().
• layerobject.c: Add msGrowLayerClasses() to ensure there is at least one extra class in the classes list.
• mapdraw.c: colorbuffer and mindistancebuffer in msDrawQueryLayer() will need to be dynamically sized and allocated on the heap.
• mapfile.c: initLayer() will need to initialize maxclasses to MS_MAXCLASSES, and allocate class list accordingly.
• mapogcsld.c: modify to use msGrowLayerClasses() instead of checking limit.

MS_MAXSTYLES

• map.h: add maxstyles field to classObj (RFC-24 already converted the array of styles in a classObj to an array of pointers).
• mapfile.c: set maxstyles to MS_MAXSTYLES and allocate accordingly in initClass(). Use msGrowClassStyles() as apppriate.
• classoject.c: Add msGrowClassStyles() to ensure there is an unused class.
• classobject.c: Use msGrowClassStyles() in msInsertStyle() and in the style constructor.

Files and objects affected

map.h
mapfile.c
mapsymbol.c
mapdraw.c
mapobject.c
mappluginlayer.c
layerobject.c
mapogcsld.c
classobject.c
mapscript/swiginc/layer.i
mapscript/swiginc/class.i
mapscript/swiginc/style.i

Backwards compatibility issues

None.

Test suite

Python unit test entries will be added to exceed the builtin maximums for all of layers, classes, styles and symbols. An msautotest entry with a large number of classes will also be added.

Bug ID

• 302: https://trac.osgeo.org/mapserver/ticket/302

Voting history

Vote completed on 2007-07-19:

+1 from DanielM, SteveL, FrankW, SteveW, UmbertoN, TamasS, AssefaY

Comments/Questions from the review period

• Has the following issue already been dealt with by RFC-24 since the code already allocates less than MS_MAX_LAYERS in initMap()? Did RFC-24 make it a formal requirement to use the proper insert/add methods to add layers, styles and classes?

Answer: the implementation adopted with MS RFC 24: Mapscript memory management tries to save memory by only allocating the number of layers that are effectively needed. A map with 5 layers will allocate exactly 5 layers, a map with 50 layers will allocate 50, and so on up to the hard-coded limit of MS_MAX_LAYERS. This is also true for classes and style.

The arrays of pointers are for obvious reasons always allocated to MS_MAX_LAYERS size, but memory usage is reduced anyway because arrays of pointers are much smaller that arrays of structs. This represents a substantial change from before, when MS_MAX_LAYERS blank layers where always allocated causing a sure waste of memory in small maps.

MS RFC 17: Dynamic Allocation of layers, styles, classes and symbols should then invoke the various msGrow*() methods to grow the arrays when numlayers == maxlayers-1. I’d suggest to grow the array to a sensible size (like half of the current size) as the impact on memory allocation is going to be mitigated by the dynamic allocation approach introduced by MS RFC 24: Mapscript memory management

MS RFC 24: Mapscript memory management did not make a formal requirement to use the proper insert methods as those and the object constructors are the only way to add a layer, class or style to a map. I do not know if this is also true for symbols.

• “Because MapScript application often explicitly initialize “blank” layers, classes and styles directly, and then increment the count, we can’t depend on all access going through the proper insert/add methods. For this reason we preserve the old MS_MAX values to establish the initial allocation. This should mean that existing applications will continue to work at some cost in unused memory. But well behaved MapScript applications using insert methods to increase sizes will be able to grow beyond the initial allocation.”

I’m not aware of any MapScript way to bypass the insert or the constructors (i.e. with direct manipulation of the arrays). It there was such way (which I doubt) I suggest that we forbid it by explicitly changing the MS_MAX_* names and making required fields immutable in swig.

• Comment from Tamas: The mappluginlayer stuff should be concretized a bit. That static repository of the vtables would prevent from loading the same library twice. That array should also be allocated dynamically since I don’t think we will ever have MS_MAXLAYERS number of different plugin layers.