org.knime.base.node.preproc.groupby

Class GroupByNodeModel

java.lang.Object

org.knime.core.node.NodeModel

org.knime.base.node.preproc.groupby.GroupByNodeModel

Direct Known Subclasses:

Pivot2NodeModel

public class GroupByNodeModel
extends NodeModel

The NodeModel implementation of the group by node which uses the GroupByTable class implementations to create the resulting table.

Author:

Tobias Koetter, University of Konstanz

Field Summary

Fields

Modifier and Type	Field and Description
protected static String	CFG_COLUMN_NAME_POLICY Configuration key for the aggregation column name policy.
(package private) static String	CFG_DATA_TYPE_AGGREGATORS Configuration key for the data type based aggregation methods.
protected static String	CFG_ENABLE_HILITE Configuration key for the enable hilite option.
protected static String	CFG_GROUP_BY_COLUMNS Configuration key of the selected group by columns.
protected static String	CFG_IN_MEMORY Configuration key for the in memory option.
protected static String	CFG_MAX_UNIQUE_VALUES Configuration key for the maximum none numerical values.
(package private) static String	CFG_PATTERN_AGGREGATORS Configuration key for the pattern based aggregation methods.
protected static String	CFG_RETAIN_ORDER Configuration key for the retain order option.
protected static String	CFG_SORT_IN_MEMORY Deprecated.
protected static String	CFG_VALUE_DELIMITER Configuration key for the value delimiter option.

Constructor Summary

Constructors

Constructor and Description
GroupByNodeModel() Creates a new group by model with one in- and one out-port.
GroupByNodeModel(int ins, int outs) Creates a new group by model.

Method Summary

Modifier and Type	Method and Description
static void	checkDuplicateAggregators(ColumnNamePolicy namePolicy, List<ColumnAggregator> aggregators)
protected static List<ColumnAggregator>	compGetColumnMethods(DataTableSpec spec, List<String> excludeCols, ConfigRO config) Compatibility method used for compatibility to versions prior Knime 2.0.
protected static ColumnNamePolicy	compGetColumnNamePolicy(NodeSettingsRO settings) Compatibility method used for compatibility to versions prior Knime 2.0.
protected DataTableSpec[]	configure(PortObjectSpec[] inSpecs) Configure method for general port types.
protected GlobalSettings	createGlobalSettings(ExecutionContext exec, BufferedDataTable table, List<String> groupByCols, int maxUniqueVals) Creates the GlobalSettings object that is passed to all AggregationMethods.
protected DataTableSpec	createGroupBySpec(DataTableSpec origSpec, List<String> groupByCols) Generate table spec based on the input spec and the selected columns for grouping.
protected GroupByTable	createGroupByTable(ExecutionContext exec, BufferedDataTable table, List<String> groupByCols) Create group-by table.
protected GroupByTable	createGroupByTable(ExecutionContext exec, BufferedDataTable table, List<String> groupByCols, boolean inMemory, boolean sortInMemory, boolean retainOrder, List<ColumnAggregator> aggregators) Deprecated. sortInMemory is no longer required
protected GroupByTable	createGroupByTable(ExecutionContext exec, BufferedDataTable table, List<String> groupByCols, boolean inMemory, boolean retainOrder, List<ColumnAggregator> aggregators) Create group-by table.
(package private) static SettingsModelInteger	createVersionModel()
protected PortObject[]	execute(PortObject[] inData, ExecutionContext exec) Execute method for general port types.
static List<ColumnAggregator>	getAggregators(DataTableSpec inputSpec, Collection<String> groupColumns, List<ColumnAggregator> columnAggregators, Collection<PatternAggregator> patternAggregators, Collection<DataTypeAggregator> dataTypeAggregators, List<ColumnAggregator> invalidColAggrs) Creates a List with all ColumnAggregators to use based on the given input settings.
protected List<ColumnAggregator>	getColumnAggregators()
protected ColumnNamePolicy	getColumnNamePolicy()
protected String	getDefaultValueDelimiter()
protected static Collection<String>	getExcludeList(DataTableSpec origSpec, List<String> columns) Determine list of column not present in the original spec.
protected List<String>	getGroupByColumns() Returns list of columns selected for group-by operation.
protected HiLiteHandler	getOutHiLiteHandler(int outIndex) Returns the HiLiteHandler for the given output index.
protected void	inMemoryChanged() Deprecated. obsolete to be notified when consistent settings are loaded into the model (since 2.6)
protected boolean	isProcessInMemory()
protected boolean	isRetainOrder()
protected boolean	isSortInMemory() Deprecated. sort in memory is no longer required
protected void	loadInternals(File nodeInternDir, ExecutionMonitor exec) Load internals into the derived NodeModel.
protected void	loadValidatedSettingsFrom(NodeSettingsRO settings) Sets new settings from the passed object in the model.
protected void	reset() Override this function in the derived model and reset your NodeModel.
protected void	saveInternals(File nodeInternDir, ExecutionMonitor exec) Save internals of the derived NodeModel.
protected void	saveSettingsTo(NodeSettingsWO settings) Adds to the given NodeSettings the model specific settings.
protected void	setHiliteMapping(DefaultHiLiteMapper mapper) Applies a new mapping to the hilite translator.
protected void	setInHiLiteHandler(int inIndex, HiLiteHandler hiLiteHdl) This implementation is empty.
protected void	validateSettings(NodeSettingsRO settings) Validates the settings in the passed NodeSettings object.

Methods inherited from class org.knime.core.node.NodeModel

addWarningListener, computeFinalOutputSpecs, configure, continueLoop, createInitialStreamableOperatorInternals, createMergeOperator, createStreamableOperator, execute, finishStreamableExecution, getAvailableFlowVariables, getAvailableInputFlowVariables, getCredentialsProvider, getInHiLiteHandler, getInPortType, getInputPortRoles, getInteractiveNodeView, getLogger, getLoopEndNode, getLoopStartNode, getNrInPorts, getNrOutPorts, getOutPortType, getOutputPortRoles, getWarningMessage, iterate, notifyViews, notifyWarningListeners, onDispose, peekFlowVariableDouble, peekFlowVariableInt, peekFlowVariableString, pushFlowVariableDouble, pushFlowVariableInt, pushFlowVariableString, removeWarningListener, resetAndConfigureLoopBody, setWarningMessage, stateChanged

Methods inherited from class java.lang.Object

clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

Field Detail

CFG_GROUP_BY_COLUMNS

protected static final String CFG_GROUP_BY_COLUMNS

Configuration key of the selected group by columns.

See Also:

Constant Field Values

CFG_MAX_UNIQUE_VALUES

protected static final String CFG_MAX_UNIQUE_VALUES

Configuration key for the maximum none numerical values.

See Also:

Constant Field Values

CFG_ENABLE_HILITE

protected static final String CFG_ENABLE_HILITE

Configuration key for the enable hilite option.

See Also:

Constant Field Values

CFG_SORT_IN_MEMORY

@Deprecated
protected static final String CFG_SORT_IN_MEMORY

Deprecated.

This setting was used prior KNIME 2.6.

See Also:

Constant Field Values

CFG_RETAIN_ORDER

protected static final String CFG_RETAIN_ORDER

Configuration key for the retain order option.

See Also:

Constant Field Values

CFG_IN_MEMORY

protected static final String CFG_IN_MEMORY

Configuration key for the in memory option.

See Also:

Constant Field Values

CFG_COLUMN_NAME_POLICY

protected static final String CFG_COLUMN_NAME_POLICY

Configuration key for the aggregation column name policy.

See Also:

Constant Field Values

CFG_VALUE_DELIMITER

protected static final String CFG_VALUE_DELIMITER

Configuration key for the value delimiter option.

See Also:

Constant Field Values

CFG_DATA_TYPE_AGGREGATORS

static final String CFG_DATA_TYPE_AGGREGATORS

Configuration key for the data type based aggregation methods.

See Also:

Constant Field Values

CFG_PATTERN_AGGREGATORS

static final String CFG_PATTERN_AGGREGATORS

Configuration key for the pattern based aggregation methods.

See Also:

Constant Field Values

Constructor Detail

GroupByNodeModel

public GroupByNodeModel()

Creates a new group by model with one in- and one out-port.

GroupByNodeModel

public GroupByNodeModel(int ins,
                        int outs)

Creates a new group by model.

Parameters:

ins - number of data input ports

outs - number of data output ports

Method Detail

createVersionModel

static SettingsModelInteger createVersionModel()

Returns:

version model

inMemoryChanged

@Deprecated
protected void inMemoryChanged()

Deprecated. obsolete to be notified when consistent settings are loaded into the model (since 2.6)

Call this method if the process in memory flag has changed.

loadInternals

protected void loadInternals(File nodeInternDir,
                             ExecutionMonitor exec)
                      throws IOException

Load internals into the derived NodeModel. This method is only called if the Node was executed. Read all your internal structures from the given file directory to create your internal data structure which is necessary to provide all node functionalities after the workflow is loaded, e.g. view content and/or hilite mapping.

Specified by:

loadInternals in class NodeModel

Parameters:

nodeInternDir - The directory to read from.

exec - Used to report progress and to cancel the load process.

Throws:

IOException - If an error occurs during reading from this dir.

See Also:

NodeModel.saveInternals(File,ExecutionMonitor)

saveInternals

protected void saveInternals(File nodeInternDir,
                             ExecutionMonitor exec)
                      throws IOException

Save internals of the derived NodeModel. This method is only called if the Node is executed. Write all your internal structures into the given file directory which are necessary to recreate this model when the workflow is loaded, e.g. view content and/or hilite mapping.

Specified by:

saveInternals in class NodeModel

Parameters:

nodeInternDir - The directory to write into.

exec - Used to report progress and to cancel the save process.

Throws:

IOException - If an error occurs during writing to this dir.

See Also:

NodeModel.loadInternals(File,ExecutionMonitor)

saveSettingsTo

protected void saveSettingsTo(NodeSettingsWO settings)

Adds to the given NodeSettings the model specific settings. The settings don't need to be complete or consistent. If, right after startup, no valid settings are available this method can write either nothing or invalid settings.

Method is called by the Node if the current settings need to be saved or transfered to the node's dialog.

Specified by:

saveSettingsTo in class NodeModel

Parameters:

settings - The object to write settings into.

See Also:

NodeModel.loadValidatedSettingsFrom(NodeSettingsRO), NodeModel.validateSettings(NodeSettingsRO)

validateSettings

protected void validateSettings(NodeSettingsRO settings)
                         throws InvalidSettingsException

Validates the settings in the passed NodeSettings object. The specified settings should be checked for completeness and consistency. It must be possible to load a settings object validated here without any exception in the #loadValidatedSettings(NodeSettings) method. The method must not change the current settings in the model - it is supposed to just check them. If some settings are missing, invalid, inconsistent, or just not right throw an exception with a message useful to the user.

Specified by:

validateSettings in class NodeModel

Parameters:

settings - The settings to validate.

Throws:

InvalidSettingsException - If the validation of the settings failed.

See Also:

NodeModel.saveSettingsTo(NodeSettingsWO), NodeModel.loadValidatedSettingsFrom(NodeSettingsRO)

checkDuplicateAggregators

public static void checkDuplicateAggregators(ColumnNamePolicy namePolicy,
                                             List<ColumnAggregator> aggregators)
                                      throws IllegalArgumentException

Parameters:

namePolicy - ColumnNamePolicy to use

aggregators - List of ColumnAggregator to check

Throws:

IllegalArgumentException - if the aggregators contain a duplicate

Since:

2.10

loadValidatedSettingsFrom

protected void loadValidatedSettingsFrom(NodeSettingsRO settings)
                                  throws InvalidSettingsException

Sets new settings from the passed object in the model. You can safely assume that the object passed has been successfully validated by the #validateSettings(NodeSettings) method. The model must set its internal configuration according to the settings object passed.

Specified by:

loadValidatedSettingsFrom in class NodeModel

Parameters:

settings - The settings to read.

Throws:

InvalidSettingsException - If a property is not available.

See Also:

NodeModel.saveSettingsTo(NodeSettingsWO), NodeModel.validateSettings(NodeSettingsRO)

setHiliteMapping

protected final void setHiliteMapping(DefaultHiLiteMapper mapper)

Applies a new mapping to the hilite translator.

Parameters:

mapper - new hilite mapping, or null

reset

protected void reset()

Override this function in the derived model and reset your NodeModel. All components should unregister themselves from any observables (at least from the hilite handler right now). All internally stored data structures should be released. User settings should not be deleted/reset though.

Specified by:

reset in class NodeModel

setInHiLiteHandler

protected void setInHiLiteHandler(int inIndex,
                                  HiLiteHandler hiLiteHdl)

This implementation is empty. Subclasses may override this method in order to be informed when the hilite handler changes at the inport, e.g. when the node (or an preceding node) is newly connected.

Overrides:

setInHiLiteHandler in class NodeModel

Parameters:

inIndex - The index of the input.

hiLiteHdl - The HiLiteHandler at input index. May be null when not available, i.e. not properly connected.

getOutHiLiteHandler

protected HiLiteHandler getOutHiLiteHandler(int outIndex)

Returns the HiLiteHandler for the given output index. This default implementation simply passes on the handler of input port 0 or generates a new one if this node has no inputs.

This method is intended to be overridden

Overrides:

getOutHiLiteHandler in class NodeModel

Parameters:

outIndex - The output index.

Returns:

HiLiteHandler for the given output port.

configure

protected DataTableSpec[] configure(PortObjectSpec[] inSpecs)
                             throws InvalidSettingsException

Configure method for general port types. The argument specs represent the input object specs and are guaranteed to be subclasses of the PortObjectSpecs that are defined through the PortTypes given in the constructor unless this model is an InactiveBranchConsumer (most nodes are not). Similarly, the returned output specs need to comply with their port types spec class (otherwise an error is reported by the framework). They may also be null (out spec not known at time of configuration) or inactive (output and downstream nodes are inactive).

For a general description of the configure method refer to the description of the specialized NodeModel.configure(DataTableSpec[]) methods as it addresses more use cases.

Overrides:

configure in class NodeModel

Parameters:

inSpecs - The input data table specs. Items of the array could be null if no spec is available from the corresponding input port (i.e. not connected or upstream node does not produce an output spec). If a port is of type BufferedDataTable.TYPE and no spec is available the framework will replace null by an empty DataTableSpec (no columns) unless the port is marked as optional as per constructor.

Returns:

The output objects specs or null.

Throws:

InvalidSettingsException - If this node can't be configured.

createGroupBySpec

protected final DataTableSpec createGroupBySpec(DataTableSpec origSpec,
                                                List<String> groupByCols)
                                         throws InvalidSettingsException

Generate table spec based on the input spec and the selected columns for grouping.

Parameters:

origSpec - original input spec

groupByCols - group-by columns

Returns:

a new table spec containing the group-by and aggregation columns

Throws:

InvalidSettingsException - if the group-by can't by generated due to invalid settings

getExcludeList

protected static Collection<String> getExcludeList(DataTableSpec origSpec,
                                                   List<String> columns)

Determine list of column not present in the original spec.

Parameters:

origSpec - original spec given at the in-port of this node

columns - to check against input spec

Returns:

a list of columns not present in input spec

Since:

2.8

getGroupByColumns

protected final List<String> getGroupByColumns()

Returns list of columns selected for group-by operation.

Returns:

group-by columns

execute

protected PortObject[] execute(PortObject[] inData,
                               ExecutionContext exec)
                        throws Exception

Execute method for general port types. The argument inObjects represent the input objects and the returned array represents the output objects. The elements in the argument array are generally guaranteed to be not null and subclasses of the PortObject classes that are defined through the PortTypes given in the constructor. Similarly, the returned output objects need to comply with their port types object class (otherwise an error is reported by the framework) and must not be null. There are few exceptions to these rules:

• Nodes with optional inputs (as specified in the constructor) may find null elements in the array.
• Node that implement InactiveBranchConsumer may find instances of InactiveBranchPortObject in case the corresponding input is inactive.
• All nodes are allowed to return InactiveBranchPortObject elements in case the output should be inactivated.
• Loop end nodes may return null in case they restart the loop by setting the corresponding flags.

For a general description of the execute method refer to the description of the specialized NodeModel.execute(BufferedDataTable[], ExecutionContext) methods as it addresses more use cases.

Overrides:

execute in class NodeModel

Parameters:

inData - The input objects.

exec - For BufferedDataTable creation and progress.

Returns:

The output objects.

Throws:

Exception - If the node execution fails for any reason.

createGroupByTable

protected final GroupByTable createGroupByTable(ExecutionContext exec,
                                                BufferedDataTable table,
                                                List<String> groupByCols)
                                         throws CanceledExecutionException

Create group-by table.

Parameters:

exec - execution context

table - input table to group

groupByCols - column selected for group-by operation

Returns:

table with group and aggregation columns

Throws:

CanceledExecutionException - if the group-by table generation was canceled externally

createGroupByTable

protected final GroupByTable createGroupByTable(ExecutionContext exec,
                                                BufferedDataTable table,
                                                List<String> groupByCols,
                                                boolean inMemory,
                                                boolean retainOrder,
                                                List<ColumnAggregator> aggregators)
                                         throws CanceledExecutionException

Create group-by table.

Parameters:

exec - execution context

table - input table to group

groupByCols - column selected for group-by operation

inMemory - keep data in memory

retainOrder - reconstructs original data order

aggregators - column aggregation to use

Returns:

table with group and aggregation columns

Throws:

CanceledExecutionException - if the group-by table generation was canceled externally

Since:

2.6

createGroupByTable

@Deprecated
protected final GroupByTable createGroupByTable(ExecutionContext exec,
                                                            BufferedDataTable table,
                                                            List<String> groupByCols,
                                                            boolean inMemory,
                                                            boolean sortInMemory,
                                                            boolean retainOrder,
                                                            List<ColumnAggregator> aggregators)
                                                     throws CanceledExecutionException

Deprecated. sortInMemory is no longer required

Create group-by table.

Parameters:

exec - execution context

table - input table to group

groupByCols - column selected for group-by operation

inMemory - keep data in memory

sortInMemory - does sorting in memory

retainOrder - reconstructs original data order

aggregators - column aggregation to use

Returns:

table with group and aggregation columns

Throws:

CanceledExecutionException - if the group-by table generation was canceled externally

See Also:

createGroupByTable(ExecutionContext, BufferedDataTable, List, boolean, boolean, List)

createGlobalSettings

protected GlobalSettings createGlobalSettings(ExecutionContext exec,
                                              BufferedDataTable table,
                                              List<String> groupByCols,
                                              int maxUniqueVals)

Creates the GlobalSettings object that is passed to all AggregationMethods.

Parameters:

exec - the ExecutionContext

table - the BufferedDataTable

groupByCols - the names of the columns to group by

maxUniqueVals - the maximum number of unique values per group

Returns:

the GlobalSettings object to use

Since:

2.6

getDefaultValueDelimiter

protected String getDefaultValueDelimiter()

Returns:

the default value delimiter

Since:

2.10

isRetainOrder

protected boolean isRetainOrder()

Returns:

true if the row order should be retained

isProcessInMemory

protected boolean isProcessInMemory()

Returns:

true if all operations should be processed in memory

isSortInMemory

@Deprecated
protected boolean isSortInMemory()

Deprecated. sort in memory is no longer required

Returns:

true if any sorting should be performed in memory

getColumnAggregators

protected List<ColumnAggregator> getColumnAggregators()

Returns:

list of column aggregator methods

getColumnNamePolicy

protected ColumnNamePolicy getColumnNamePolicy()

Returns:

column name policy used to create resulting pivot columns

compGetColumnNamePolicy

protected static ColumnNamePolicy compGetColumnNamePolicy(NodeSettingsRO settings)

Compatibility method used for compatibility to versions prior Knime 2.0. Helper method to get the ColumnNamePolicy for the old node settings.

Parameters:

settings - the settings to read the old column name policy from

Returns:

the ColumnNamePolicy equivalent to the old setting

compGetColumnMethods

protected static List<ColumnAggregator> compGetColumnMethods(DataTableSpec spec,
                                                             List<String> excludeCols,
                                                             ConfigRO config)

Compatibility method used for compatibility to versions prior Knime 2.0. Helper method to get the aggregation methods for the old node settings.

Parameters:

spec - the input DataTableSpec

excludeCols - the columns that should be excluded from the aggregation columns

config - the config object to read from

Returns:

the ColumnAggregators

getAggregators

public static List<ColumnAggregator> getAggregators(DataTableSpec inputSpec,
                                                    Collection<String> groupColumns,
                                                    List<ColumnAggregator> columnAggregators,
                                                    Collection<PatternAggregator> patternAggregators,
                                                    Collection<DataTypeAggregator> dataTypeAggregators,
                                                    List<ColumnAggregator> invalidColAggrs)

Creates a List with all ColumnAggregators to use based on the given input settings. Columns are only added once for the different aggregator types in the order they are added to the function e.g. all column that are handled by one of the given ColumnAggregator are ignored by the pattern and data type based aggregator all columns that are handled by one of the pattern based aggregators is ignored by the data type based aggregators.

Parameters:

inputSpec - the DataTableSpec of the input table

groupColumns - the columns to group by

columnAggregators - the manually added ColumnAggregators

patternAggregators - the PatternAggregators

dataTypeAggregators - the DataTypeAggregators

invalidColAggrs - empty List that is filled with the invalid column aggregators can be null

Returns:

the list of all ColumnAggregators to use based on the given aggregator

Since:

2.11