Skip to content

layer_adaptation

Functions:

Name Description
expand_weights

Expands weights along the given axis by n_dims_to_add.
get_expansion_instructions

Returns a dictionary containing information on how
get_in_out_axes

Returns a dictionary containing information on how a

expand_weights 

expand_weights(
    weights: Tensor,
    axis: int,
    n_dims_to_add: int,
    init_fn: Callable,
    n_subparams: int = 1,
)

Expands weights along the given axis by n_dims_to_add. The expanded weights are created by evenly splitting the original weights into its subparams and appending new weights to them.

Parameters:

Name Type Description Default
weights Tensor

Parameter to be expanded.

 required
axis int

Axis along which to expand the parameter.

 required
n_dims_to_add int

Number of dims to add to each sub-parameter within the parameter.

 required
init_fn Callable

Function to initiate the new weights with.

 required
n_subparams int

Number of sub-parameters contained in the parameter.

 1

Returns:

Type Description
weights_expanded

The expanded weights as a pytorch parameter.
Source code in deep_river/utils/layer_adaptation.py 
129
130
131
132
133
134
135
136
137
138
139
140
141
142
143
144
145
146
147
148
149
150
151
152
153
154
155
156
157
158
159
160
161
162
163
164
165
166
167
168
169
170
171
def expand_weights(
    weights: torch.Tensor,
    axis: int,
    n_dims_to_add: int,
    init_fn: Callable,
    n_subparams: int = 1,
):
    """
    Expands `weights` along the given axis by `n_dims_to_add`.
    The expanded weights are created by evenly splitting the
    original weights into its subparams and appending new weights
    to them.

    Parameters
    ----------
    weights
        Parameter to be expanded.
    axis
        Axis along which to expand the parameter.
    n_dims_to_add
        Number of dims to add to each sub-parameter within the parameter.
    init_fn
        Function to initiate the new weights with.
    n_subparams
        Number of sub-parameters contained in the parameter.

    Returns
    -------
    weights_expanded
        The expanded weights as a pytorch parameter.

    """
    shape_new_weights = list(weights.shape)
    shape_new_weights[axis] = n_dims_to_add
    all_weights = []
    for chunk in torch.chunk(weights, chunks=n_subparams, dim=axis):
        new_weights = torch.empty(
            *shape_new_weights, dtype=weights.dtype, device=weights.device
        )
        init_fn(new_weights)
        all_weights.extend([chunk, new_weights])
    weights_expanded = torch.cat(all_weights, dim=axis)
    return nn.Parameter(weights_expanded)

get_expansion_instructions 

get_expansion_instructions(param_shapes: Dict) -> Dict

Returns a dictionary containing information on how each parameter of a layer contained in param_shapes corresponds to the input and output dimensionality given its shape string.

Parameters:

Name Type Description Default
param_shapes Dict

Dictionary containing all parameters of a layer as keys and their corresponding shape strings as values.

 required

Returns:

Type Description
instructions

Dictionary specifying which axes of each parameter have to be altered to modify the input- or output dimensionality as well as the number of sub-parameters contained in the axes.
Source code in deep_river/utils/layer_adaptation.py 
 92
 93
 94
 95
 96
 97
 98
 99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
122
123
124
125
126
def get_expansion_instructions(
    param_shapes: Dict,
) -> Dict:
    """
    Returns a dictionary containing information on how
    each parameter of a layer contained in param_shapes
    corresponds to the input and output dimensionality
    given its shape string.

    Parameters
    ----------
    param_shapes
        Dictionary containing all parameters of a layer
        as keys and their corresponding shape strings as values.

    Returns
    -------
    instructions
        Dictionary specifying which axes of each parameter
        have to be altered to modify the input- or output
        dimensionality as well as the number of
        sub-parameters contained in the axes.

    """

    instructions = {}
    for key, shape_str in param_shapes.items():
        if shape_str == "I":
            instruction = "input_attribute"
        elif shape_str == "O":
            instruction = "output_attribute"
        else:
            instruction = get_in_out_axes(shape_str)
        instructions[key] = instruction
    return instructions

get_in_out_axes 

get_in_out_axes(shape_str: str)

Returns a dictionary containing information on how a specific parameter's axis sizes correspond to the input and output dimensionality given its shape string.

Parameters:

Name Type Description Default
shape_str str

String specifying the shape of a parameter.

 required

Returns:

Type Description
axes

Dictionary specifying which axes have to be altered to modify the input- or output dimensionality as well as the number of sub-parameters contained in the axes.
Source code in deep_river/utils/layer_adaptation.py 
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
def get_in_out_axes(shape_str: str):
    """
    Returns a dictionary containing information on how a
    specific parameter's axis sizes correspond to the input
    and output dimensionality given its shape string.

    Parameters
    ----------
    shape_str
        String specifying the shape of a parameter.

    Returns
    -------
    axes
        Dictionary specifying which axes have to be
        altered to modify the input- or output
        dimensionality as well as the number of
        sub-parameters contained in the axes.

    """
    check_shape_str(shape_str)
    shape_str = shape_str.strip("()")
    axis_strs = shape_str.split(",")
    axes: Dict = {"input": [], "output": []}
    for idx, axis_str in enumerate(axis_strs):
        input_output = re.findall(r"o|i", axis_str)

        if input_output:
            numbers = re.findall(r"\d+", axis_str)
            n_subparams = int(numbers[0]) if numbers else 1
            target = "output" if input_output[0] == "o" else "input"
            axes[target].append({"axis": idx, "n_subparams": n_subparams})

    return axes