Skip to content

inline bias_act! fallback method#737

Open
ancapdev wants to merge 1 commit into
FluxML:masterfrom
ancapdev:patch-1
Open

inline bias_act! fallback method#737
ancapdev wants to merge 1 commit into
FluxML:masterfrom
ancapdev:patch-1

Conversation

@ancapdev

Copy link
Copy Markdown

This may be a bit niche, but I build very small allocation free models with Flux and StaticArrays. bias_act!() is called from various Flux layers and appears sensitive to inlining heuristics. Beyond some level of complexity (size of inputs and type of activation) it allocates.

Example:

N = 32
X = rand(SVector{N, Float32})
b = rand(SVector{N, Float32})
@btime bias_act!(leakyrelu, $(Ref(X))[], $(Ref(b))[]);

Before:

 386.302 ns (5 allocations: 800 bytes)

After:

  2.988 ns (0 allocations: 0 bytes)

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant