Repository navigation
Add an AVX DOMATCOPY_RT/CT kernel for x86_64 - #6135
Open
aalquwayfili wants to merge 1 commit into
Open
aalquwayfili wants to merge 1 commit into
aalquwayfili wants to merge 1 commit into
Conversation
The double precision transposed omatcopy used the generic C kernels from kernel/arm on x86_64. This adds a blocked kernel that transposes 4x4 blocks in AVX registers, with a plain C fallback for targets without AVX. When the leading dimensions allow it, the first rows and columns are peeled so the blocked part starts on 32-byte boundaries, which keeps 16-byte aligned (malloc'd) buffers from running slower than before.
This branch has not been deployed
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Part of #4902. On x86_64 only single precision has its own transposed omatcopy kernels, so
domatcopywith transpose goes through the generic C ones in kernel/arm. This adds an AVX kernel for DOMATCOPY_RT and DOMATCOPY_CT that transposes 4x4 blocks in registers and walks the matrix in 32x32 tiles. Targets without AVX get the same loop in plain C.One thing I ran into: with buffers from malloc (16-byte aligned) the blocked loop was slower than the old kernel at some sizes, e.g. 1000x1000 row-major. So when lda/ldb are multiples of 4 it does the first few rows and columns on their own and starts the blocked part on 32-byte boundaries. With that it was faster at every size I tried.
Single thread on an i9-9960X (SKYLAKEX build), malloc'd buffers,
cblas_domatcopywithCblasTrans, old -> new in GB/s (the box wasn't idle, so treat the small sizes as rough):The output is bitwise the same as the generic kernels for every m, n up to 37 with padded lda/ldb, every 8-byte offset of A and B, alpha in {1, -2.5, 0, 3e-300} and inputs with NaN, Inf, -0 and denormals, built with -mavx, -mavx2, -march=skylake-avx512 and without AVX. The CT alpha == 0 case still writes zeros without reading A, like before. utest_ext (1541) and the BLAS tests pass with make, and the cmake build picks up the kernel too.
This doesn't make it multithreaded, which is probably what's needed to get close to MKL on the sizes in #4902. I can look at that next if you think it's worth it.