CSR matrix format

[msgsvc]

Hi，

I want to change default matrix format to CSR. What should I do?

best regards,

Jackie

[Yvan Fournier]

Hello,

Could you explain why you posted this in the discussion (development topics) forum, and not in the general usage one ?

Check the examples in src/user_examples/cs_user_performance_tuning.c.

Regards,

Yvan

[msgsvc]

Thank you Yvan for the question.

I want to enhance matrix.vector product's performance with usage of GPU parallel and CSR matrix format have good performance for GPU Acceleration of Parallel. I could revise and re-compile resouce codes.

I don't know how to choose CSR matrix format for the only matrix from beginning to end, and don't produce block.

best

Jackie

[Yvan Fournier]

Hello,

The example in the file I mentioned helps you to force CSR for matrix types supporting it (i.e. non blocks). You can also use MSR (CSR with separate diagonals) which also supports blocks, and is often very slightly faster than CSR (but CSR is better if you want to link to libraries with built-in conversions from CSR to other formats).

If you want to tes GPU accelearation of SpMV products, you'il need to add GPU-enabled functions for SpMV in src/base/cs_matrix.c. Those products are based on function pointers, to allow multiple functions for autotuning, so that part should be relatively easy. To add a function rather than modify one (if you want to keep autotuning) you should also look at the cs_matrix_variant_build_list in the same file. To run the code with some operator benchmarking enabled (which requires a mesh, but does not resuire setting up data such as boundary conditions), you can start with the benchmark mode of the code (not available through the GUI, buit available through the cs_user_scripts.py file, or running the code using "code_saturne run --initialize", then in the RESU/<run_id> folder, editing "run_solver", and adding "--benchmark" after cs_solver.

Regards,

Yvan

[msgsvc]

Hello,

Thanks a lot, I just want to modify matrix types and GPU accelearation of SpMV products. After I revised the file you mentioned, I still have some questions. This time I just modify matrix types.

1. The examples in src/user/cs_user_performance_tuning.c.
2. When I use callgirnd&qcachegrind tools based on the call graph, it shows SpMV products have _block_3_jacobi functions and cs_matrix_vector_multiply which call mat_vec_p_l_msr and mat_vec_p_l_native functions both.
3. It's one block, isn't it?

Last but not least, I want to know how to know where I can get compiler profile(add parameter) informantions.

I upload a revised cs_user_performance_tuning file.

Best Regards,

Jackie

[Yvan Fournier]

Hello,

You set the CSR option for scalar matrices, but for the velocity, the matrix filling is not scalar but BLOCK33_DIAG (or some similar name), which means diagonal blocks are 3x3 (coupling velocity components) and extra-diagonal blocks are of type aij.Identity (so a single value needs to be saved for each block). This variant cannot yet be converted to CSR or block CSR (except maybe for conversion to PETSc) in Code_Saturne.

Also, you should remove the "set_tuning" calls, as I am not sure tuning may not take place and change your settings. Forcing CSR, this way, most linear systems should use CSR, except perhaps the pressure correction ,where you need to use multigrid as a solver, or PCG preconditioned by Jacobi, but not by multigrid (PCG preconditionned by multigrid uses Gauss-Seidel as a smoother, which is only coded for MSR, and forces an MSR format for most operations).

Regards,

[msgsvc]

Hello,

I've already read a paper named "Accelerating the Conjugate Gradient Algorithm with GPU in CFD Simulations", which mentioned CUBlas library, CUsparse library, MAGMA library and ViennaCL library. I don't know how to use these libraries in Code Saturne. And are these libraries used in "Block matrices"?

Moreover, you said I should remove the "set_tuning" calls. Does it mean three calls(in src/user/cs_user_performance_tuning.c) as follow:

• 1.cs_matrix_set_tuning(CS_MATRIX_SCALAR, 1);

• 2.cs_matrix_set_tuning(CS_MATRIX_SCALAR_SYM, 1);

• 3.cs_matrix_set_tuning_runs(10, 0.5);

Lastly, If I want to know Code Saturne theory about how to link physic model and solver calls ( such as the pressure correction which need too use multigrid as a solver), what should I do?

Best Regards,

Jackie

[Yvan Fournier]

Hello,

Yes, you should replace the lines you mention with the one forcing a given format, "set_variant" in the examples.

Most matrices are scalar (i.e. not "block", and the ones which use blocks (such as for Velocity) use a special storage, with full blocks on the diagonal, and a constant diagonal for others, so generic "full block" formats may have a hard time competing, even on a GPU, as they would require more memory bandwidth.

There have been some tests on GPUS (http://vecpar.fe.up.pt/2016/docs/VECPAR ... aper_8.pdf), but the one referenced has been done on matrixes dumped by Code_Saturne, but not yet directly in the full code. For a complete case, other tests have been run using GPU's through PETSc (http://emit.tech/EMiT2016/Moulinec-EMIT ... celona.pdf), but the performance advantage si smaller.

In any case, if you want to use libraries directly, using CSR is only a first step: you would have to call the aforementioned libraries in the cs_matrix.c code (CUSparse does not seem too difficult to use, and would probably be only slight more complex to plug than the MKL (plugged as an option in the same file), but ensuring most of the work done by linear solvers is done by the GPU (including all BLAS1 operations, etc.) is a much larger task.

Regards,

Yvan

[msgsvc]

Thank you Yvan for the question.

I replace the lines and put this file in attachment. I want to know the revise to see if it corrects.
If wrong, could you revise this file and reload?

best

Jackie

[Yvan Fournier]

Hello,

Yes, this should work for scalar matrices. You might want to add the same setting for symmetric scalar matrices (CS_MATRIX_SCALAR_SYM) such as the one used for the pressure correction (in which case if you are using version 4.2 or above,you cannot use the default PCG+Multigrid preconditionner, but must use either PCG+jacobi preconditionner, or Multigrid as a solver, which was the default before version 4.2).

Regards,

Yvan