Os processos de descarga consomem muito CPU

O servidor é uma instância do EC2, significa salvar arquivos no NAS (NFS) do HTTPD.

Os processos como flush-0: 32 consomem mais de 90 da CPU e média de carga: 65,50, 64,02, 66,59.

De acordo com o gráfico, está aumentando todos os dias, enquanto a média de carga inicial foi de ~ 1,01, 2,02, 1,80 em 4 núcleos. Adicionei outra instância semelhante no Load Balancer e sua utilização da CPU é de apenas cerca de% 6 ATM.

O que esses processos de descarga fazem exatamente?

Talvez devêssemos desativar o cache de atributos NFS se os clientes precisarem apenas gravar dados?

Isso poderia ser devido à fragmentação de pacotes?

Aqui estão algumas estatísticas de nfsstat -s -4:

=================================================================
Server 0:

Server rpc stats:
calls      badcalls   badauth    badclnt    xdrcall
715054137   0          0          0          0       

Server nfs v4:
null         compound     
993       0% 715053143 99% 

Server nfs v4 operations:
op0-unused   op1-unused   op2-future   access       close        commit       
0         0% 0         0% 0         0% 143229323  6% 78092765  3% 36693816  1% 
create       delegpurge   delegreturn  getattr      getfh        link         
3486926   0% 0         0% 0         0% 679872421 28% 158406682  6% 0         0% 
lock         lockt        locku        lookup       lookup_root  nverify      
0         0% 0         0% 0         0% 95872524  4% 0         0% 0         0% 
open         openattr     open_conf    open_dgrd    putfh        putpubfh     
78173920  3% 0         0% 46442107  1% 1668      0% 715044032 29% 0         0% 
putrootfh    read         readdir      readlink     remove       rename       
7110      0% 42081145  1% 9116904   0% 0         0% 7026      0% 1257      0% 
renew        restorefh    savefh       secinfo      setattr      setcltid     
14        0% 81591622  3% 81659244  3% 0         0% 21028018  0% 3244      0% 
setcltidconf verify       write        rellockowner bc_ctl       bind_conn    
3244      0% 0         0% 114086560  4% 0         0% 0         0% 0         0% 
exchange_id  create_ses   destroy_ses  free_stateid getdirdeleg  getdevinfo   
0         0% 0         0% 0         0% 0         0% 0         0% 0         0% 
getdevlist   layoutcommit layoutget    layoutreturn secinfononam sequence     
0         0% 0         0% 0         0% 0         0% 0         0% 0         0% 
set_ssv      test_stateid want_deleg   destroy_clid reclaim_comp 
0         0% 0         0% 0         0% 0         0% 0         0% 

=================================================================
Server 1:

Server rpc stats:
calls      badcalls   badauth    badclnt    xdrcall
307172153   0          0          0          0       

Server nfs v4:
null         compound     
427       0% 307171725 99% 

Server nfs v4 operations:
op0-unused   op1-unused   op2-future   access       close        commit       
0         0% 0         0% 0         0% 58451998  5% 32717934  3% 15557564  1% 
create       delegpurge   delegreturn  getattr      getfh        link         
1424670   0% 0         0% 0         0% 291829363 28% 67959378  6% 0         0% 
lock         lockt        locku        lookup       lookup_root  nverify      
0         0% 0         0% 0         0% 41790934  4% 0         0% 0         0% 
open         openattr     open_conf    open_dgrd    putfh        putpubfh     
32741459  3% 0         0% 18993781  1% 75        0% 307167167 30% 0         0% 
putrootfh    read         readdir      readlink     remove       rename       
3108      0% 18598329  1% 3892199   0% 0         0% 742       0% 2         0% 
renew        restorefh    savefh       secinfo      setattr      setcltid     
3         0% 34142743  3% 34166131  3% 0         0% 8963430   0% 1449      0% 
setcltidconf verify       write        rellockowner bc_ctl       bind_conn    
1449      0% 0         0% 54628017  5% 0         0% 0         0% 0         0% 
exchange_id  create_ses   destroy_ses  free_stateid getdirdeleg  getdevinfo   
0         0% 0         0% 0         0% 0         0% 0         0% 0         0% 
getdevlist   layoutcommit layoutget    layoutreturn secinfononam sequence     
0         0% 0         0% 0         0% 0         0% 0         0% 0         0% 
set_ssv      test_stateid want_deleg   destroy_clid reclaim_comp 
0         0% 0         0% 0         0% 0         0% 0         0% 

=================================================================
Server 2:

Server rpc stats:
calls      badcalls   badauth    badclnt    xdrcall
53026598   0          0          0          0       

Server nfs v4:
null         compound     
89        0% 53026508 99% 

Server nfs v4 operations:
op0-unused   op1-unused   op2-future   access       close        commit       
0         0% 0         0% 0         0% 9605179   5% 5472897   3% 2633853   1% 
create       delegpurge   delegreturn  getattr      getfh        link         
231276    0% 0         0% 0         0% 50395149 28% 11903036  6% 0         0% 
lock         lockt        locku        lookup       lookup_root  nverify      
0         0% 0         0% 0         0% 7528324   4% 0         0% 0         0% 
open         openattr     open_conf    open_dgrd    putfh        putpubfh     
5477948   3% 0         0% 2760996   1% 15        0% 53025580 30% 0         0% 
putrootfh    read         readdir      readlink     remove       rename       
633       0% 3566001   2% 673466    0% 0         0% 11        0% 0         0% 
renew        restorefh    savefh       secinfo      setattr      setcltid     
2         0% 5704253   3% 5709223   3% 0         0% 1526967   0% 292       0% 
setcltidconf verify       write        rellockowner bc_ctl       bind_conn    
292       0% 0         0% 10439844  5% 0         0% 0         0% 0         0% 
exchange_id  create_ses   destroy_ses  free_stateid getdirdeleg  getdevinfo   
0         0% 0         0% 0         0% 0         0% 0         0% 0         0% 
getdevlist   layoutcommit layoutget    layoutreturn secinfononam sequence     
0         0% 0         0% 0         0% 0         0% 0         0% 0         0% 
set_ssv      test_stateid want_deleg   destroy_clid reclaim_comp 
0         0% 0         0% 0         0% 0         0% 0         0% 

=================================================================
Server 3:

Server rpc stats:
calls      badcalls   badauth    badclnt    xdrcall
63045403   0          0          0          0       

Server nfs v4:
null         compound     
121       0% 63045280 99% 

Server nfs v4 operations:
op0-unused   op1-unused   op2-future   access       close        commit       
0         0% 0         0% 0         0% 11504749  5% 6504139   3% 3119453   1% 
create       delegpurge   delegreturn  getattr      getfh        link         
271128    0% 0         0% 0         0% 59865633 28% 14058385  6% 0         0% 
lock         lockt        locku        lookup       lookup_root  nverify      
0         0% 0         0% 0         0% 8852565   4% 0         0% 0         0% 
open         openattr     open_conf    open_dgrd    putfh        putpubfh     
6521385   3% 0         0% 3365913   1% 15        0% 63043988 30% 0         0% 
putrootfh    read         readdir      readlink     remove       rename       
874       0% 4209822   2% 791702    0% 0         0% 6         0% 0         0% 
renew        restorefh    savefh       secinfo      setattr      setcltid     
9         0% 6775367   3% 6792509   3% 0         0% 1811226   0% 409       0% 
setcltidconf verify       write        rellockowner bc_ctl       bind_conn    
409       0% 0         0% 12368747  5% 0         0% 0         0% 0         0% 
exchange_id  create_ses   destroy_ses  free_stateid getdirdeleg  getdevinfo   
0         0% 0         0% 0         0% 0         0% 0         0% 0         0% 
getdevlist   layoutcommit layoutget    layoutreturn secinfononam sequence     
0         0% 0         0% 0         0% 0         0% 0         0% 0         0% 
set_ssv      test_stateid want_deleg   destroy_clid reclaim_comp 
0         0% 0         0% 0         0% 0         0% 0         0% 

=================================================================
Server 4:

Server rpc stats:
calls      badcalls   badauth    badclnt    xdrcall
817288490   0          0          0          0       

Server nfs v4:
null         compound     
1204      0% 817287285 99% 

Server nfs v4 operations:
op0-unused   op1-unused   op2-future   access       close        commit       
0         0% 0         0% 0         0% 164320609  6% 89471711  3% 42448842  1% 
create       delegpurge   delegreturn  getattr      getfh        link         
4101436   0% 0         0% 0         0% 778155935 28% 180629867  6% 0         0% 
lock         lockt        locku        lookup       lookup_root  nverify      
0         0% 0         0% 0         0% 109104313  4% 0         0% 0         0% 
open         openattr     open_conf    open_dgrd    putfh        putpubfh     
89598740  3% 0         0% 53534516  1% 9727      0% 817288175 29% 0         0% 
putrootfh    read         readdir      readlink     remove       rename       
8020      0% 46348481  1% 10773529  0% 0         0% 100880    0% 12342     0% 
renew        restorefh    savefh       secinfo      setattr      setcltid     
80        0% 93709338  3% 93712518  3% 0         0% 24303185  0% 3352      0% 
setcltidconf verify       write        rellockowner bc_ctl       bind_conn    
3352      0% 0         0% 127464001  4% 0         0% 0         0% 0         0% 
exchange_id  create_ses   destroy_ses  free_stateid getdirdeleg  getdevinfo   
0         0% 0         0% 0         0% 0         0% 0         0% 0         0% 
getdevlist   layoutcommit layoutget    layoutreturn secinfononam sequence     
0         0% 0         0% 0         0% 0         0% 0         0% 0         0% 
set_ssv      test_stateid want_deleg   destroy_clid reclaim_comp 
0         0% 0         0% 0         0% 0         0% 0         0% 

=================================================================
Server 5:

Server rpc stats:
calls      badcalls   badauth    badclnt    xdrcall
804660319   0          0          0          0       

Server nfs v4:
null         compound     
1204      0% 804659114 99% 

Server nfs v4 operations:
op0-unused   op1-unused   op2-future   access       close        commit       
0         0% 0         0% 0         0% 161331719  6% 88318366  3% 41571552  1% 
create       delegpurge   delegreturn  getattr      getfh        link         
4090384   0% 0         0% 0         0% 764533960 28% 177969216  6% 0         0% 
lock         lockt        locku        lookup       lookup_root  nverify      
0         0% 0         0% 0         0% 107385644  4% 0         0% 0         0% 
open         openattr     open_conf    open_dgrd    putfh        putpubfh     
88321805  3% 0         0% 54307425  2% 444       0% 804647492 29% 0         0% 
putrootfh    read         readdir      readlink     remove       rename       
8353      0% 45980723  1% 10410930  0% 0         0% 88471     0% 440       0% 
renew        restorefh    savefh       secinfo      setattr      setcltid     
21        0% 92410970  3% 92412629  3% 0         0% 23733174  0% 3688      0% 
setcltidconf verify       write        rellockowner bc_ctl       bind_conn    
3688      0% 0         0% 125268800  4% 0         0% 0         0% 0         0% 
exchange_id  create_ses   destroy_ses  free_stateid getdirdeleg  getdevinfo   
0         0% 0         0% 0         0% 0         0% 0         0% 0         0% 
getdevlist   layoutcommit layoutget    layoutreturn secinfononam sequence     
0         0% 0         0% 0         0% 0         0% 0         0% 0         0% 
set_ssv      test_stateid want_deleg   destroy_clid reclaim_comp 
0         0% 0         0% 0         0% 0         0% 0         0% 

=================================================================
Server 6:

Server rpc stats:
calls      badcalls   badauth    badclnt    xdrcall
795385017   0          0          0          0       

Server nfs v4:
null         compound     
1204      0% 795383812 99% 

Server nfs v4 operations:
op0-unused   op1-unused   op2-future   access       close        commit       
0         0% 0         0% 0         0% 158633282  5% 87331357  3% 41400927  1% 
create       delegpurge   delegreturn  getattr      getfh        link         
4080179   0% 0         0% 0         0% 756063664 28% 176355823  6% 0         0% 
lock         lockt        locku        lookup       lookup_root  nverify      
0         0% 0         0% 0         0% 106692513  4% 0         0% 0         0% 
open         openattr     open_conf    open_dgrd    putfh        putpubfh     
87333398  3% 0         0% 53591273  2% 187       0% 795371861 29% 0         0% 
putrootfh    read         readdir      readlink     remove       rename       
8367      0% 45030006  1% 10352133  0% 0         0% 80473     0% 151       0% 
renew        restorefh    savefh       secinfo      setattr      setcltid     
28        0% 91411943  3% 91413728  3% 0         0% 23629833  0% 3707      0% 
setcltidconf verify       write        rellockowner bc_ctl       bind_conn    
3707      0% 0         0% 124033436  4% 0         0% 0         0% 0         0% 
exchange_id  create_ses   destroy_ses  free_stateid getdirdeleg  getdevinfo   
0         0% 0         0% 0         0% 0         0% 0         0% 0         0% 
getdevlist   layoutcommit layoutget    layoutreturn secinfononam sequence     
0         0% 0         0% 0         0% 0         0% 0         0% 0         0% 
set_ssv      test_stateid want_deleg   destroy_clid reclaim_comp 
0         0% 0         0% 0         0% 0         0% 0         0% 

=================================================================
Server 7:

Server rpc stats:
calls      badcalls   badauth    badclnt    xdrcall
801916264   0          0          0          0       

Server nfs v4:
null         compound     
1204      0% 801915059 99% 

Server nfs v4 operations:
op0-unused   op1-unused   op2-future   access       close        commit       
0         0% 0         0% 0         0% 161888929  6% 88285947  3% 41212864  1% 
create       delegpurge   delegreturn  getattr      getfh        link         
4069479   0% 0         0% 0         0% 762072130 28% 177131560  6% 0         0% 
lock         lockt        locku        lookup       lookup_root  nverify      
0         0% 0         0% 0         0% 106437411  3% 0         0% 0         0% 
open         openattr     open_conf    open_dgrd    putfh        putpubfh     
88288337  3% 0         0% 54779036  2% 191       0% 801903449 29% 0         0% 
putrootfh    read         readdir      readlink     remove       rename       
8225      0% 45488312  1% 10259565  0% 0         0% 76243     0% 177       0% 
renew        restorefh    savefh       secinfo      setattr      setcltid     
4         0% 92355900  3% 92357993  3% 0         0% 23515286  0% 3558      0% 
setcltidconf verify       write        rellockowner bc_ctl       bind_conn    
3558      0% 0         0% 123741908  4% 0         0% 0         0% 0         0% 
exchange_id  create_ses   destroy_ses  free_stateid getdirdeleg  getdevinfo   
0         0% 0         0% 0         0% 0         0% 0         0% 0         0% 
getdevlist   layoutcommit layoutget    layoutreturn secinfononam sequence     
0         0% 0         0% 0         0% 0         0% 0         0% 0         0% 
set_ssv      test_stateid want_deleg   destroy_clid reclaim_comp 
0         0% 0         0% 0         0% 0         0% 0         0% 

=================================================================
Server 8:

Server rpc stats:
calls      badcalls   badauth    badclnt    xdrcall
804732833   1          1          0          0       

Server nfs v4:
null         compound     
1204      0% 804731628 99% 

Server nfs v4 operations:
op0-unused   op1-unused   op2-future   access       close        commit       
0         0% 0         0% 0         0% 161428939  6% 88340891  3% 41568432  1% 
create       delegpurge   delegreturn  getattr      getfh        link         
4085332   0% 0         0% 0         0% 764396486 28% 177796853  6% 0         0% 
lock         lockt        locku        lookup       lookup_root  nverify      
0         0% 0         0% 0         0% 107176837  3% 0         0% 0         0% 
open         openattr     open_conf    open_dgrd    putfh        putpubfh     
88342790  3% 0         0% 54344886  2% 187       0% 804720008 29% 0         0% 
putrootfh    read         readdir      readlink     remove       rename       
8226      0% 46219141  1% 10381361  0% 0         0% 83380     0% 160       0% 
renew        restorefh    savefh       secinfo      setattr      setcltid     
0         0% 92426602  3% 92428282  3% 0         0% 23736349  0% 3554      0% 
setcltidconf verify       write        rellockowner bc_ctl       bind_conn    
3554      0% 0         0% 125088530  4% 0         0% 0         0% 0         0% 
exchange_id  create_ses   destroy_ses  free_stateid getdirdeleg  getdevinfo   
0         0% 0         0% 0         0% 0         0% 0         0% 0         0% 
getdevlist   layoutcommit layoutget    layoutreturn secinfononam sequence     
0         0% 0         0% 0         0% 0         0% 0         0% 0         0% 
set_ssv      test_stateid want_deleg   destroy_clid reclaim_comp 
0         0% 0         0% 0         0% 0         0% 0         0% 

=================================================================
Server 9:

Server rpc stats:
calls      badcalls   badauth    badclnt    xdrcall
800961003   0          0          0          0       

Server nfs v4:
null         compound     
1204      0% 800959798 99% 

Server nfs v4 operations:
op0-unused   op1-unused   op2-future   access       close        commit       
0         0% 0         0% 0         0% 161394704  6% 88264733  3% 41642226  1% 
create       delegpurge   delegreturn  getattr      getfh        link         
4098314   0% 0         0% 0         0% 761225542 28% 172733291  6% 0         0% 
lock         lockt        locku        lookup       lookup_root  nverify      
0         0% 0         0% 0         0% 102217363  3% 0         0% 0         0% 
open         openattr     open_conf    open_dgrd    putfh        putpubfh     
88272429  3% 0         0% 53937975  2% 467       0% 800948292 30% 0         0% 
putrootfh    read         readdir      readlink     remove       rename       
8312      0% 45893437  1% 10409370  0% 0         0% 83127     0% 478       0% 
renew        restorefh    savefh       secinfo      setattr      setcltid     
35        0% 92369729  3% 92371221  3% 0         0% 23772628  0% 3637      0% 
setcltidconf verify       write        rellockowner bc_ctl       bind_conn    
3637      0% 0         0% 124997490  4% 0         0% 0         0% 0         0% 
exchange_id  create_ses   destroy_ses  free_stateid getdirdeleg  getdevinfo   
0         0% 0         0% 0         0% 0         0% 0         0% 0         0% 
getdevlist   layoutcommit layoutget    layoutreturn secinfononam sequence     
0         0% 0         0% 0         0% 0         0% 0         0% 0         0% 
set_ssv      test_stateid want_deleg   destroy_clid reclaim_comp 
0         0% 0         0% 0         0% 0         0% 0         0%

linux amazon-ec2 linux-kernel

— Roman Newaza
fonte

Quanto é muito?

— Fred Foo

já que a pergunta é sobre um bug do kernel que informa a versão do kernel teria ajudado.

— Dmitri Chubarov

processos de liberação são responsáveis pelo gerenciamento de write-back de páginas sujas no sistema de arquivos de onde vieram. Eles nunca devem ocupar muita CPU; a maior parte do tempo é gasta aguardando disco (ou rede para NFS e similares). Se você estiver vendo alto uso da CPU em um processo de liberação, pode ser um bug do kernel - tente reiniciar, isso deve limpar o estado.

— bdonlan
fonte

Oi! O que são páginas sujas? Sim, acabei de encerrar a Instância com falha e os outros estão operando normalmente em ATM.

— Roman Newaza

Páginas sujas são partes de arquivos armazenados em cache na memória, que foram modificados na memória por algum processo, mas ainda não foram gravados no disco. Normalmente, essa gravação ocorre em segundo plano, a menos que a quantidade de páginas sujas seja muito alta, a memória livre muito baixa ou o programa solicite explicitamente - quando a gravação ocorre em segundo plano, é tarefa do flush acionar essa gravação.

— bdonlan

e se eu correr echo 3 > /proc/sys/vm/drop_cachespara liberar pagecache, dentries e inodes?

— Roman Newaza

@RomanNewaza, isso eliminará apenas caches que não estão bloqueados ou sujos. lave normalmente não faz nada com caches não-sujas, por isso duvido que iria fazer a diferença (e isso vai desempenho gravemente ferido em um servidor até que as caches re-quente)

— bdonlan

Eu vejo. Costuras como o problema foi após a atualização do kernel.

— Roman Newaza