systeme:ceph
Différences
Ci-dessous, les différences entre deux révisions de la page.
Les deux révisions précédentesRévision précédenteProchaine révision | Révision précédente | ||
systeme:ceph [2017/05/02 17:22] – root | systeme:ceph [2017/05/06 11:10] (Version actuelle) – [Piste de réflexion personnelle] root | ||
---|---|---|---|
Ligne 1264: | Ligne 1264: | ||
==== Procédures ==== | ==== Procédures ==== | ||
+ | === Installer ISCSI GW === | ||
+ | Exemple avec 2 gateway ISCSI. | ||
+ | |||
+ | * On crée une image de 3G sur le pool **cephfs_data** : | ||
+ | < | ||
+ | $ rbd create cephfs_data/ | ||
+ | </ | ||
+ | |||
+ | < | ||
+ | Toutes les commandes ci-dessous sont à faire sur les 2 serveurs. | ||
+ | </ | ||
+ | |||
+ | * On installe les paquets : | ||
+ | < | ||
+ | $ zypper in -t pattern ceph_iscsi | ||
+ | </ | ||
+ | |||
+ | * On active le service au boot pour générer la config : | ||
+ | < | ||
+ | $ systemctl enable lrbd.service | ||
+ | </ | ||
+ | |||
+ | * On édite la config avec la config **lrbd -e** : | ||
+ | < | ||
+ | { | ||
+ | " | ||
+ | { | ||
+ | " | ||
+ | " | ||
+ | } | ||
+ | ], | ||
+ | " | ||
+ | { | ||
+ | " | ||
+ | " | ||
+ | { | ||
+ | " | ||
+ | " | ||
+ | }, | ||
+ | { | ||
+ | " | ||
+ | " | ||
+ | } | ||
+ | ] | ||
+ | } | ||
+ | ], | ||
+ | " | ||
+ | { | ||
+ | " | ||
+ | " | ||
+ | " | ||
+ | ] | ||
+ | }, | ||
+ | { | ||
+ | " | ||
+ | " | ||
+ | " | ||
+ | ] | ||
+ | } | ||
+ | ], | ||
+ | " | ||
+ | { | ||
+ | " | ||
+ | " | ||
+ | { | ||
+ | " | ||
+ | " | ||
+ | { | ||
+ | " | ||
+ | } | ||
+ | ] | ||
+ | } | ||
+ | ] | ||
+ | } | ||
+ | ] | ||
+ | } | ||
+ | </ | ||
+ | |||
+ | * On applique la config : | ||
+ | < | ||
+ | $ lrbd | ||
+ | rbd -p cephfs_data --name client.admin map gigix-iscsi | ||
+ | /dev/rbd0 | ||
+ | targetcli / | ||
+ | Created RBD storage object cephfs_data-gigix-iscsi using / | ||
+ | targetcli /iscsi create iqn.2003-01.org.linux-iscsi.iscsi.x86: | ||
+ | Created target iqn.2003-01.org.linux-iscsi.iscsi.x86: | ||
+ | Selected TPG Tag 1. | ||
+ | Created TPG 1. | ||
+ | targetcli / | ||
+ | Created TPG 2. | ||
+ | targetcli / | ||
+ | The TPG has been disabled. | ||
+ | targetcli / | ||
+ | The TPG has been disabled. | ||
+ | targetcli / | ||
+ | Selected LUN 0. | ||
+ | Created LUN 0. | ||
+ | targetcli / | ||
+ | Selected LUN 0. | ||
+ | Created LUN 0. | ||
+ | targetcli / | ||
+ | Using default IP port 3260 | ||
+ | IP address 192.168.122.11 does not exist on this host. | ||
+ | Created network portal 192.168.122.11: | ||
+ | targetcli / | ||
+ | Using default IP port 3260 | ||
+ | Created network portal 192.168.122.13: | ||
+ | targetcli / | ||
+ | Parameter authentication is now ' | ||
+ | Parameter demo_mode_write_protect is now ' | ||
+ | Parameter generate_node_acls is now ' | ||
+ | targetcli / | ||
+ | Parameter authentication is now ' | ||
+ | Parameter demo_mode_write_protect is now ' | ||
+ | Parameter generate_node_acls is now ' | ||
+ | targetcli / | ||
+ | The TPG has been enabled. | ||
+ | </ | ||
+ | |||
+ | ou | ||
+ | |||
+ | < | ||
+ | $ systemctl start lrbd | ||
+ | </ | ||
+ | |||
+ | * On vérifie : | ||
+ | < | ||
+ | $ targetcli ls | ||
+ | o- / ......................................................................................................................... [...] | ||
+ | o- backstores .............................................................................................................. [...] | ||
+ | | o- fileio ................................................................................................... [0 Storage Object] | ||
+ | | o- iblock ................................................................................................... [0 Storage Object] | ||
+ | | o- pscsi .................................................................................................... [0 Storage Object] | ||
+ | | o- rbd ...................................................................................................... [1 Storage Object] | ||
+ | | | o- cephfs_data-gigix-iscsi ...................................................... [/ | ||
+ | | o- rd_mcp ................................................................................................... [0 Storage Object] | ||
+ | o- ib_srpt ........................................................................................................... [0 Targets] | ||
+ | o- iscsi .............................................................................................................. [1 Target] | ||
+ | | o- iqn.2003-01.org.linux-iscsi.iscsi.x86: | ||
+ | | o- tpg1 ........................................................................................................... [disabled] | ||
+ | | | o- acls ........................................................................................................... [0 ACLs] | ||
+ | | | o- luns ............................................................................................................ [1 LUN] | ||
+ | | | | o- lun0 ................................................. [rbd/ | ||
+ | | | o- portals ...................................................................................................... [1 Portal] | ||
+ | | | ||
+ | | o- tpg2 ............................................................................................................ [enabled] | ||
+ | | o- acls ........................................................................................................... [0 ACLs] | ||
+ | | o- luns ............................................................................................................ [1 LUN] | ||
+ | | | o- lun0 ................................................. [rbd/ | ||
+ | | o- portals ...................................................................................................... [1 Portal] | ||
+ | | o- 192.168.122.13: | ||
+ | o- loopback .......................................................................................................... [0 Targets] | ||
+ | o- qla2xxx ........................................................................................................... [0 Targets] | ||
+ | o- tcm_fc ............................................................................................................ [0 Targets] | ||
+ | o- vhost ............................................................................................................. [0 Targets] | ||
+ | </ | ||
+ | |||
+ | < | ||
+ | Maintenant on passe à la partie cliente. | ||
+ | </ | ||
+ | |||
+ | * On vérifie que l'on voit bien les luns : | ||
+ | < | ||
+ | $ iscsiadm -m discovery -t sendtargets -p 192.168.122.11 | ||
+ | 192.168.122.11: | ||
+ | 192.168.122.13: | ||
+ | </ | ||
+ | |||
+ | < | ||
+ | $ iscsiadm -m node -p 192.168.122.11 --login | ||
+ | Logging in to [iface: default, target: iqn.2003-01.org.linux-iscsi.iscsi.x86: | ||
+ | Login to [iface: default, target: iqn.2003-01.org.linux-iscsi.iscsi.x86: | ||
+ | |||
+ | $ iscsiadm -m node -p 192.168.122.13 --login | ||
+ | Logging in to [iface: default, target: iqn.2003-01.org.linux-iscsi.iscsi.x86: | ||
+ | Login to [iface: default, target: iqn.2003-01.org.linux-iscsi.iscsi.x86: | ||
+ | </ | ||
+ | |||
+ | * On liste les devices : | ||
+ | < | ||
+ | $ lsscsi -s | ||
+ | [0: | ||
+ | [2: | ||
+ | [3: | ||
+ | </ | ||
+ | |||
+ | * Editer le fichier ''/ | ||
+ | <code data multipath.conf> | ||
+ | defaults { | ||
+ | user_friendly_names yes | ||
+ | } | ||
+ | |||
+ | devices { | ||
+ | device { | ||
+ | vendor " | ||
+ | product " | ||
+ | path_grouping_policy " | ||
+ | path_checker " | ||
+ | features " | ||
+ | hardware_handler "1 alua" | ||
+ | prio " | ||
+ | failback " | ||
+ | rr_weight " | ||
+ | no_path_retry 12 | ||
+ | rr_min_io 100 | ||
+ | } | ||
+ | } | ||
+ | </ | ||
+ | |||
+ | * On liste les paths : | ||
+ | < | ||
+ | $ multipath -ll | ||
+ | mpatha (3600140571dc15dc9fa13437ae8840470) dm-2 SUSE ,RBD | ||
+ | size=3.0G features=' | ||
+ | `-+- policy=' | ||
+ | |- 2:0:0:0 sda 8:0 active ready running | ||
+ | `- 3:0:0:0 sdb 8:16 active ready running | ||
+ | </ | ||
+ | |||
+ | * On créer la table des partitions : | ||
+ | < | ||
+ | $ parted / | ||
+ | |||
+ | $ parted / | ||
+ | Modèle: Mappeur de périphériques Linux (multipath) (dm) | ||
+ | Disque / | ||
+ | Taille des secteurs (logiques/ | ||
+ | Table de partitions : gpt | ||
+ | Disk Flags: | ||
+ | |||
+ | Numéro | ||
+ | | ||
+ | </ | ||
+ | |||
+ | * On crée le FS xfs : | ||
+ | < | ||
+ | $ mkfs.xfs / | ||
+ | meta-data=/ | ||
+ | | ||
+ | | ||
+ | data | ||
+ | | ||
+ | naming | ||
+ | log =internal log | ||
+ | | ||
+ | realtime =none | ||
+ | </ | ||
+ | |||
=== Modifier la crush map === | === Modifier la crush map === | ||
Sauvegarde la crush map decompilée : | Sauvegarde la crush map decompilée : | ||
Ligne 1411: | Ligne 1660: | ||
</ | </ | ||
+ | === Quota CephFS === | ||
+ | == Mettre un quota CephFS == | ||
+ | < | ||
+ | $ setfattr -n ceph.quota.max_bytes -v 100000000 / | ||
+ | $ setfattr -n ceph.quota.max_files -v 10000 / | ||
+ | </ | ||
+ | |||
+ | == Voir les quotas Cepfs == | ||
+ | < | ||
+ | $ getfattr -n ceph.quota.max_bytes /some/dir | ||
+ | $ getfattr -n ceph.quota.max_files /some/dir | ||
+ | </ | ||
+ | |||
+ | == Supprimer un quota CephFS == | ||
+ | < | ||
+ | $ setfattr -n ceph.quota.max_bytes -v 0 /some/dir | ||
+ | $ setfattr -n ceph.quota.max_files -v 0 /some/dir | ||
+ | </ | ||
+ | |||
+ | |||
+ | ==== Piste de réflexion personnelle ==== | ||
+ | Piste de réflexion avec serveur disposant de 24 emplacements disques : | ||
+ | |||
+ | < | ||
+ | Plus de journal sur SSD avec BlueStore (tech preview) ? On gagne en disque ? | ||
+ | </ | ||
+ | |||
+ | * Piste HDD : | ||
+ | < | ||
+ | 2 OS | ||
+ | 4 SSD journal | ||
+ | 16 HDD data (2To) => 32T / Serveur | ||
+ | ======= | ||
+ | 22/24 hdd | ||
+ | |||
+ | 64G RAM ? | ||
+ | </ | ||
+ | |||
+ | * Piste SSD (prix ?): | ||
+ | < | ||
+ | 2 OS | ||
+ | 4 SSD journal | ||
+ | 18 SSD DATA (1To) ou si pas de journal séparé sur 22 SSD data => entre 18 et 22T / serveur | ||
+ | ================== | ||
+ | 24/24 ssd | ||
+ | |||
+ | Entre 18 et 22G RAM ? | ||
+ | </ | ||
+ | |||
+ | **Pour solution HDD :** | ||
+ | * 16 * 100M ~= 1600M/s (à 2000M/s) | ||
+ | * Solution RDMA infinidand ? | ||
+ | * Il faudrait donc 1 carte dual port 56 Gbps (par exemple Mellanox ConnectX4) ou 2 cartes dual port 10Gbps si moins cher (2 lacp) : | ||
+ | * 1 port pour cluster ceph pour la réplication et rebuid des osd | ||
+ | * 1 port pour le réseau public => les clients (Ethernet ou Infinidand pour clients ?) | ||
+ | |||
+ | **Protocol RDMA pour :** | ||
+ | * gagner en latence | ||
+ | * gagner en débit | ||
+ | * gagner en CPU | ||
+ | **Tunning :** | ||
+ | * numactl ? 1 osd par cpu (+bind mémoire) => ExecStart=numactl --physcpubind=%i --membind=0 <service ceph> (plus compliqué avec 2 proc. Si 2 proc : pair sur socket pair => Ex : osd.3 sur socket 1 car impair). 2 cpu réservés à l'OS (les 2 derniers => isolcpu) ? | ||
+ | * jumbo frame | ||
+ | * tunning sysctl + udev | ||
+ | * xio (non supporté en production) ? (voir [[https:// | ||
+ | * rdma (https:// | ||
+ | < | ||
+ | [global] | ||
+ | ... | ||
+ | ms_type=async+rdma | ||
+ | ms_async_rdma_device_name=mlx5_0 | ||
+ | </ |
systeme/ceph.1493745723.txt.gz · Dernière modification : 2017/05/02 17:22 de root