GlusterFS OpenSource Distributed File System


GlusterFS is a distributed file system that can manage disk storage resources from multiple servers into a single global namespace. It is a type of distributed replicated network file-system, fully POSIX compliant and supports storage paradigms such as Block Storage and Object Storage.

GlusterFS stores the data on stable Linux file-systems like ext4, xfs etc. So it doesn’t need an additional metadata server for keeping metadata.

It is possible to build 3 types of volumes: distributed, replicated and striped. I strongly recommend reading GlusterFS architecture documentation from that link. GlusterFS has a well-documented architecture.

Some Advantages from GlusterFS architecture documentation:

  • Can use any on disk filesystem that supports extended attributes
  • Accessible using industry-standard protocols like NFS and SMB
  • Provides replication, quotas, geo-replication, snapshots and bitRot detection
  • Open Source

I added some basic terms that we will use throughout this post. Also recommended to check this link for much more detail.

Distributed File System — A file system that allows multiple clients to access data which is spread across cluster peer. The servers allow the client to share and store data just like they are working on locally.

Cluster — a group of peer computer that works together.

Trusted Storage Pool — A storage pool is a trusted network of storage servers.

Fuse — File system in Userspace (FUSE) is a loadable kernel module that lets non-privileged users create files without editing kernel codes. FUSE module only provides a bridge to access to actual kernel interface.

Glusterd — The GlusterFS daemon/service process that needs to be run all members of the trusted storage pool to manage volumes and cluster membership.

Brick — the basic unit of storage in GlusterFS. Represented by an export directory on a server in a trusted storage pool.

Volume — logical collection of bricks.

If you are excited to setup GlusterFS file system and check its’ features, please have a look for more detail about how to install and configure GlusterFS”.

The requirement to install and test GlusterFS:

Configure and Install Binaries

DNS configuration

You should define “hostname” and “ip” information to DNS server or hosts files. I will prefer to add information to “/etc/hosts” file.

#vi /etc/hosts
192.168.56.102 dockert01
192.168.56.103 dockert02
192.168.56.104 dockert03
192.168.56.1 client

Define GlusterFS repository on both client and storage server

Define GlusterFS repositories for all servers in a trusted pool and also for the client. I added the Centos base repository for GlusterFS. You should check GlusterFS repository which belongs to your Linux distribution.

# cat /etc/yum.repos.d/glusterfs.repo 
[gluster5]
name=Gluster 5
baseurl=http://mirror.centos.org/centos/7/storage/x86_64/gluster-5/
gpgcheck=0
enabled=1

c) Install Glusterfs-Server on Storage Nodes

#yum install -y glusterfs-server

#systemctl start glusterd

#systemctl enable glusterd

Configure firewall rules

I always prefer to disable firewalld daemon on my test platform. But if you have any obligation to enable firewalld , you may define firewall rules for all Server and clients.

Disable firewallD:

#systemctl stop firewalld
#systemctl disable firewalld
#firewall-cmd –reload

Or define rules:

#firewall-cmd  --permanent --zone=public --add-rich-rule='rule family="ipv4" source address="<ipaddress>" accept'
#firewall-cmd –reload

 

Create local file system on storage nodes

Add the disk to storage servers

You need to perform these steps at all trusted storage pool servers. You need to add your storage server to an additional disk. And mount point should be separated from the root file system.

[root@dockert01 ~]# fdisk  /dev/sdb
Welcome to fdisk (util-linux 2.23.2).

Changes will remain in memory only, until you decide to write them.
Be careful before using the write command.

Device does not contain a recognized partition table
Building a new DOS disklabel with disk identifier 0x7dbe5c4a.

Command (m for help): n --> New Partition
Partition type:
   p   primary (0 primary, 0 extended, 4 free)
   e   extended
Select (default p): p
Partition number (1-4, default 1): 1 --> Partition Number
First sector (2048-16777215, default 2048):  -->Press Enter
Using default value 2048
Last sector, +sectors or +size{K,M,G} (2048-16777215, default 16777215): 
Using default value 16777215
Partition 1 of type Linux and of size 8 GiB is set

Command (m for help): w --> Write Changes to disk
The partition table has been altered!

Calling ioctl() to re-read partition table.
Syncing disks.

[root@dockert01 ~]# mkfs.xfs /dev/sdb1 
meta-data=/dev/sdb1              isize=512    agcount=4, agsize=524224 blks
         =                       sectsz=512   attr=2, projid32bit=1
         =                       crc=1        finobt=0, sparse=0
data     =                       bsize=4096   blocks=2096896, imaxpct=25
         =                       sunit=0      swidth=0 blks
naming   =version 2              bsize=4096   ascii-ci=0 ftype=1
log      =internal log           bsize=4096   blocks=2560, version=2
         =                       sectsz=512   sunit=0 blks, lazy-count=1
realtime =none                   extsz=4096   blocks=0, rtextents=0

[root@dockert01 ~]# mkdir -p /data/gluster
[root@dockert01 ~]# mount  /dev/sdb1 /data/gluster/
[root@dockert01 ~]# df -h /data/gluster
/dev/sdb1                8.0G   33M  8.0G   1% /data/gluster

Add Storage nodes to the trusted storage pool

[root@dockert01 ~]# gluster peer probe dockert02
peer probe: success. 
[root@dockert01 ~]# gluster peer probe dockert03
peer probe: success. 
[root@dockert01 ~]# gluster peer status
Number of Peers: 2
Hostname: dockert02
Uuid: 182c8b4d-cad1-42a3-a77c-e7654c6110e8
State: Peer in Cluster (Connected)
Hostname: dockert03
Uuid: 60ddf4ee-22ea-4c93-bad7-e428c37dc3d5
State: Peer in Cluster (Connected)

[root@dockert01 ~]# gluster pool list
UUID					Hostname 	State
182c8b4d-cad1-42a3-a77c-e7654c6110e8	dockert02	Connected 
60ddf4ee-22ea-4c93-bad7-e428c37dc3d5	dockert03	Connected 
07f4232c-5da6-4e0f-9b5d-0f65234f751a	localhost	Connected 

Create GlusterFS volume group

You need to create a brick directory called "gvol0" on all storage nodes.

[root@dockert01 ~]# mkdir -p /data/gluster/gvol0
[root@dockert02 ~]# mkdir -p /data/gluster/gvol0
[root@dockert03 ~]# mkdir -p /data/gluster/gvol0

You may want to check this link to supported architecture on glusterFS. I will perform to setup the process for  "Replicated Glusterfs Volume" architecture with 3 replicas.  The number of replicas can be defined while create volume.

 

***Perform this  step on one  of the  nodes  in trusted pool.***
[root@dockert01 ~]# gluster volume create gvol0 replica 3 dockert01:/data/gluster/gvol0 dockert02:/data/gluster/gvol0 dockert03:/data/gluster/gvol0 
volume create: gvol0: success: please start the volume to access data
[root@dockert01 ~]# gluster volume start gvol0
volume start: gvol0: success
[root@dockert01 ~]# 

Get Information about volume

[root@dockert01 ~]# gluster volume info gvol0
Volume Name: gvol0
Type: Replicate
Volume ID: fa97f509-52c9-4243-bdda-a600e549c574
Status: Started
Snapshot Count: 0
Number of Bricks: 1 x 3 = 3
Transport-type: tcp
Bricks:
Brick1: dockert01:/data/gluster/gvol0
Brick2: dockert02:/data/gluster/gvol0
Brick3: dockert03:/data/gluster/gvol0
Options Reconfigured:
transport.address-family: inet
nfs.disable: on
performance.client-io-threads: off

 

Test GlusterFS file system

You should install glusterFS client package on the client-server which needs to access the file.

[root@client mnt]#yum install  glusterfs-client-xlators.x86_64
[root@client mnt]#mount -t glusterfs dockert01:/gvol0 /mnt/glusterfs01/
[root@client mnt]#mount -t glusterfs dockert02:/gvol0 /mnt/glusterfs02/
[root@client mnt]#mount -t glusterfs dockert03:/gvol0 /mnt/glusterfs03/
[root@client mnt]#cd /mnt/
[root@client mnt]# ls   -lrt  glusterfs*
glusterfs:
total 0
glusterfs03:	→ Node03
total 0
-rw-r--r--. 1 root root 0 Dec 25 15:04 file01
-rw-r--r--. 1 root root 0 Dec 25 15:05 file03
-rw-r--r--. 1 root root 0 Dec 25 15:05 file02

glusterfs02: 	→ Node02
total 0
-rw-r--r--. 1 root root 0 Dec 25 15:04 file01
-rw-r--r--. 1 root root 0 Dec 25 15:05 file03
-rw-r--r--. 1 root root 0 Dec 25 15:05 file02

glusterfs01:	→ Node01
total 0
-rw-r--r--. 1 root root 0 Dec 25 15:04 file01
-rw-r--r--. 1 root root 0 Dec 25 15:05 file03
-rw-r--r--. 1 root root 0 Dec 25 15:05 file02

I'm a IT Infrastructure and Operations Architect with extensive experience and administration skills and works for Turk Telekom. I provide hardware and software support for the IT Infrastructure and Operations tasks.

205 Total Posts
Follow Me

76 Comments

  1. Other Drugs viagra connect usa - saleviabuy.com, generic viagra online for sale

  2. buying viagra online safe http://fmedrx.com/ cheap viagra online pharmacy buy viagra online canadian

  3. generic viagra reviews http://usggrxmed.com/ best place to buy viagra online buy generic viagra

  4. lyrics viagra culture viagra free trial sample viagra pulmonary hypertension dosage

  5. canada pharmacy viagra paradiseviagira.com viagra generico

  6. how long will effects of viagra last horse on viagra does viagra make your hair grow

  7. viagra on line paying with paypal viagra on line viagra india pharmacy

  8. where can i buy viagra melbourne https://purevigra.com shelf life of generic viagra

  9. viagra laboratorio bago https://buybuyviamen.com maxima dosis viagra

  10. viagra over night delivery viagra canada free sample viagra in sydney nsw

  11. preços do remedio viagra where can you buy viagra in hong kong viagra for alzheimer's

  12. otc viagra 2017 how to get viagra without a prescription viagra pharmacy usa

  13. purchasing viagra in usa viagra online pills sildenafil 20 mg without a prescription

  14. buy viagra online with mastercard real viagra online prescription canadian pharmacy viagra 50 mg

  15. non prescription viagra best viagra for sale viagra 100 buy

  16. site de venda de viagra viagra oral suspension can you buy viagra in us

  17. viagra men under 50 fda counterfeit viagra viagra mecanisme action

  18. college essay 2018 write a personal essay grammar help

  19. viagra no brasil buy viagra in melbourne cialis & viagra

  20. write me a research paper on compost compare and contrast essay help format for writing a history research paper

  21. the perfect cover letter things to write an essay about write a five paragraph essay

  22. essay on persuasion website to write an essay for me quality essay writing

  23. research paper 2016 freelance essay writers student coursework

  24. essay application for college write thesis methods of essay writing

  25. data analysis dissertation read write think research paper strong thesis

  26. order viagra onlin viagra for cheap on line viagra pharmacy paypal

  27. prescription viagra cost viagra for sale in canada viagra online with american express

  28. cialis 20mg cialis fastest shipping cheap cialis canadian

  29. order original cialis where to find cheap cialis original cialis com

  30. viking_sun viagra genuine viagra online online viagra reviews

  31. viagra for sale on line what are the different kinds of viagra? viagra for sale

  32. cialis reviews generic cialis quick shipping cheap cialis

  33. cheap viagra express shipping buy viagra without rx viagra super active plus

  34. walmartviagra order viagra china viagra on line

  35. synthetic viagra buy viagra without prescription? best viagra sites

  36. online drugstore pharmacy treatment for erectile dysfunction treatment for erectile dysfunction

  37. cheap erectile dysfunction pills online pharmacy rx world best drugstore shampoo

  38. erectile cheapest pharmacy drugs from canada

  39. professional pharmacy erectile dysfunction medications ed treatment

  40. impotence best canadian pharmacy my canadian pharmacy

  41. viagra cialis canada viagra australia where to buy viagra 100mg cheap viagra online canadian pharmacy buy viagra super active order viagra no prescription viagra from canada soft viagra with dapoxetine will viagra make you last longer what\'s works better viagra or cialis? brand viagra viagra free samples viagra 100mg online viagra and shelf life generic viagra

  42. 100mg viagra on line brand viagra where to get viagra samples directions for viagra purchsse viagra online alternatives to viagra men buy viagra with discover card buyng viagra in sydney purchase generic viagra viagra alternatives australia viagrapills viagra online cheap buy viagra no prescription by cipro viagra without percription viagra 100mg

  43. viagra chep viagra online canada no prescription over the counter viagra for sale in ireland http://genqpviag.com/ - viagra triangle chicago ’

  44. cialis paypall best price on cialis 20mg generic cialis cipla http://21cialismen.com/ - originalcialis ’

  45. cialis tablets for sell brand cialis for sale cialis online australia http://phrcialiled.com/ - cheap online cialis ’

  46. free viagra samples australia viagra livraison 24h viagra with dapoxetine http://acialaarx.com/ - how much does a viagra prescription cost ’

  47. cialis cialis 20 mg canada can i buy cialis without prescription pay pal http://cialijomen.com/ - cialis ’

  48. viagra walgreens suppliers of generic viagra how to make viagra at home http://genericrxxx.com/ - viagra ’

  49. viagra in canada viagra substitutes over the counter online viagra in australia http://llviabest.com/ - how much viagra required australia ’

  50. mastercard cialis cialis tablets australia cialisВ® online http://mycialedst.com/ - cialis and paypal ’

  51. buy viagra with discover card generic viagra cheap overnight viagra with dapoxetine https://canadianpharmacy-usx.com/organic.htm - viagra patent ends ’

  52. cheap cialis buy brand cialis online cialis on lone http://loncialis.com/ - brand cialis australia ’

  53. cialis on ebay cheap generic cialis online free cialis canada http://cialmenshoprx.com/ - cialis store in philippines ’

  54. differences between viagra levitra and cialis but viagra with no prescription on line viagra http://xz-pharmacyonline.com/en/career-opportunities.html - fast shipping on viagra au ’

  55. buy cialis generic cialis black australia how much does cialis cost in canada http://sjcialis.com/ - cialis 60 mg ’

  56. cialis on paypal health canada drug database cheap cialis 20mg https://xz-pharmacyonline.com - Copegus ’

  57. cialis from canada Remeron canadian pharmacies without prescriptions https://canadianpharmacy-yy.com/ - Kamagra Gold ’

  58. viagra and viagra viagra levitra viagra offers whats better viagra viagra or levitra http://erectilejyzd.com/index.php?called=terms - low price viagra ’

  59. how to buy cialis from canada canadian pharmacy cialis cialisprofessional https://impotencecdny.com/ - Lozol ’

  60. buy cheap viagra online us buy viagra australia online cheapest sildenafil online http://canadianpharmacy-yy.com/Fitness/how-to-get-a-six-pack.html - sildenafil 100mg for sale ’

  61. cialis over the counter in spain cialis paypal bezahlen cialis 20mg shopping https://impotencecdny.com/info/history.php - cialis 60mg ’

  62. viagra for entertainment purposes canadian pharmacy buy viagra in mexico http://erectilejyzd.com/ - buy prescription drugs canada ’

  63. publix pharmacy online ordering canadian pharcharmy Viagra

  64. prescription cost comparison discount viagra online online pharmacy without scripts

  65. northwest pharmacy/com canadian pharmaceuticals viagra generic canadian pharmacy

  66. viagra maintain erection after ejaculation viagra toronto headquarters generic viagra online next day shipping

  67. forum tried viagra no prescription viagra for women in hyderabad viagra in kuwait

  68. cash advance income tax refund payday loans nampa id payday loans listowel

  69. walmart viagra price drug price grapefruit juice with viagra

  70. cialis sirve para las mujeres side effects cialis levitra viagra wo bekomme ich cialis gГјnstig

  71. payday and title loans online payday loans with no direct deposit payday loans near dallas tx

  72. payday loans centurion payday loans silsbee tx ace cash advance fontana ca

  73. cialis potassium achat cialis pharmacie francaise cialis stay hard after ejaculation

  74. viagra 6800mg viagra online safe buy viagra 200 mg

  75. viagra generic vs brand name http://kloviagrli.com/ order viagra in israel

  76. viagra cialis enzyte cialis pills buy c5 cialis

Leave a Reply