QEMU-GlusterFS native integration

October 29, 2012

GlusterFS is a distributed file system implemented in user space. It is strictly not a native file system in itself but is an aggregator of different file systems. GlusterFS can aggregate individual file system mount points or directories (called bricks in gluster terminology) to provide a single unified file system namespace. In addition to NFS and CIFS, the most common
way to access GlusterFS namespace is via FUSE based Gluster native client.

gluster

More information on creating and mounting GlusterFS volume can be obtained from GlusterFS website.

GlusterFS for virtualization

Until recently using GlusterFS volumes to host VM images and data was sub-optimal due to the FUSE overhead involved in accessing gluster volumes via GlusterFS native client. However this has changed now with two specific enhancements:

– A new library called libgfapi is now available as part of GlusterFS that  provides POSIX-like C APIs for accessing gluster volumes. libgfapi support will be available from GlusterFS-3.4 release.
– QEMU (starting from QEMU-1.3) will have GlusterFS block driver that uses libgfapi and hence there is no FUSE overhead any longer when QEMU works with VM images on gluster volumes.

GlusterFS with its pluggable translator model can serve as a flexible storage backend for QEMU. QEMU has to just talk to GlusterFS and GlusterFS will hide different file systems and storage types underneath. Various GlusterFS storage features like replication and striping will automatically be available for QEMU. Efforts are also on to add block device backend in Gluster via Block Device (BD) translator that will expose underlying block devices as files to QEMU. This allows GlusterFS to be a single storage backend for both file and block based storage types.

QEMU with native GlusterFS block driver

QEMU with native GlusterFS block driver

GlusterFS specifcation in QEMU

VM image residing on gluster volume can be specified on QEMU command line using URI format:

gluster[+transport]://[server[:port]]/volname/image[?socket=…]

gluster is the protocol.

transport specifies the transport type used to connect to gluster management daemon (glusterd). Valid transport types are tcp, unix and rdma. If a transport type isn’t specified, then tcp type is assumed.

server specifies the server where the volume file specification for the given volume resides. This can be either hostname, ipv4 address or ipv6 address. ipv6 address needs to be within square brackets [ ]. If transport type is unix, then server field should not be specified. Instead the socket field needs to be populated with the path to unix domain socket.

port is the port number on which glusterd is listening. This is optional and if not specified, QEMU will send 0 which will make gluster to use the default port. If the transport type is unix, then port should not be specified.

volname is the name of the gluster volume which contains the VM image.

image is the path to the actual VM image that resides on gluster volume.

Examples:

gluster://1.2.3.4/testvol/a.img
gluster+tcp://1.2.3.4/testvol/a.img
gluster+tcp://1.2.3.4:24007/testvol/dir/a.img
gluster+tcp://[1:2:3:4:5:6:7:8]/testvol/dir/a.img
gluster+tcp://[1:2:3:4:5:6:7:8]:24007/testvol/dir/a.img
gluster+tcp://server.domain.com:24007/testvol/dir/a.img
gluster+unix:///testvol/dir/a.img?socket=/tmp/glusterd.socket
gluster+rdma://1.2.3.4:24007/testvol/a.img

(GlusterFS URI description and above examples are taken from QEMU documentation)

Configuring QEMU with GlusterFS backend

While building QEMU from source, in addition to the normal configuration options, ensure that –enable-uuid and –enable-glusterfs options are is specified explicitly with ./configure script. (Update Feb 2013: A fix in QEMU-1.3 time frame makes the use of –enable-uuid unnecessary for GlusterFS support in QEMU)

Update Aug 2013: Starting with QEMU-1.6, pkg-config is used to configure the GlusterFS backend in QEMU. If you are using GlusterFS compiled and installed from sources, then the GlusterFS package config file (glusterfs-api.pc) might not be present at the standard path and you will have to explicitly add the path by executing this command before running the QEMU configure script:

export PKG_CONFIG_PATH=/usr/local/lib/pkgconfig/

Without this, GlusterFS driver will not be compiled into QEMU even when GlusterFS is present in the system.

Creating a VM image on GlusterFS backend

qemu-img command can be used to create VM images on gluster backend. The general syntax for image creation looks like this:

qemu-img create gluster://server/volname/path/to/image size

Example:

To create a raw image, qemu-img create gluster://1.2.3.4/testvol/dir/a.img 5G
To create a qcow2 image, qemu-img create -f qcow2 gluster://server.domain.com:24007/testvol/a.img 5G

Booting VM image from GlusterFS backend

A VM image a.img residing on gluster volume testvol can be booted using QEMU like this:

qemu-system-x86_64 -drive file=gluster://1.2.3.4/testvol/a.img,if=virtio

In addition to VM images, gluster drives can also be used as data drives:

qemu-system-x86_64 -drive file=gluster://1.2.3.4/testvol/a.img,if=virtio -drive file=gluster://1.2.3.4/datavol/a-data.img,if=virtio

Here a-data.img from datavol gluster volume appears as a 2nd drive for the guest.

Performance numbers

The following numbers from FIO benchmark are to show the performance advantage of using QEMU’s GlusterFS block driver instead of the usual FUSE mount while accessing the VM image.

Test setup

Host Dual core x86_64 system running Fedora 17 kernel (3.5.6-1.fc17.x86_64)
Guest Fedora 17 image, 4 way SMP, 2GB RAM, using virtio and cache=none QEMU options

QEMU options

FUSE mount qemu-system-x86_64 –enable-kvm –nographic -smp 4 -m 2048 -drive file=/mnt/F17,if=virtio,cache=none => /mnt is GlusterFS FUSE mount point
GlusterFS block driver in QEMU (FUSE bypass) qemu-system-x86_64 –enable-kvm –nographic -smp 4 -m 2048 -drive file=gluster://bharata/test/F17,if=virtio,cache=none
Base (VM image accessed directly from brick) qemu-system-x86_64 –enable-kvm –nographic -smp 4 -m 2048 -drive file=/test/F17,if=virtio,cache=none => /test is brick directory

FIO load files

Sequential read direct IO ; Read 4 files with aio at different depths
[global]
ioengine=libaio
direct=1
rw=read
bs=128k
size=512m
directory=/data1
[file1]
iodepth=4
[file2]
iodepth=32
[file3]
iodepth=8
[file4]
iodepth=16
Sequential write direct IO ; Write 4 files with aio at different depths
[global]
ioengine=libaio
direct=1
rw=write
bs=128k
size=512m
directory=/data1
[file1]
iodepth=4
[file2]
iodepth=32
[file3]
iodepth=8
[file4]
iodepth=16


FIO READ numbers

aggrb (KB/s) minb (KB/s) maxb (KB/s)
FUSE mount 15219 3804 5792
QEMU’s GlusterFS block driver (FUSE bypass) 39357 9839 12946
Base 43802 10950 12918

FIO WRITE numbers

aggrb (KB/s) minb (KB/s) maxb (KB/s)
FUSE mount 24579 6144 8423
QEMU’s GlusterFS block driver (FUSE bypass) 42707 10676 17262
Base 42393 10598 15646

Updated numbers

Here are the recent FIO numbers averaged from 5 runs using latest QEMU (git commit: 03a36f17d77) and GlusterFS (git commit: cee1b62d01). The test environment remains same as above with the following two changes:

  • The GlusterFS volume has write-behind translator turned off
  • The host kernel is upgraded to 3.6.7-4.fc17.x86_64

FIO READ numbers

aggrb (KB/s) % Reduction from Base
Base 44464 0
FUSE mount 21637 -51
QEMU’s GlusterFS block driver (FUSE bypass) 38847 -12.6

FIO WRITE numbers

aggrb (KB/s) % Reduction from Base
Base 45824 0
FUSE mount 40919 -10.7
QEMU’s GlusterFS block driver (FUSE bypass) 45627 -0.43

GlusterFS support in oVirt

While I described how to use GlusterFS as a storage backend for QEMU manually, there have been efforts to enable QEMU-GlusterFS native support from libvirt, VDSM and oVirt as well. We now have GlusterFS enabled completely from oVirt which allows user to use self-help portal of oVirt to create GlusterFS volume and use it as storage backend to host VM images. The GlusterFS storage domain work in VDSM and the enablement of the same from oVirt allows oVirt to exploit the QEMU-GlusterFS native integration rather than using FUSE for accessing GlusterFS volume.

Deepak C Shetty has created a nice video demo of how to use oVirt to create a GlusterFS storage domain and boot VMs off it.

UNMAP/Discard support in QEMU-GlusterFS

UNMAP support in QEMU-GlusterFS is explained here.


Trekking in Himalayas – Bara Bhangal expedition

October 16, 2012

After my last Himalayan trek to Auden’s Col in 2008, the only next opportunity came in 2012 when everything looked set for another sojourn to Himalayas. Naren and I started checking out the possible routes in the beginning of the year and Mayali pass trek excited us. Kaushal Desai from Above 1400ft who had taken us to Roopkund trek in 2006 was ready to arrange this trek, but he also suggested an equally challenging Bara Bhangal Trek in Himachal Pradesh Himalayas. Bara Bhangal trek looked very interesting to us since it involved an ascent to cross one pass, then a descent and a 2nd ascent to cross another pass. Given that Kaushal had already done this trek earlier, we decided to settle for Bara Bhangal. The trek was scheduled for end of September and beginning of Oct 2012. Given that this was the last trek of the season, we were assured of snow and extreme cold weather during the trek. Initially 6 of us planned to do this trek, but eventually only 4 made it. In addition to me, the group had Naren whose last trek to Himalayas was to Roopkund in 2006, Pradeep – my cousin for whom this was the first Himalayan experience and Vijay – Naren’s friend who had done Roopkund two years back.

We had a morning flight from Bangalore to Delhi and hence had to start quite early from home. Most of us had been busy with work lately and had stretched well into the night the previous day. We had a 5PM Dev Travels volvo bus from Delhi to Manali whose place of departure wasn’t fixed. We were told to board the bus from Sri Lanka Bhavan, Cannaught Place after we landed in Delhi. Accordingly, we proceeded to Cannaught Place, finished lunch in a restaurant and spent most of the afternoon in AC indoors rather than in dusty and hot Delhi outdoors. Coming from Bangalore, where both public and private bus transports operate very methodically, we were prepared for some chaos in the bus arrangements in Delhi, where double booking of seats seem to be the order of the day. The only improvement I observed this time was that the bus didn’t have any seat numbering and hence there was no scope for any claim and quarrel! We had to settle for one row backwards. On routes such as this (Delhi-Manali) where typically there are no repeat travelers, the bus operators rule the roost with utter disregard to customer quality of service. We reached Manali at 8AM the next day  and checked into Taj Palace Hotel.

Day 0 (29Sep). Acclimatization in Manali

Manali is at an altitude of 1850m, as per my altimeter watch. The Mall road in Manali is very inviting and we spent a few hours shopping warm clothing for our families. Next as part of acclimatization plan, we walked for around 2km and gained 150m altitude to reach Hadima temple. Hadimba temple is dedicated to Bheema’s Rakshasi-wife Hidimba. The temple has an interesting architecture and is made completely out of wood.

Hadimba Temple, Manali

Hadimba Temple, Manali

After lunch we met Kaushal who briefed us about the trek. We were a bit disappointed to know that he wasn’t accompanying us but his friend would be guiding us on the trek.  In the evening, we visited the Vashist temple at 2000m. There is a hot sulphur spring behind this temple where one could take bath. We prayed for a smooth trek in the adjoining Rama temple and decided to reach Manali by walk (around 2km). Shop keepers kept pestering us to visit them and have a look at “Chingu”, but none of them was ready to explain what chingu is. As per them, chingu can’t be explained but should be experienced by seeing! Finally out of curiosity we decided to checkout chingu and entered a shop. Chingu is essentially a blanket made of high quality wool (Pashmina goat’s probably) and it is apparently so expensive and valued that its not sold, but is instead rented out! The shop keeper went on and on about chingu and showed us all the valuable gifts we get if we were to rent out an expensive chingu. He was ready with address and contact details of hundreds of Indians and foreigners who had bought chingu on rent. Unwilling to be part of his fooled-buyers’ list, we came out.

Tension-free shop, Manali

No discount, no bargaining, no tension!

Late evening was spent in the gullies of Manali. The night turned to be nightmare for us when Naren and I discovered hundreds of bed bugs in our room. We were promptly shifted to an adjoining hotel at 11PM in the night while Pradeep and Vijay were having sound sleep in the next room!

Day 1 (30Sep). Manali to Lama Dugh

Lalith, Kaushal’s friend and our guide to Bara Bhangal trek, picked us up from the Hotel in the morning and we reached Kaushal’s house which is near the Hadimba temple. Lalith hails from Jibhi, Himachal Pradesh and is an accomplished mountaineer. He runs his own adventure treks and climbs too. We met our cook Hansraj and Horseman Shyam at Kaushal’s place. So totally we were a group of 7 with 4 members, 3 support staff and 1 horse and 8 mules. After tea at Kaushal’s house we hit the trail (2000m) at 10AM. The trail starts right in front of Kaushal’s house and proceeds into the pine forest. Thanks to our mini acclimatization expeditions yesterday, all of us felt comfortable in negotiating the initial steep uphill climb. At 11AM we reached a landmark called Bada Pathar (2260m) which was in the midst of a tiny forest clearing. Good views of Manali city can be obtained from this point. Indrasan peak was visible from here.

Trail leading to pine forest

Next part of the trail was again steep ascent in the pine forest and we reached 2800m at 1.15PM. From here, the trail is along the water pipeline that runs from above till Kaushal’s place. As we reached higher, the pine trees mostly gave way to plants which were head-high. We reached Lama Dugh camp site (2900m) at 2PM.

Campsite at Lama Dugh

The camp site had a small stream that served as our water source. The camp site was beautiful with greenery all around. Horses reached shortly afterwards. The tents were pitched and lunch (Biryani Rice) was provided at 3.45PM. After lunch we did a small uphill climb of 150m to help us acclimatize better. We could still get mobile signals from here. 150m above our campsite, there was a shepherd camping with his sheep. A shepherd dog followed us on the trail and even returned to our camp site.

Day 2 (01Oct). Lama Dugh to Riyali Thach

I got up at 5.30AM, the temperature was 7°C. We had roasted wheat bread and banana for breakfast and a hot cup of tea. One of the specialities of Roopkund trek with Kaushal was rich food and we had explicitly requested him to keep up the standards 🙂  It was 8.30AM by the time we started the day’s hike. Initial path led us up to the shepherd tent where we could see the Manalsu stream flowing down.  This stream would eventually join the Beas river. From this height, Dauladhar range was visible towards the West, Peer Panjal towards NE and Rohtang Pass was visible towards North.

Shepherd tent near Lama Dugh

We reached 3165m at 9.30AM and 3400m at 11AM. Now we were truly past the tree line. It was a continuous but gradual ascent and first glimpse of Kalihani pass was visible at 1.15PM from the height of 3565m. By now the horses went past us.  At 12.35PM we climbed on to a ridge and reached the highest point for the day at 3770m. It was extremely windy, but the surroundings were spectacular. Greater Himalayan range and Deo Tibba peak were visible from here. We spent a few minutes taking photographs from this ridge.

Next part of the trail was a continuous descent until the camp site. First we descended rapidly into a monstrous valley, crossed a stream and continued the descent. We could see our kitchen tent down below towards our right and were about to descend straight when we saw our guide waiting for us at a vantage point towards our left. Apparently he had chosen a bit more circuitous but easier route to descend. A group of Britishers led by a Nepali Sherpa had joined us by now.  All of us reached Riyali Thach camping site (3335m) together at 2.30PM. We were welcomed with a cup of juice and later tea. Naren looked tired and he attributed that to lack of lunch at the right time.

The camp site was beside a stream which served as our water source. After lunch we did a small acclimatization ascent of around 150m.

I had a sumptuous dinner of roti-subzi and dal-chawal, hit the bed at 8.30PM and had a good night’s sleep.

Day 3 (02Oct). Riyali Thach to Kalihani base

I got up at 5.45AM, the outside temperature was 3°C. It was too cold to attempt brushing of teeth. Bread, fruits and cornflakes were provided for breakfast. At 8.30AM we started the trek. First we had to cross a big herd of sheep. The initial part of the trail involved walking alongside a stream that would eventually join the Fulez Nala. At 10AM we went beyond a snow bridge and crossed over to the other side of the stream. I slipped over during river crossing resulting in both my shoes becoming completely wet. Fortunately it was very sunny and I could manage with a change over set of socks from Lalith.

Ridge enroute Kalihani base

At 11AM we reached a height of 3500m and we were on a ridge. From here it was a continuous walk along this ridge. By this time I was in a rhythm and left others behind to continue the ascent in my pace. I reached 3600m at 11.30AM and 3700m at 11.45AM. Here the ridge ended and a steep ascent over a zig-zag path started. I reached 3800m at 12PM and Pradeep joined me here for finish the packed lunch which we were carrying. Naren had finished his lunch much earlier and was experimenting with a slow pace and hence reached a few minutes late. 5min ascent further led to a flat land (plateau) from where our campsite was visible. We spent a few minutes to enjoy the sunny midday and took photographs of the surrounding mountains. We reached Kalihani base camp (3825m) after crossing a moraine at 1.30PM.

Indrasan (left) and Deo Tibba (right) as seen from Kalihani base

Day 4 (03Oct). Crossing Kalihani pass

At 3AM I was woken up by the sound of sleet shower hitting the tents. Heavy snow or shower would reduce our chances of crossing the Kalihani pass today, but fortunately it lasted only for around 30min. We got out of the tents at 4.30AM, the temperature was 2°C. The surroundings were all white, thanks to the snow. We finished early breakfast of oats and tea and got all set to hit the trail. Fortunately as per Lalith’s estimate, the clouds subsided and Sun rose behind the Deo Tibba at 6.15AM. We started the hike at 6.30AM.

First bits of snow enroute Kalihani Pass

The initial part of the trail was an ascent through moraine and I reached 4020m at 7.10AM. I spent over 20min here for rest of the group to catch up. Next stretch of the trail involved steep ascent over scree. At 9.30AM, I touched the snowline at an altitude of 4435m. I saw Lalith’s footsteps but he was nowhere to be seen. Rest of the group was far behind. I had reached a flat ground covered with snow and finally saw Lalith at a height exploring the route towards the pass. Rest of the group joined us after half an hour.

A View enroute Kalihani Pass

Mountain range seen from Kalihani Pass

The next part of the trail was through moraine with snow. I reached 4450m at 11AM. The going was tough since the altitude was making our progress slow. As we gained height, we were mostly walking over a feet of snow. Naren and Pradeep were suffering from mild altitude sickness. While Pradeep closely followed me, Naren’s progress was slow. Vijay was coming with Naren. We still had to walk 5hours on the other side of the pass to reach our designated camping ground. Horseman was keen to know if he should stop the horses at some intermediate point or should make it to the eventual camping ground at Devi Ki Marhi. After some deliberations with Lalith, we decided that we will do the full day trail today and camp at Devi Ki Marhi. Since the progress of Naren and Vijay was too slow, I decided to stick with them to motivate them to move fast.

The Kalihani Pass

At 12PM, we finally made it to Kalihani Pass at 4645m. The packed lunch was finished on the Pass. We spent around 30min on the Pass and started getting down on the other side. The initial trail was a gradual descent through the snow. We went beside 3 beautiful lakes during the descent. We had now joined Kalihani Nala and were mostly walking beside it over the moraine. It was a long and tiresome walk over the boulders. At 3.30PM we reached the place (4055m) where the British group had camped. They offered us hot tea which was badly needed by us. Devi Ki Marhi camp was still 3 more hours away and one trekker from the camp remarked that we are indeed brave to attempt that. I wasn’t sure if we were brave or stupid at that time 🙂

The descent

Next it was a long monotonous walk alongside the river. Horses were well ahead of us and Lalith was nowhere in sight. Pradeep, Vijay and I were walking together in an increased pace while Naren was trailing behind. As we progressed ahead, we completely lost both Lalith in the front and Naren at the back. Many times we would see a huge rock at a distance that would appear exactly like our kitchen tent but only to be fooled later. The walk turned a bit more challenging when sleet shower started at 5.30PM. Though we were equipped to walk in sleet shower or snow or rain, it was seriously hindering our visibility as it was turning dark quickly. We finally saw Lalith coming back in search of us. After 30min walk in the shower were came across a shepherd shelter (built of stones) where we decided to wait for Naren and Lalith.

Lake on the other side of Kalihani Pass

Shepherd tents are so well designed that one could easily spend a night in them in times of emergency. There was a fireplace within the shelter. Lalith started the fire and left us to find out how far our campsite was. We sustained the fire till 8.30PM when Lalith was back with Hansraj who served us much needed noodles. Naren had high fever by this time and had already vomited once. It took some amount of persuasion to get him consume food and get started towards our campsite. We left the shepherd shelter at 9PM and slowly started moving towards Devi Ki Marhi. The going was very tough since our pace was too slow for the guide and it was taking huge effort for Naren to even move a step further. He was feeling so sleepy that given to himself, he would have slept there on the trail itself. Again it took us some effort to motivate Naren and it was 11.15PM by the time we reached the camp site of Devi Ki Marhi at 3555m. The only positive aspect of the night trek was the spectacular moonrise behind a peak, the moonrise never looked more beautiful. Looking back we feel we had done 2 days or at least 1.5 days worth of hike in a single day. Kaushal’s original itinerary for this day was a bit over-optimistic.

Day 5 (04Oct). Devi Ki Marhi to Dal Marhi

Previous day had been very tough and I got up late at 6.15AM, the temperature was -5°C.  As usual our camp site was on the banks of a stream (Kalihani Nala) and I could see our horses grazing at a height across the Nala near Devi Ki Marhi temple. (Temple was just a miniature shelter for God built of stones). I could also see horseman enjoying the morning camp fire near a shepherd tent across the Nala. I quickly joined him since any fire is so inviting at such chilling temperatures.

Horseman Shyam enjoying early morning fire near a shepherd shelter

Devi Ki Marhi campsite

I was apprehensive if everyone in the group would have enough energy today to even start the day’s hike, but fortunately everyone including Naren who was still suffering from fever, agreed to continue the hike and shift our base to the next camp site Dal Marhi today. We started at 9.30AM and crossed a stream over a bridge which consisted of flat stones laid over two pine logs.

We were walking alongside this stream

The day’s trek mostly involved walking beside the river and crossing moraines every now and then. We crossed a few streams that were joining the main Kalihani Nala. At one place, I removed my shoes and crossed the river barefoot for the fear of accidentally soaking my shoes. At around 11.45AM we were atop a small hill measuring 3415m. The highlight of the day’s trek was crossing a land slide area which was so slippery since the loose soil barely provided any grip.  I crossed the 10ft stretch by sitting down while rest of the group took Lalith’s help.  At 2.15PM, I had reached the base of a ridge at 3275m which required straight ascent. It took half an hour to reach the top at 3450m. We had to immediately descend on the other side, cross a stream that required Lalith’s help and then continue the ascent. It was Naren’s turn this time to soak the shoes in the river!

Vijay negotiating the landslide area

Pradeep and I gradually increased our pace and reached the Dal Marhi camp site (3660m) at 4.15PM. Naren and Vijay came in after 30min. The camp site was beautiful with Dauladhar range behind us. We were in Himayalan Bear territory now and we could see bear excreta at many places in the campsite as well as the trail.

Day 6 (05Oct). Dal Marhi to Bara Bhangal

Naren wasn’t feeling well through out the night and finally I had to wake up the cook at 5AM to prepare hot tea for him. I couldn’t get sleep again and roamed around the campsite for a bit before settling down to do some Pranayama. I was distracted by the sound of stones falling down from the landslide area in front of our camp site. The camp site was surrounded by high-rise hills and hence sun rays didn’t hit the campsite even hours after sunrise. -1°C was recorded at 7.30AM. Nothing usually gets done until Sun is out and hence we had a delayed start at 9.45AM.

Dauladhar range as seen from Dal Marhi

The day’s hike started with a steep ascent but was mostly downhill for rest of the day. At around 11.30AM, we had reached tree line and were walking beside rhododendron, birch and pine trees. I really dislike descending since its so hard on knees. Stepping down sideways to avoid pressure on knees would eventually lead to pain in the ankles. At 3PM we had reached 3000m after which we did a 300m straight descent. Naren was doing well and was leading the group while I remained at the end.  At 4PM we had reached the outskirts of Bara Bhangal village where we could see the cultivated fields. Pradeep showed us Ganja plants and it appeared that the villagers were growing them freely everywhere. The final descent was very tough and tiring, I eventually reached Bara Bhangal village (2455m) at 4.30PM.

Shephard shelter enroute Bara Bhangal

The campsite was beside a Government school. Ravi river flowed across our campsite. Pradeep got acquainted with the teacher of the school and brought him to our camp site. Masterji was kind enough to escort us through the village by-lanes. Bara Bhangal has a population of around 750, limited power thanks to a mini-hydel project and a satellite phone that works for 3 hours a day. We could see people mainly growing Rajma and corn in the village.

Bridge across Ravi river in Bara Bhangal

To reach any village or city from Bara Bhangal one has to either go back and climb Kalihani Pass or climb and cross the Thamsar Pass. So having come till Bara Bhangal, we had no easy exit paths, we had to cross another pass to get back to civilization. This hardship came as a surprise to some of our team members but that only increased their resolve to complete the trek and get back to a safer place!

Day 7 (06Oct). Bara Bhangal to Mardh

Camping at low altitude in Bara Bhangal gave me an opportunity to brush my teeth and wash my face after a gap of 5 days!  We started the hike at 8.15AM. First we crossed the Ravi river and started ascending on the right bank of a stream. The ascent was mostly steep and all of us were doing pretty good since we had acclimatized well by now. Today our guide didn’t start with us and we were all by our own. There was a path leading down to the stream and a possible opportunity to cross over to the left bank of the stream, but we ignored it owing to the sparse usage of the path. We continued to ascend on the right bank of the stream and reached 3050m at 11AM. By now we could see a clear path on the left bank as well. Unable to decide the route, we waited for the guide. The guide and the horses appeared at 12.45PM, but on the left bank of the stream. We had lost precious 1 hour 45min in the mountains and we were on the wrong side of the river! It was impossible to cross the stream due to strong currents and we had to climb further up and cross over at a more safer point.

A peak seen from Mardh

We finished lunch and started the further ascent through moraine that consisted of huge rocks. By 2PM we were pleasantly surprised to find ourselves in the campsite of Mardh (3400m). This camping site was apparently chosen to provide grazing opportunity for the horses which wouldn’t have been possible if we had camped any higher and nearer to Thamsar pass.

Pradeep and Vijay (Right) helping Lalith (Left) to pitch the tent at Mardh

Having reached earlier than anticipated, Pradeep, Vijay and I did a small acclimatization trek to keep us in good shape for the big day tomorrow. Vijay was in mood for some talk and he described the life around the Ranganatha Temple in Srirangam, Trichy, TN and also gave us a good account of the history of Sri Ranganatha during the times of Muslim onslaught. Spirits were up in the kitchen tent too and Lalith prepared Pakodas for us!

Day 8 (07Oct). Crossing Thamsar Pass

We had a very long day when we crossed the previous Pass (Kalihani) and hence wanted to make sure that we start early this time to give us enough buffer time. I got up at 5AM, the temperature was 0°C. After breakfast that involved oats and cornflakes, we started the trek at 6.30AM. We made good progress initially and reached 3800m at 8.15AM. Next we crossed over to the other bank of the Thamsar stream and reached a shepherd tent that could have served as a campsite. Looking back I felt we should have camped here the previous night which would have given a good head start for attempting the Pass. Lalith spotted 4 grazing animals on the top of a high mountain. They were appearing as tiny black dots to the naked eye. We suspected them to be bears, but on camera they appeared more like yak than bear.

We crossed these mountains via Thamsar Pass

At 9.45AM, we had reached 4030m from where Thamsar Pass became visible clearly. Next we had to cross a tiny stretch of glacier and another 100m ascent took us to the banks of a huge lake.  We had an early lunch here and started the ascent beside the beautiful blue lake. At 12.15PM, we touched snowline at 4300m. At 12.45PM we were at the base of  Thamsar Pass with a frozen lake (4375m) separating us and the Pass. Unlike the approach to Kalihani Pass, the final approach to Thamsar Pass was easy and it involved a gradual ascent path that circled around the frozen lake.

Way to Thamsar Pass

Lalith was the first to reach Thamsar Pass (4520m) followed by me at 1.20PM. I thanked the Mountain God stationed on Thamsar Pass for allowing us the safe passage. The weather which was kind till now suddenly changed. We could spare only 20min on the Pass and sleet shower started. All of us practically rushed down on the other side. It was continuous descent from here onwards and we reached a possible campsite (3900m) at 3PM. The horseman had a better campsite in mind and we continued the descent. By now the sleet shower had turned into snow fall and was gradually getting heavier. Thanks to the snow, reaching a campsite was the only thing in our minds and hence all pain in the knees and ankles were almost forgotten! We reached an empty shepherd house (3400m) built of stones at 4PM. It had two rooms and served as good shelter for us in the night. The snowfall continued almost till 5PM. As per Kaushal’s itinerary we were supposed to reach Plachak camp site today. From Mardh, it involves an ascent of 1km and a descent of 2km which is probably too much to cover in a day.

Mountain range seen from Thamsar Pass

Clouds closing in from the other side of Thamsar Pass

We had crossed two mountain passes successfully and celebrated the achievement by lighting a huge campfire. All the firewood in the shepherd shelter was consumed in the night. The taste of piping hot ginger lemon water with honey on a moonlit night in the mountains will remain with me for years to come!

Day 9 (08Oct). Back to civilization

We started the last day of the trek at 8.30AM in the morning. As per itinerary, we were supposed to reach Billing by walk and then proceed to Bir and from there Dharamshala. But since that was too much for a day, our horseman Shyam suggested an alternative – reach Bada Gaon and from there travel on road to Dharamshala. The last day’s trek is usually tricky since there is no challenge to look forward to. When you are expecting a quick finish to the day, the long walk to the destination could mentally wear you down. This happened with us too, walk to Bada Gaon turned out to be a long one.

The proud group L2R – Hansraj the cook, Naren, Shyam the Horseman, Me, Pradeep and Lalith our guide

Its business as usual for these Bara Bhangal kids who cross Thamsar Pass every year before the winter sets in!

We had to cross 4 ice bridges over Barot Nala today and that was the only excitement to otherwise monotonous descend through pine forests. We reached Plachak (2565m) at 11.40AM. From Plachak, the route crosses over to the right bank of the river and continues further in the pine forest. At 2PM we crossed over to the left bank over a bridge and from here it was an hour’s walk to our destination. We were at Bada Gaon (2270m) at 3PM. Lalith had to hitchhike in a car to reach Barot to fetch us a taxi from Bada Gaon to Ghattasni. At Ghattasni we changed over to another taxi to reach Dharamshala at 10PM.

Views from Kalihani Pass

Views from Thamsar Pass

More photos here

Day-wise stats

Day Start time End time Start altitude End Altitude Max altitude reached
1. Manali to Lama Dugh 10AM 2PM 2000m 2900m 2900m
2. Lama Dugh to Riyali Thach 8.30AM 2.30PM 2900m 3335m 3770m
3. Riyali Thach to Kalihani base 8.30AM 1.30PM 3335m 3825m 3825m
4. Crossing Kalihani Pass 6.30AM 11.15PM (with 3hrs break) 3825m 3555m 4645m
5. Devi Ki Marhi to Dal Marhi 9.30AM 4.15PM 3555m 3660m 3660m
6. Dal Marhi to Bara Bhangal 9.45AM 4.30PM 3660m 2455m 3660m
7. Bara Bhangal to Mardh 8.15AM 2PM (with 1.45hrs break) 2455m 3400m 3400m
8. Crossing Thamsar Pass 6.30AM 4PM 3400m 3400m 4520m
9. Thamsar base to Bada Gaon 8.30AM 3PM 3400m 2270m 3400m