There is nothing wrong with your set: Why rsync is slow for large files 
There's a lot of confusion and badly written questions returned when you Google for "rsync incremental file transfer". Most of the babble is related to command-line fail when doing incremental backups of a directory tree. Of course "rsync -a" skips files that are unchanged between the source and destination. But that may not be what you mean by "incremental file transfer". What you might mean is - "why is rsync taking so long to transfer a large file that is unchanged except for a few lines." A good example is a log file that is getting appended to.

For this we have one of the great things about rsync: the "Tridgell Algorithm" (www.samba.org/~tridge/phd_thesis.pdf) Unless you specify otherwise, rsync will try to transfer only the parts of existing files that have changed since the last copy.

You might think this would speed up your large file transfer. But in today's world of fast networks and (usually) underpowered virtual hosts, it usually doesn't. The Tridgell Algorithm is CPU intensive, and rsync is single threaded (probably by necessity, since using multiple threads to access a single file sequentially would be rather pointless.)

Example: Times for copying a 4 1/2GB file over a fast network on a slow server:

time rsync -va (etc):
delta-transmission enabled
4763951706 100% 125.50MB/s 0:00:36 (xfer#1, to-check=0/1)
total: matches=69025 hash_hits=103974 false_alarms=0 data=130595
sent 552238 bytes received 406893 bytes 8093.93 bytes/sec
total size is 4763951706 speedup is 4966.95
real 1m58.382s
user 0m38.945s
sys 0m11.891s

time rsync -Wva:
delta-transmission disabled for local transfer or --whole-file
native_stderr.log
4763965456 100% 56.92MB/s 0:01:19 (xfer#1, to-check=0/1)
total: matches=0 hash_hits=0 false_alarms=0 data=4763965456
sent 30 bytes received 4764547173 bytes 50418488.92 bytes/sec
total size is 4763965456 speedup is 1.00
real 1m33.719s
user 0m46.032s
sys 0m26.128s

There are even inconsequential differences for a small file:

rsync -va:
delta-transmission enabled
1059189 100% 202.02MB/s 0:00:00 (xfer#1, to-check=0/1)
total: matches=1034 hash_hits=1038 false_alarms=0 data=373
sent 6240 bytes received 4679 bytes 7279.33 bytes/sec
total size is 1059189 speedup is 97.00
real 0m0.290s
user 0m0.044s
sys 0m0.013s

rsync -Wva:
delta-transmission disabled for local transfer or --whole-file
trace.log
1066221 100% 48.42MB/s 0:00:00 (xfer#1, to-check=0/1)
total: matches=0 hash_hits=0 false_alarms=0 data=1066221
sent 30 bytes received 1066517 bytes 711031.33 bytes/sec
total size is 1066221 speedup is 1.00
real 0m0.246s
user 0m0.042s
sys 0m0.022s

So there you go, why you don't see much improvement from rsync's delta-transmission mode. THIS IS THE WAY IT IS SUPPOSED TO WORK. And the man page says as much:

-W, --whole-file
With this option rsync’s delta-transfer algorithm is
not used and the whole file is sent as-is instead.
The transfer may be faster if this option is used when
the bandwidth between the source and destination
machines is higher than the bandwidth to disk (espe-
cially when the “disk” is actually a networked
filesystem). This is the default when both the source
and destination are specified as local paths.

So, we love Tridgell's Algorithm, but is isn't always faster.


[ view entry ] ( 121 views )   |  permalink
Welcome "Pinkpi", a Raspberry Pi Wireless AP 
Wow, that was easy: http://learn.adafruit.com/setting-up-a- ... cess-point

I didn't have to specify a driver in hostapd.conf because Raspbian 7 knows about the Wifi dongle already (RT5370) and although Ladyada suggests rebuilding hostapd the v1.0 that comes with the distro seems to work.

That USB dongle is a $9 Ebay-crap winmodem that I hope to get working. So I can take Pinkpi on trips to the backs of beyond that don't have intenetz, only dialup.

[UPDATE: There is no hope of getting a winmodem working on a Pi, unless you know how to write codecs. The various "linmodem" projects are highly dependent on the X86 architecture. If you want a winmodem really cheap, let me know, this one is useless. Pppd does work on a Pi, if you have a modem that talks serial. You can use a serial USB dongle, or, for the more adventurous, the Pi GPIO pins.

[ view entry ] ( 446 views )   |  permalink
Parted Is a Time Machine 
The rules for making new filesystems are only getting more complicated in RHEL/Centos 6. It's like going back in time to 1990...

[ view entry ] ( 349 views )   |  permalink
"Volname" replacement for Fedora 
The 'volname' command disappeared from Fedora a couple releases ago. Instead, use 'blkid':
#!/bin/sh
DEV=/dev/cdrom
volname=`/usr/sbin/blkid $DEV | sed -e 's/^.*://'`
eval $volname
echo $LABEL


[ view entry ] ( 328 views )   |  permalink
A Sub-$500 Haswell White Box 
I needed to replace my ancient single-core Athlon box running CentOS 5. It's been a trouper, but it's really slow now. CentOS 5 only has Firefox 17 in their repo, and that Firefox is too brain-damaged to use the
box's GPU card, so graphic-intensive apps like Google Maps and big bloated sites like Facebook just barely run.

The objective was to build a Haswell-based whitebox for less than $500, and I did it. I had to cut a few corners, like getting a cheap-ass motherboard, and an i5 instead of an i7. They still make motherboards with PS2 keyboard and mouse connectors! I thought those had gone long ago. So I can reuse my IBM Model M keyboard and my Wells Fargo mouse with the floating dog:


Components:

Intel Core i5-4670 Processor - Quad Core, 6MB L3 Cache, 3.4GHz, $215. To stay under $500 I had to settle for an i5 instead of an i7. I'd rather have a 2-core i7 than a 4-core i5, but this one will work just fine. (2 core i5's and i7's are not available in a desktop form factor anyway.) It's the fastest i5 available at this time and it doesn't heat the room up at idle like the Athlon did.

MSI B85M-P33 Intel B85 Motherboard - MicroATX, LGA 1150, Intel B85 Chipset, $65. This is a really cheap motherboard but I don't need anything fancy. It has "Military Class 4 Technology", "Dark Choke", and "Solid Cap!" W00t.

Seagate Barracuda 1TB Hard Drive, 7200 RPM, 64MB cache, $70. A disk is a disk is a disk, although 64MB cache is better than 32GB. 7200 RPM is perfectly fine for a home system, more RPMs will just generate more heat and fail sooner. I am just astonished how disks just get cheaper and cheaper.

LG Electronics 24X SATA DVD+-R, $18. You need one for backups.

Patriot Viper Xtreme 8GB DIMM, DDR3, 1600MHz, PC3-12800, $85. Cost more than the mobo! You need at least 4GB to do anything, 8GB is better. Running a basic KDE desktop seems to use about 2 to 3GB. You might need 16GB if you do a lot of industrial scale compiling or virtualization. With 8 I can do some basic experimentation with KVM, Xen, etc.

Power supply and case: Reused from previous projects. The power supply is a 300W with the old ATX-1 20-pin P1 and 4-pin P2 but works fine for the modest load of my modest configuration, which should be pulling no more than 120W. If you do this make sure your mobo has a P2 connection and watch your power budget. You need P2 or a modern PSU since the one 12V pin on the old ATX-1 P1 can't handle more than about 8A without getting hot.

$2 for a PATA to SATA power adapter. 2 SATA data csbles came with the mobo.

Keyboard and Mouse: See above. (My model M is really a 90s vintage Lexmark Model M, sorry.)

Some gotchas:

VMware ESXi would not load on this mobo, the Realtek 8111B ethernet was not supported. Centos 6 had some problems with the 8111B but Realtek has the latest driver online and it's easy to install. Fedora 19 works fine with the 8111B out of the box.

The Haswell CPU series is so new that the i5 GPU isn't fully supported by the Centos 6 i915 driver. (It works, but you can't adjust gamma, which makes it fairly useless.) The i915 driver in Fedora 19 is OK, though. So I'm a born again Fedora user.

The MacAlly USB mouse I've been using for years doesn't work with either Fedora or CentOS and this mobo. Whatever, it's a dumpster-dived mouse. Sooner or later, I'll get a new freebie at a trade show or find one in the trash. Only suckers pay for mice.

The mobo BIOS and lm_sensors report 10 deg C different temps for the CPU. The CPU heat sink is not even sensibly warm, so I don't care too much about this. The BIOS says the fan is turning at only 1700 RPM when idle - so this PC is really quiet. Hoo-ray.

The mobo's fan and case temp sensors are not detected by lm_sensors. Not a problem.

The fan was bad in the CPU cooler Intel shipped with the CPU. With typical excellent Intel service, support didn't ask any dumb questions and they replaced it lickety split and with zero hassle.

Aaarrgh, fonts! Fonts suck AGAIN. Nothing seems to look quite right. It took me a while to get them right on the old box, too (see my previous entries.) Part of the problem is that Firefox still does its own rendering and hinting seems to be broken. Chrome is even worse, since you can't micro-manage the font and DPI settings like you can in Firefox. KDE and its apps, on the other hand, seem to be OK. Like all open-source things, this PC is a work in progress. Forward...

[ view entry ] ( 455 views )   |  permalink
Welcome "Greenpi", a Raspberry Pi Airplay adapter for the stereo 
I did something useful and computer-related for the house! Raspberry Pi-based Airplay adapter. Costs a little more than a used Airport, but less than a new one:

$40 - Pi
$15 - Wifi dongle
$6 - case
$? - 5V regulated wall wart (dumpster dived)



[ view entry ] ( 411 views )   |  permalink

| 1 | 2 | 3 | 4 | 5 | 6 | 7 | 8 | 9 | Next> Last>>