immutability
Posted: January 10, 2013 | Author: Sideb0ard | Filed under: abstraction, big data, database, future, ideas, programming, video | Comments OffExcellent talk from Pat Helland, “Immutability Changes Everything” :
some fav Velocity talks
Posted: July 2, 2012 | Author: Sideb0ard | Filed under: big data, configuration management, cybernetics, devops, engineering, hacking, internet, resiliency, video | Comments OffVelocity was on last week, and I was following along enviously on twitter – I’m making a promise to myself that I’ll be along in person next year!
Quite a few of the talks now appearing online – here’s a couple I’ve watched so far..
Graylog2 and Party Gorilla
Posted: June 7, 2012 | Author: Sideb0ard | Filed under: big data, cloud, logging | Comments Off
I discovered Splunk about 6 months ago, and aside from the unfortunate name and the truly evil pricing model, I was quite taken with the app itself, a searchable realtime interface to a centralized logging server.
I read about Logstash and Graylog2 a few months back, which seemed to offer a similar functionality, but just had never had the time or opportunity to implement it until now. There are many moving pieces so it can be a bit confusing at first, but the confusion simply stems from the flexibility of how you configure the components. I found the explanation on this blog piece to be the simplest way to explain how the components can be connected.
Graylog2 by itself can take the place of your central Syslog server. It utilizes quite a few other pieces of software to run – Elasticsearch, MongoDB and some ruby gems for the web interface, so even in this most simplified setup, there are still quite a few steps to getting it up and running. There are several tutorials online and linked from the Graylog2 website – the one I followed, and would highly recommend was this blog post – took me a few hours from initial futzing around to having a working setup.
If you already have a central Syslog server up and running, you might not be so keen to change all your hosts (why not? you’re running Puppet surely! :) – in which case, you can configure Rsyslog or Syslog-ng to forward to your Graylog2, or alternatively, use Logstash to parse, optionally transform and forward your syslogs. This post has some decent info on setting up like this.
But of course, the most important thing is that they have a “motherfucking party gorilla with a motherfucking party hat” for their logo, and how can you argue with that??!
( Here’s why! )
Backblaze
Posted: May 16, 2012 | Author: Sideb0ard | Filed under: big data, cloud, storage, video | Comments Offthe legendary Mike Hoover turned me on to Backblaze a few months back – they’re a cloud storage provider, who opened up the design for their chassis and storage pod solution so you can build your own “Storage Pod 2.0: a 135-terabyte, 4U server for $7,384″ (blog post here)
Like Treacle, Solving Perl Net::SFTP Slow Transfer Speeds
Posted: February 19, 2012 | Author: Sideb0ard | Filed under: big data, cloud, distribution, internet, networking, perl, programming, unix | Comments Off
I’ve been trying to track down problems with really slow network transfer speeds between my servers and several DSPs. I knew it wasn’t local I/O, as we could hit around 60Mb/s to some services, whereas the problematic ones were a sluggish 0.30Mb/s; I knew we weren’t hitting our bandwidth limit, as cacti showed us daily peaks of only around 500Mb/s of our 600Mb/s line.
I was working with the network engineer on the other side, running tcpdump captures while uploading a file and analysing that in Wireshark’s IO Graphs – stream looked absolutely fine, no lost packets, big non-changing tcp receive windows. We were pretty much stumped, and the other engineer recommend i look into HPN-SSH, which does indeed sound very good, but first i started playing around with trying different ciphers and compression.
Our uploads are all run via a perl framework, which utilises Net::SFTP in order to do the transfers. My test program was also written in perl and using the same library. In order to try different cyphers i started testing uploads with the interactive command line SFTP. Boom! 6Mb/s upload speed. Biiiig difference from the Net::SFTP client. I started playing with blowfish cipher and trying to enable compression with Net::SFTP – it wasn’t really working, it can only do Zlib compression, which my SSHD server wouldn’t play with until i specifically enabled compression in the sshd_config file.
After much more digging around, i came across reference to Net::SFTP::Foreign, which uses the installed ssh binary on your system for transport rather than relying on the pure perl Net::SSH.
Syntax is very similar, so it was a minor rewrite to switch modules, yet such a massive payback, from 0.30Mb/s up to 6Mb/s.
(It turns out the DSPs i mentioned earlier who could achieve 60Mb/s were actually FTP transfers, not SFTP)
real-time stats >
Posted: July 20, 2011 | Author: Sideb0ard | Filed under: big data, future, networking, real-time, twitter, video | Leave a comment »Found this Facebook engineering video quite fascinating, a nicely detailed platform overview of FB’s new real-time analytics system:
In the comment section, someone posted a link to a somewhat similar presentation from Twitter about their real-time solution::
(hello Johan!)
zettabytez
Posted: June 24, 2011 | Author: Sideb0ard | Filed under: big data, future, infographic | Leave a comment »yow!
from cisco blog
Curtis and Foo
Posted: June 20, 2011 | Author: Sideb0ard | Filed under: big data, cybernetics, future, twitter | Leave a comment »
Very nice article & thoughts by Greg Borenstein over at Ideas for Dozens, covering the recent O’Reilly FOO camp, Adam Curtis documentaries and the rise of Big Data thinking..