Ran into an interesting problem today where I was getting ‘dict is not callable’ in a python project I was working on. The problem line looked innocent enough:
if type(foo) == string:
After a bit of digging, it turned out that I had accidentally overridden the built-in type function within the scope of the function I was working on:
for type in someArrayOfTypes:
Stumbled across this and had a good laugh. I used to support a large distributed zabbix environment where each zabbix node was back-ended by a mysql database. Constant pain. Everything from general performance problems (ui is slow again? better run another optimize job) to various application problems (hey, replication is failing and the app logs are hemorrhaging various query failures, sweet!).
After switching all of our performance data capture to fixed-size databases (first rrd via pnp4nagios, later graphite back-ended by whisper), IMHO there is no reason to ever suffer storing performance data (or really any other high-turnover, high-write-rate datasets) in a relational database ever again. Constant growth means eternal maintenance work (ref. the optimize jab earlier =D), and working with fixed-size databases end all of that pain.
Thanks to one Daniel Miessler’s awesome work with his VimBlog plugin, I apparently can now write blog entries from vim. I think my nerd cred just rose by about 5000 points. =D
Credit where credit is due.
Back when I was messing around with emacs, one of my favorite features was rgrep. The rgrep function allowed you to recursively grep an arbitrary directory for a pattern. The matches (eg. the grep output) appeared in a new buffer, and when you browsed through the matches, the files would automatically open in a split with point on the match itself. Really handy.
Vim has a similar built-in feature called vimgrep, which is native to vim. However, it’s typically a bit slow, probably because it’s native vimscript and I don’t know if it any intelligence in terms of skipping unlikely files (eg. large binaries, various git dotfiles).
Enter the silver searcher, plus the ag plugin to run the tool from within vim. Now you can search large directories/repos almost instantly, then interactively browse through the matches in the quickfix buffer. Really cool stuff!
In a long-forgotten age, the arch unix wizards bestowed upon us the mighty select system call, which allows us to poll file descriptors for their read/write availability so we don’t block before trying to read/write. And as with many *nix os calls, there’s a native python interface.
I recently built a simple scripted daemon fo running a command indefinitely, periodically checking the output, and sending passive check results to icinga in the event of any problems found (omping is the command in question in case that matters =). I was looking into several solutions for running this process asynchronously and polling the output in a non-blocking way, and I ran into a couple of snags along the way.
My first go-to for this sort of thing is typically the Popen method found within the subprocess module in Python. However, while you can asynchronously spawn new processes and poll() them for completion, there seems to be no way to read the process’ stdout without blocking until the process completes (or receives a SIGKILL/SIGTERM or otherwise dies). My quest led me to Twisted, a powerful (and somewhat intimidating) framework for asynchronous event-driven programming.
After working with subprocesses in Twisted for a bit, I realized that there had to be an easier way to solve this somewhat straightforward use case. Then I figured, why not use a named pipe? Trivial to implement one-way inter-process communication. If it works for icinga with thousands of passive checkresults submitted per second, it has to work here too.
A simple fork() (one instance to run the command and write to the pipe, and another to read said pipe and handle output) and everything worked perfectly. Huzzah! Score another victory for the unix way? =D
Well the emacs journey was fun and I managed to pick up a few things, but I’ve moved back to Vim. Modal text editing is far faster than the alternative, and while there are several plugins in emacs that emulate the vim experience (evil probably being the best), none of it comes close.
Also, after getting syntastic up and running, I have no idea how I’ve ever developed stuff without a linter before. It’s almost essential in a dynamically-typed language like python where you wind up catching most syntax errors during runtime. What’s cool is that syntastic stores errors found in a location list buffer so you can easily navigate to where the problems are, all while never leaving the keyboard.
Apparently there are python hooks to extend Vim with so you’re not just stuck with native vimscript. It’s amazing what stuff people have come up with. The best plugin source I’ve found so far is vimawesome.com. After installing the Vundle plugin, you can just add git links into your .vimrc and you’re a :PluginInstall away from installing everything. Cool!