Lesson #1: Don't code when you're distracted.
Some hours later, the problem manifested. The queue workers came down, and AR (which is totally dependent on them for its core functionality) immediately stopped doing the thing customers pay me money to do. My monitoring system picked up on this and attempted to call me -- which would have worked great, except my cell phone was in a box that wasn't unpacked yet.
Lesson #2a: If you're running something mission critical, and your only way to recover from failure means you have to wake up when the phone rings, make sure that phone stays on and by you.
Later that evening I felt a feeling of vague unease about my change earlier and checked my email from my iPad. My inbox was full of furious customers who were observing, correctly, that I was 8 hours into an outage. Oh dear. I ssh'ed in from the iPad, reverted my last commit, and restarted the queue workers. Queues quickly went down to zero. Problem solved right?
Lesson #3: If at all possible, avoid having to resolve problems when exhausted/distracted. If you absolutely must do it, spend ten extra minutes to make sure you actually understand what went wrong, what your recovery plan is, and how that recovery plan will interact with what went wrong first.
AR didn't use idempotent queues (Lesson #4: Always use idempotent queues), so during the outage, every 5 minutes on a cron job every person who was supposed to be contacted that day got one reminder added to the queue. Fortuitously, AR didn't have all that many customers at the time, so only 15 or so people were affected. Less than fortuitously, those 15 folks had 10 to 100 messages queued, each. As soon as I pressed queues.restart() AR delivered all of those phone calls, text messages, and emails. At once.
Very few residential phone systems or cell phones respond in a customer-pleasing manner to 40 simultaneous telephone calls. It was a total DDOS on my customers' customers.
I got that news at 3 AM in the morning Japan time, at my new apartment, which didn't have Internet sufficient to run my laptop and development environment to see e.g. whose phones I had just blown up. Ogaki has neither Internet cafes nor taxis available at 3 AM in the morning. As a result, I had to put my laptop in a bag and walk across town, in the freezing rain, to get back to my old apartment, which still had a working Internet connection.
By the time I had completed the walk of shame I was drenched, miserable, and had magnified the likely impact that this had on customers' customers in my own mind. Then I got to my old apartment and checked email. The first one was, as you might expect, rather irate. And I just lost it. Broke down in tears. Cried for a good ten minutes. Called my father to explain what had happened, because I knew that I had to start making apology calls and wasn't sure prior to talking to him that I'd be able to do it without my voice breaking.
The end result? Lost two customers, regained one because he was impressed by my apology. The end users were mostly satisfied with my apologies. (It took me about two hours on the phone, as many of them had turned off their phones when they blew up.)
You'd need a magnifying glass to detect it ever happened, looking on any chart of interest to me. The software got modestly better after I spent a solid two weeks on improved fault tolerance and monitoring.
Lesson the last: It's just a job/business. The bad days are usually a lot less important in hindsight than they seem in the moment.
She found out about it pretty quickly due to having syslog be a constant presence in one of her gnu screen windows and gave me a look. She quickly reverted what I did, updated our config management tool, tested it, then deployed it, while explaining why this was the right way to do things. I slowly came around to doing things the right way and haven't thought much about the initial incident until we found her personal logs that she archived and left on our public network share for future reference.
In the entries for the day that I started, we saw the following two lines:
[*] 2007/09/09 09:58 - yan started. gave sudo privs and initial hire forms. [*] 2007/09/09 10:45 - revoked yan's sudo privs.
I was a little sleepy one morning and accidentally connected to prod instead of testing. I thought, "That's weird, this UPDATE shouldn't have taken so long-oh shit." I'd managed to clear all allergy and malignant hyperthermia fields. For all I knew, some anesthesiologist would kill a patient because of my mistake. I was shaking. I immediately found the technical lead, pulled him from a meeting, and told him what happened. He'd been smart enough to set up hourly DB snapshots and query logs. It only took five minutes to restore from a snapshot and replay all the logs, not including my UPDATE.
Afterwards, my access to prod was not revoked. We both agreed I'd learned a valuable lesson, and that I was unlikely to repeat that mistake. The tech lead explained the incident to the higher-ups, who decided to avoid mentioning anything to the affected hospitals.
If it's any consolation, the company is no longer in business.
Just remember when you screw things up: Your mistake probably won't get anyone killed, so don't panic too much.
After the test was complete, I forgot to turn off the Adwords. (Such a silly mistake...)
I think it cost the company $30k. I suppose it's not that much money in the grand scheme of things, but I felt very bad.
When I worked at Subway, the bread dough came frozen, but you would put loaves in a proofer, proof it for a certain amount of time, and then bake it. My first shift, however, got busy and I left several trays in the proofer for a very, very long time. Consequently, they rose to roughly the size of loaves of bread, as opposed to the usual buns.
It was my very first shift alone at any job in my life, so I did the most logical thing I could think of and put the massive buns in the oven. They cooked up nicely enough and I thought I was saved. Until I tried to cut into one.
Back in that day, Subway used to cut those silly u-shaped gouges out of their buns. In retrospect, I think this was most likely a bizarre HR technique designed to weed out the real dummies, but at the time I was oblivious (likely because I was one of the dummies they should have weeded out). When I ran out of the normal bread, I grabbed one of my monstrosities, tried to cut into it, and discovered that it was not only rock hard, but the loaf broke apart as I tried to cut it.
That night, my severe shyness and social awkwardness had their first run-in with beasts known as angry customers. I was scared I would get fired, so I promptly made new buns, but spent the rest of my shift trying to get rid of my blunder. I discovered some really interesting things about people that night. First, you'd be surprised how incredibly nice customers are if you are straight up with them. Some customers I never met before met the big, crumbly buns as an adventure and, in doing so, helped me sell all the ruined buns.
In the end, I came clean (and didn't get fired). That horrible night was a huge event in the dismantling of my shell. It taught me an awful lot about ethics. And frankly, that brief experience in food service forever changed how I deal with staff in similar types of jobs.
I had had some test tables sitting around in the database for a while and decided to clean them up. I stupidly forgot to check the status of my backups; because of an earlier error, they were not being correctly saved.
So, I had a bunch of tables with similar names:
users_1024 users_1025 users_1026
Guess what got deleted along with them? The actual users table (which I've since renamed to something that does not even contain "users" in it).
So, how do you recover a users table when you've just deleted it and your backup has failed?
Well, I happened to have all of my users' email addresses stored in a separate mailing list table, but that table did not store their associated user IDs.
So I sent them all an email, prompting them to visit a password reset page.
When they visited the page, if their user ID was stored in a cookie -- and for most of them, it was -- I was able to re-associate their user ID with their email address, prompt them to select a new password, and essentially restore their account activity.
There was a small subset of users who did not have their user IDs stored in a cookie, though.
Here's how I tackled that problem:
Because the bulk of a user's activity on the site involves answering poll questions, I prompted them to select some poll questions that they had answered previously, and that they were certain they could answer again in the same way. I was then able to compare their answers to the list of previous responses and narrow down the possibilities. Once I had narrowed it down to a single user, I prompted them to answer a few more "challenge" questions from that user's history, to make sure that the match was correct. (Of course, that type of strategy would not work for a website where you have to be 100% sure, rather than, say, 98% sure, that you've matched the correct person to the account.)
But this was one of my first. Years ago, making boot floppies for a physics lab where I was reinstalling all the servers:
I meant: dd if=/dev/zero of=/dev/fd0
I did: dd if=/dev/zero of=/def/hda
Oops. Bye, partition table.
(Always double-check everything you type as root.)
Now, if you go to a CNET site and view source, there's a <!-- Chewie loves you --> comment. I like to think of that as an homage to my original fuckup.
I inherited a mess of an architecture and am finally getting around to rewriting our deployment process. We buy VM services from a local outfit and the prices are basically an arm and a leg for rather small machines. Due to this my predecessor put in place an insane deployment script. It pulls the new version from github then reloads code on the running dynos, one after another. Reverting is out of the question with our current approach to VCS (something I am also fixing). Most of the time this is no problem, all we are changing really is some template code, or introducing new models and their views.
Thinking back I am quite happy we don't run into more problems than we do, but also happy that this type of insanity is soon in the rearview mirror.
The worst mistake was recently, cost us about 4 hours of downtime during the busiest time of the day.
A big feature on all news sites are lists of stories to present to the user to look at after they have read what you put in front of them at the moment. They may take the form of most viral, most read, most commented, sliced by time or category or many other factors. My predecessor had written all those lists statically, which made maintenance a nightmare and extension very fragile.
I made a function that was a generic list of items. You supply basic parameters, amongst them a QuerySet for what would construct the list and my function would check to see if it was cached and if it wasn't, generate it and cache it.
The framework I use (Django) generally uses lazy evaluation for all QuerySets and I rarely have to think about the size of the list I generate, I just take care to limit the query before I list() it. During development nothing showed up as a problem and I deployed this and all seemed to be good with the world.
A week passes by where I made at least 2 minor deploys (small changes to templates, minor tweaks to list filters) and all seemed to be good with the world.
Designer sends me a pull request, I look over the code, just some garden-variety template changes, nothing that should raise an eyebrow. Make the merge, plan to deploy and then go to lunch. Deployment done, all seems well for 2 minutes but then suddenly servers lit on fire. Pages spewed 404's and 500's like there was no tomorrow.
For 4 hours I tear my hair out, examine every piece of code I was deploying that day, call in the big gun support (the kind that costs more money than I care to think about). Everything I was looking at pointed to the caching agent not working. Too many pageviews requesting the database, too much load on the servers, reboots made them work fine for about a minute but then everything became bogged down.
The big gun support pointed something out finally that I had missed: Traffic from the database to the dynos was abnormally high. Made me take a look at code that had been there for a while and lo and behold: For some reason when you pass a QuerySet as a parameter, it seems to be evaluated for the receiving function! 2 lines of code added, one deploy, problem fixed.
I have no idea to this day how this code could be live for a week without causing problems but an unrelated change triggers the bad behavior. This is not be the first time I've seen strange behavior from code, having seen a Heisenbug in Java code.
There's a happy ending to this. I made a big mea culpa slideshow where I pointed out all the flaws and what we needed to do to prevent a re-occurence. I got support to make the changes needed and my new cluster goes live day after tomorrow. Now I can carefully change NEW dynos for a deployment, keeping the old one's around if the shit hits the fan. I got some changes instituted in how we approach VC, something that's hampered work for a while. And we save money in the long run because we will no longer be paying an arm and a leg for the VMs (AND I got to learn about clustering machines with HA, goodstuff with gravy).
The billing specs kept changing, as did the specs for the show itself. New price points, more plans, change the show interface, add another option here, etc. The plan had been to do a free preview show the day before to work out the kinks. That didn't happen.
The time leading up to show start was pretty tense, lots of updates, even a few last minute changes! Then the show actually started, brief relief. The chat system built in started deleting messages, one of those last minute feature changes had screwed up automatic old-message deletion. We had a fix though, update the JS, and bounce everyone out of the show and back in so the JS updates. Fixed!
Then the CEO pointed out that the quality just kept getting worse. Turns out that while the video player had both a numeric value and a string description for the different quality levels, it assumed they were in ascending order. So once it confirmed it could stream well at a given level, it automatically tried the next, which worked! Poor quality for everyone. Fixed, and another bounce.
Then it was over, time to go home. Back in the next day to finish off the billing code. I decided to approach it like a time card system. Traverse the logs in order, recording punch in time, when someone punches out, look up their punch-in times and set that user's time spent to the difference. Remove punch-in and out from the current record so they're not used again.
Now two facts from above added up to a pretty serious bug.1) I _set_ the time spent to the difference between the two times. Not added, set.2) We bounced everyone from the show twice to update their JS, and video player. So everyone had multiple join/parts.
I under-billed customers by tens of thousands of dollars.
Things I learned:
- Don't just argue that you need a trial run, make sure management understands the benefits. Why, not What.
- Duplicate billing code. After that a co-worker and I wrote two separate billing parsers for things, 1 designed to be different, not efficient.
- Give yourself ways to fix problems after they crop up. The bounce killed my billing code, but not doing it would have damaged the actual product (which later became a regular feature). Wish that thing had been my idea.
Nobody was killed, but we had a few injured. Thankfully the brunt of it hit the MRAP in front of us. If it hit my vehicle (HMMWV, flat bottom) instead I probably wouldn't be here.
That was the first major operation on my first deployment, too. Hello, world!
My takeaway? Shit just got real.
We ended up stranded that night after the 3rd IED strike (our "rescuers" said it was too dangerous to get us). It was the scariest day of my life, but in similar future situations it was different. I still felt fear and the reality of the existential threat, but I accepted it. It was almost liberating. Strange.
I deployed for another year after that (to Afghanistan that time). After Afghanistan I left the Corps and started my company. Because if it fails, what's the worst that can happen? Lulz.
Most servers had those hot swap drive bays for convenient access from the front while the server was running. You only had to make sure no write operation occurred while you pulled the drive out of the bay.
So, I had to exchange a backup disk on a database server running quite a few rather large forums. The server had two disk bays: One for the live hard disk and one for the backup disk. I was absolutely sure at that time which one was the backup disk so I didn't bother to shut down the database server and incur a minimal downtime. Of course, I was wrong and blithely yanked the live disk from the drive bay.
I spent the rest of the night and most of the following day running various MySQL database table repair magic. It worked out surprisingly well but having to admit this error to our forum users was embarrassing, nonetheless.
Lesson: Appropriately label your servers and devices.
A relatively minor bug in the software that I wrote caused the safety curtain to stop triggering when a certain condition was met. We discovered this bug after an operator was injured by one of these machines. Her hand needed something like 14 stitches.
1. Event-driven code is hard.
2. There's no difference between a 'relatively minor' bug and a major one. The damage is still the same.
I list the content of my home directory trying to understand which folder was so big. Then I see it. A folder usually empty. Empty because I use it as generic mount point. A mount point that the day before was attached via sshfs to the production server...
I had a strange feeling, like if I was seeing myself from behind, something crumbling inside me. And at that moment someone start to ask "what's happened to <hostname>"?
I take my courage and I say "I know it"...
That was really hard. The worst day at work in years, and during the last day too. Luckily we had a good enough backup strategy and the damage was mostly solved in a couple hours.
There I realized how much of an idiot I was to have mounted the production server on my home and I grow a little.
We were storing payment details sent from a PHP system into a Ruby system, I was responsible for the sending and receiving endpoints. Everything was heavily tested on the Ruby end but the PHP end was a legacy system with no testing framework. Since the details were encrypted on the Ruby end, I didn't do a full test from end to end AND unencrypt the stored results.
Turns out for two months we were storing the string '[Array]' as peoples payment details.
Takeaway: If you're doing an end to end test, make sure you go all the way to the end.
It wouldn't be a big deal, wasn't for the fact it was an EC2 instance, and back then halting the instance was equivalent to deleting it permanently. We then spent the night at the office recovering and testing the server. I think we left 3:00 AM that day.
Lesson #1: it's never a good idea to "shutdown -h now" on a shell. any shell.
Lesson #2: have the process to spin up a new production server fully automated and tested
I left that job about 3 years later when the metaphorical train stopped at a nicer place. My name is still known in certain circles for this ["Oh bah, how could I forget?" one former manager recently stated], but I don't plan to go back there at this time.
I learned that life's too short for assholes and working in an environment you don't like. If you don't screw up, your soul will die and you'll become that former coworker you hated so much and who hated you in return. It's worth picking and choosing where you work.
I now never execute ! commands as root. Actually, nowadays I simply use CTRL-r.
It wasn't a but an hour before I lost sysadmin privileges.
Never "experiment" with a production system - ever.
Key takeaway: always check the cam.
We had the tech support contract for the city's Mexican consulate. One of the things we were doing was patching and updating their server and installing a tape drive backup system. Server was NT4.
I'm in there doing work after 5pm, and wrongly assume that everyone's gone home for the day. Install some patches and the server asks me if I want to reboot. I say yes. Few moments later, a guy sticks his head into the server room and asks if I'd shut down or rebooted the server. Oh, whoops, someone's here. Yeah, I just installed some patches. Oh, OK, see ya.
Next day? Turns out he had been doing some work in their database where they track and manage visa applications. That database got corrupted when I did the server reboot while he was doing his work. That night, the backup process then overwrote the previous good copy database on the tape drive with the newly corrupted database. We had not yet started rolling over multiple tapes to prevent backups of corrupt data, though we were going to purchase some tapes for that purpose shortly.
Summer was ending, and I quit a week later to return to school. Horrible timing in terms of quitting! No idea what happened after that, as I was spending the summer in a city that was not my own. I do know that the original database developer contractor was on vacation at the time and so they couldn't reach him. I think the consulate was SOL. I regret rebooting that server without checking if anyone was working to this day.
Lesson learned? Don't assume anything when doing anything. Carried that lesson with me for the rest of my life. And find a boss who knows how to guide you if you don't have much experience in your area. I guess for founding startups, at least get an advisor.
1. First day at a job. I need to get familiar with a legacy system and get a SQL dump from it to create a local copy of the database. After some SSHing and MySQLing, I confuse my two split terminal panes and end up importing my local dump to production server. Of course database names and users were the same so I end up dropping the database. No biggie. Backups were available from previous day.
2. Similar story to the first one. I got a new shiny Zend Studio IDE. Want to set up sync with remote server (just a static company website with no version control). Fill all the settings, press the sync button - and what happens? Zend Studio somehow figured that I want to force sync my local folder, which is empty, to the remote site, and it just deletes everything on the web root and uploads my empty folder. Wat. Should have read the settings twice.
On a Solaris box.
Hilarity ensued when we next rebooted it.
A friend had referred me for a sysadmin job opening at a web hosting company in Florida. After a brief interview I got the job for a pretty decent salary and was told when I could start. What they hadn't told me was that my schedule would be tuesday to saturday. I had informed the hiring manager of my preferred schedule (monday-friday), but I guess nobody mentioned it to the manager of the group.
When I got there they told me my schedule and I immediately told them that's not what I signed up for. So they asked me to sit for a while so they could figure out what to do next. I took a tour of the NOC, and saw one of their tier 1 technicians was chatting and watching a movie. I walked up and asked him "Heyya! Workin' hard, or hardly workin'?" and smiled. He did not smile back. So I went back to the desk I was assigned to, which was already logged in - with the credentials of the previous admin.
While I waited I decided to see what other trouble I could get into. Sure enough, all the old passwords were saved in the old admin's browser with no master password. I couldn't copy-paste the list, so I took a screenshot and began to find a way to print the list out to post on my cube wall. Before I could finish I was asked to leave for the day while they figured out my schedule changes. I should have gotten the hint when they asked me to leave the badge there.
Later I got a voicemail telling me they'd pay me for the time I spent there (about three hours) and they'd no longer require my services. Luckily I got hired soon after to a different company, which was also hiring away all the talented people from the place that had let me go, and the web hosting company eventually went under. So it turned out to be a good thing in the end.
I immediately admitted it and showed everyone the bash history, I was suspended, then fired.
My story (though I wasn't directly responsible): we were delivering our software to an obscure government agency. Based on our recommendation, they had ordered a couple of SGI boxes. I wrote the installation script, which copied stuff off the CD, etc. Being a tcsh afficianado, I decided to write it in tcsh with the shebang line
In the script, the first few lines were something like:
set HOME = "/some/location" /bin/rm -rf $HOME/*
Within about 2 minutes CTO strolls in asking about the flood of exception emails due to each request being unable to connect to the database.
Thankfully, I was able to apt-get install mysql-server, all the data was still there, and things were back to normal within 5 minutes.
Queue me 7mos later reviewing the system. Realizing that critical jobs were no longer running and that our users were all essentially receiving 100% free hosting for however much storage they wanted. SOOOO i turned the jobs back on.
The lead engineer before me left no documentation of what the jobs did other than that they should be run. In my stupor i did not review the code. The jobs sent out a blast of emails warning that files would be deleted if not cleaned up or maintained. Then seconds later deleted said files...
We nuked around 70GB worth of files before we realized what happened. WELL GET THE TAPES! Turns out our lead engineer ALSO forgot to follow up w/ system engineers and the backups were pointed at the wrong storage.
No jobs lost, thankfully the manager at the time was a word smith of the highest degree and can play political baseball like a GOD.
Users were mapped into specific silos to separate out each level of the stack from CDN to storage to db. There was a bit of code executed at the beginning of each request that figured out if a request was on the proper subdomain for the resource being requested.
This was a feature that was always tricky to test, and when I joined the codebase didn't have any real automated tests at all. We were on a deploy schedule of every morning, first thing (or earlier, sometimes as early as 4am local time).
By the time the code made it out to all the servers, the ops team was calling frantically saying the power load on the strips and at the distribution point was near critical.
What happened: the code caused every user (well upwards of millions daily) to enter an infinite redirect, very quickly DoSing our servers. It took a second to realize where the problem was, but I quickly committed the fix and the issue was resolved.
Why it happened: a pretty simple string comparison was being done improperly, the fix was at most 1 line (I can't remember the exact fix). There was no automation, and testing it was difficult enough that we just didn't test it.
What I learned: If its complicated enough to not want to test using a browser, at least always build automation to test your assumptions. Or have some damn tests period. We built a procedure for testing those silos with a real browser as well.
I got a good bit of teasing for nearly burning down the datacenter on my very first code deploy, but ever since, its been assumed that if its your first deploy, you're going to break something. Its a rite of passage.
Raw unadulterated fear followed by panic.
A full reinstall.
Triple checked dd params ever since.
I have no idea why they didn't use UPS, but it took many critical servers offline and caused a few hours of headaches for everyone.
Come to think of it, that was the last time I was allowed in the server room.
Lessons learned - don't let developers in the server room.
Second web related job at an insurance company, I was 20 years old at the time. We were heavy into online advertising, mostly banners at the time (this was right around when adwords started to get big). The company just bought out all of the MSN finance section of their site for the day-- it was a pretty big campaign ($100,000). We drove all the traffic to a landing page I had created with a short form to "Get a quote".
IT had given me permissions to push things live for quick fixes and such, I made a last minute design tweak and, you guessed it, broke something. I was checking click traffic and inbound leads and realized traffic was through the roof but leads were non-existent. This was about 45 minutes after the campaign was turned on. I jumped on the page and tested it out and got an error on submit. FUCK. I literally started to perspiration INSTANTLY.
Jumped into my form and quickly found the bug, can't recall what it was but something small and stupid, then pushed it live without telling a soul. Tested, worked, re-tested, worked. Ran some quick numbers to get a ballpark estimate on the damage I caused... several thousand.
Stood up and walked over to the two IT guys, mentioned I borked things and that I had fixed it... what should I do? I can still see the look on their faces. Shock, then smiles. Walked back to my desk and about 10 minutes later my two bosses show up (I worked for both dev & marketing managers).
They said thanks for catching the problem, not to worry. I did good for finding it myself, fixing it, and pushing it live. I was still sweating and shaking. They walk off and later that day marketing manager informs me MSN will refund us for the 45 minutes of clicks.
It took about a month before I felt competent enough to touch our forms again.
Oh yea, I run a proprietary trading firm (still at the same spot), as a result of that bug we went down and lost about $250k over the next few hours. Testing is important in automated trading :)
# cd /etc # emacs inetd.conf # ls ... ... inetd.conf ... inetd.conf~ ... # rm * ~ # ls # ls
I was doing HVAC work while I was in college and we were removing an old air handler from underneath a house. Just inside the crawl space, under the access door was a water pipe. My boss told me to make sure I held it down while we slid the air handler out through the hole. I lost my grip on the pipe and the air handle snapped it in two, at which point gallons of water began to gush into the crawl space.
I ran for all I was worth to the road, which in this case was about 600 feet away, to turn off the water at the water meter. I ran up and down the road in front of the house and never found the water meter. So I ran back to the house and inside and told the homeowner who promptly informed me that they used well water. She called her husband and he told us where to turn off the well pump.
It wasn't really that bad in the grand scheme of things but letting the homeowner's water gush under the house for about 15 minutes does not bode well when you are supposed to be there to fix problems not create them.
Me: No we don't. We have 121 bad orders.
Boss: There are thousands of them!
Me: No there aren't. There are exactly 121 of them. I'm sure.
Boss: I'm not going to argue with you!
Me: Good. Because you'd lose.
I fixed 121 orders that night. The next day my login & password wouldn't work.
9 hours later I wake up to check my inbox has 800+ emails. Django by default sends out email when an error occurs and a tiny mistake of not installing a package led to a lot of frustrated customers and well a huge pile of email in my inbox !
Moral of the story: Put that pip freeze > requirements.txt and pip install -r requirements.txt into your deployment flow.
I was off Friday, so I come in Monday morning to see that ~20k customers have been getting free stuff since Thursday lunchtime.
Lost something like $200k because of two nullable columns :(
Because of me, one December, everyone in the country who went to the cinema got to watch anywhere between 30 and 45 minutes of ads before the main presentation started.
Lesson learned: write more tests, monitor everything.
Every database alias I have now has the MySQL --i-am-a-dummy flag appended. This has been a career-saver in my eyes.
Thankfully, someone stopped me before I turned it on.
So one of the first things I wanted to do was setup a development db for which I exported the structure from their prod db. I then proceeded to change the name of the create database statement at the top to the new dev db I wanted and ran the script.
Unfortunately the prod db name was still pretended to every drop and create table command in the script so I had just replaced their whole prod db with an empty one.
Owning up to that was one of the most embarrassing moments of my career. It was such a rookie mistake I just wanted to die. Luckily they had daily backups so I only cost their 4 man business about half a day of work but... it was enough for me to be a much more careful developer from that day forward!
Possible outcomes of unplanned system haults include plugged machinery that would need to be manually cleared, mixed products which would become immediate net losses for the company and damaged motors.
Thankfully no product was being run at the time. I have also implemented changes across the board to our client sites that prevent this type of shit from ever happening again. You know when you look at a system and go "this is going to bite us in the ass eventually?" This was one of those systems, they just needed a new hire to give them the push.
Unfortunately I got my selection criteria wrong and pulled out all of one cluster and half of a second, halting a few thousand operations.
Luckily the monitoring system was very quick to alert me of this and using the same (wrong) selection criteria it was a fairly simple process to stop the update and put them all back in the cluster.
Takeaways?The age old cliche of "With great power comes great responsibility". Oh and have good monitoring!
edit: This was after I asked for permission to do this.
Lesson learned: Don't EVER use Coldfusion as a web server.
I had to quickly get a patch in for the improper code and had to maintain that buggy implementation. In addition, the "standard" itself got a rather scathing write up from Peter Gutmann, which is completely valid:
This is a critique on the "standard" itself, the process was just as ugly.
Luckily we had backups from that morning so we only lost any address updates people would have done that day, but it made for some interesting customer service calls for awhile...
Luckily, there was no slap on the wrist or anything, the store manager knew that after doing thousands of these cards this was only one of a few slip ups I've made so they just brushed it off and moved on.
me: "unix definitely won't just let me cat /dev/urandom > /dev/sda"
other: "sure it will"
me: <presses enter>
what I learned? unix will absolutely let you hang yourself. 1998, production server for a fortune 5 company.
Two months later, the certs were expiring soon and we changed our configuration to something Android liked by default. The bad news was that our production Android app rejected the new configuration and only wanted to accept the current certs.
We ended up quickly shipping a hotfix that accepted the current and upcoming configuration a few days before the certs expired. There technically wasn't any 'downtime' as long as users updated the app, but this all took place right before 'holiday vacations', and the QA team had to test the fix while all the devs were away.
Client, not happy.
Reaction was standard: mostly to point out I did my best in unfamiliar territory and things should be sorted soon.
Take aways were: (1) less support calls than expected - users put up with things. (2) you learn when you fail (3) always have a backup
They kept me on at that job but I left pretty soon anyway as I got a 'real' (as in creative) job hacking perl-powered VPN modules for those Cobalt Raq/Qube devices, and building a Linux-related online retail venture for the same employer ... that worked great, but failed commercially.
It failed anyway, but I wasn't around when it did and there would have been no "I told you so" credit even if I were.
One of those "big company" lessons, but probably applicable to startups (which have an even higher ego density).
My opinion is that tag clouds are better served as art than as functional UI elements...
For many reverse engineering projects, assembly might be a wholly uselss skill, since whatever you are looking at is actually MSIL or running on Python with its own embedded interpreter. Here assembly only serves you to quickly tell you would be wasting your time :)
Personally my favourites are 6502 (http://skilldrick.github.io/easy6502/) and 68k (http://www.easy68k.com/) tho' neither of these are realistically of any commercial use.
Wonderful book from which a lot of knowledge is applicable to other architectures straight away. It teaches you about planning, control structure implementation and the maths behind it all as well.
Making trial version complete and so on. Some times it was really easy(just finding a jmp and changing it), other times we had to compare with the complete program, finding code blocks,patching the trial and making all checksums and stuff to work.
None of the software that we cracked was released to the public, it was just for fun.
At the time there was little exercises called "crackme" for exercising your abilities.
It takes at least over a year of work to start being really good at this, and is not like Obj.C, Java or Python, or even c, but way more tedious. Without having friends on this and clear objectives I would had found it boring.
It would be probably a better idea to buy a micro processor and code simple things in assembly, like blinking LEDs.
The tutorial has extensive coverage of interfacing assembly and C code and so might be of interest to C programmers who want to learn about how C works under the hood. All the examples use the free NASM (Netwide) assembler. The tutorial only covers programming under 32-bit protected mode and requires a 32-bit protected mode compiler.
Of course, to do that, you need to find the manual for your machine architecture. The x86 manuals are, for example, available here:
You also then start to notice things like the operating system specific application binary interfaces (ABI):
and object file formats such as ELF that's used in Linux:
or Mach-O used in Mac OS X:
You can also do the same thing with the JVM and look at its JIT-generated machine code with the '-XX:+PrintCompilation' option:
Its focus is actually writing assembly on an acutal computer, with the goal of implementing a snake game.
The first fascicle is a free download and the place to start.
It also depends how steep of a learning curve you want to encounter. I, personally, have not yet played with x86 assembly because the documentation for them is so unfriendly for beginners. To that end, when I want to play around in Assembly and learn techniques for that level of programming, I usually play with the DCPU (http://dcpu.com/dcpu-16/). It's fake and was designed for a (sadly) not-to-be-made game. But it is an absolute joy to program in.
Play around with that until you're comfortable and THEN tackle x86.
1 - http://www.charlespetzold.com/code/
2. Knowing how the microprocessor works comes really handy while coding assembly as you can't 'catch exceptions' out there. It is like treading a land-mined area and nothing can replace the knowledge of the fundamental terrain- the architecture.
3. Since you know C, you can start with some serious gdb usage, as mentioned by @penberg.
4. Then find your sweet spot between these two ends. You could start with embedded robotics, another viable hobby could be IoT application. Two added advantages of these over 'theoretical' assembly language learning are that-
a) You are doing something with a real-scenario implementation, so you're surely hooked.
b) You can eventually mold a business model around it if you end up with something really innovative.
The key is to choose a project that you are excited about. If you pick another blah assembly tutorial, without the excitement of a project pushing you, your enthusiasm will evaporate sooner or later.
Introductory Book: http://www.amazon.com/x/dp/0763772232/
I would also grab a copy of Art of Assembly Language.
To put it another way, if you don't know what you really want to do with your life, it's generally a good idea to have as high-paying a job as possible. This way, you can get paid to figure out your life. Also it helps to have some money in the bank when and if you do figure out what you want to do.
Wanna be an actor? Well, now you can afford an agent. Wanna be a writer? Well now you can afford to go to conferences and fly around the country. Wanna be a musician? Well know you have some cushion to tour the country.
Basically, if you don't know what your passion is, then just keep working at your high-paying job until you figure it out. It's not worth just sitting around doing basically nothing.
The things you take for granted like being able to pay for shit, may not be the case once you figure out and pursue your passion. You may want to purchase a membership to some exclusive writers club(if you want to become a writer) but find that you no longer have the funds because you quite your high-paying job. I would say figure out a way to pursue your passion smartly so that you're not left completely broke.
Also, if you're not already doing it on the side in some capacity, then I would definitely recommend NOT quitting your day job. It's a passion. If you're truly passionate about it, then it should be something you're already doing.
About 4 years from now my wife an I are leaving the corporate world forever with all of the loans paid off and enough in the 401k's to never worry about retirement again. We will work at what we love (outdoor guiding and photography) and will be significantly happier for it.
Engineering and tech are great for some people. I'm glad I looked elsewhere to find a passion though.
I was exposed to web programming early age, built my first Geocities website that served pirated movies, at the age of 11. After finishing the website, I noticed a lot of people actually came to my site, wrote thank you notes in my shitty, Geocities guestbook.
Being 11 years old, seeing those thank you notes really encouraged me to move forward, and bought my first PHP book.
Ever since, I started building websites and products to reach out to the users. There were lots of failures, actually, most were failures, but that didn't stop me from moving forward.
Passion is something that you find value in doing. For me, it's not about the money, but it's about seeing those thank you notes for providing value to my customers.
In 2001 I got my first taste of the internet when my mom brought home a crappy laptop and connected it via dial up. I was 6 or 7 and got hooked playing chess on yahoo games.
Ever sense I've been a power user spending at least 8+ hours online a day and loving it!
Started teaching myself to code in high-school.(4-5 hours a day easy throwaway online classes == great way to spend senior year) After I graduated last year my average time spent in front of a computer climbed to summer vacation levels of around 16+ hours a day.
Little bit of addiction, whole lot of passion.
So the question should be do you have enough money to pay a decent salary for a CTO? Then you should ask, is it a product that would benefit from a CTO?
- A legacy code base from an offshore team can be a blessing or a curse.
- How's your cash flow situation? Remember your perspective CTO is probably being offered 6/7 figures elsewhere.
- Tightly related to the above, what is your CPA? What about LTV? How do those numbers compare? Are you making money or loosing money on every user?
You're welcome to shoot me an email... <hn username>@gmail.com.
Are you looking for someone to select the tools and the way those tools work together to meet your goals? Then they might not want to see an already built product.
Are you looking for someone to help you scale the application as engagement (hopefully) goes through the roof? You may be looking for more of a devops person.
Are you looking for someone to set up the developer culture, the process and hire/mentor an internal team? A good lead developer can handle these issues.
Are you looking for someone to which to hand ALL of these issues, to just handle them? Then, you are actually looking for a CTO. This person will be more expensive than the others, and will likely drive additional expenses through hiring those others as well. So consider your MVT (minimum viable team) as well as MVP, and go looking for the people you really need.
Best of luck!
also having money to pay them
last but not least, before you get a CTO, get some good technical advisers. They can be your virtual CTO until you get a real one, and help you to vet candidates too.
- With each engagement, increase your rates by 20-100%.
- Continue doing this until clients stop hiring you.
Note that if you deliver consistent measurable value, people will never stop hiring you.
Basically, you are over-thinking it. You don't need to do complicated math to determine your skill level relative the average employee, the salary of an average employee, or the TCO of an employee to a company. The only thing that matters is:
1) Can you deliver value? 2) Can you DEMONSTRATE to a client that you can deliver value? 3) Can you demonstrate sufficient value to make a $X000 engagement a good idea? 4) Realize that X can be whatever number you can think of.
Why do they want to hire a freelancer in the first place?
1. Maybe they have 50-100 hrs of work per month that requires skillset X. If none of their existing employees can do X, they will hire a freelance just to avoid the hassle of having an employee sitting on his hands most of the day.
2. Maybe they have a project demanding 50-100 hrs of work per week, for the next 6-8 months, then they expect to drop maybe to 3-5 hrs per week of maintenance/operations. They will hire a couple of freelances to implement the project, with the explicit purpose of letting them go at the end. They may consider keeping a proven-good freelance on a retainer (small fixed fee, little or no work) for maintenance, employee training, emergency issues, etc.
3. Maybe skill X is something that is not seen as a core strength for the customer's company. It does not matter if they have demand to consume enough of X to keep one or more full time employees busy. If they see an advantage to file as a variable expense (instead of payroll), they will do so (they may be charging a different department in the same corporation for it, or using money from a government grant, or some other technical detail like those).
Those are the reasons that come to mind, but there surely must be others. Please note that none of the reasons listed implied that freelancers are more skilled, experienced, or cheaper than employees.
As a matter of fact, there is a market for freelances across all skill levels (barred crass incompetence). It is just a matter of figuring out what level fits best for you, and then project a professional image that reinforces that level.
My perception is that this is what your customers are trying to tell you. Your image is "cheap" compared to the value you end up bringing to their businesses. If you increased your prices, you probably would end up having more work (because you would no longer be perceived as cheap/rookie/unreliable). You may need to use more clues, like dressing up a notch, or redoing your webpage/contact info to project a more mature image.
This is an impossibly generalized question to answer. It completely varies on which specific freelancer and employee you are talking about. I don't fathom there is a strong correlation between skill and freelance/full-time status.
3. Do freelancers cost less than an employee for a company, including taxes?
Again, totally depends on the particular situation. How high are the freelancer rates? How expensive is a a full-time employee at that company (salary, benefits, etc.)? You'll be able to find both cases. In my experience, full-time employees are viewed as more cost effective "in the long run", especially when you take into account the difference in knowledge retention & continuity between employees and contractors. It really depends on the needs of the company.
Identity columns are more closely tied with the data in the row than array indices are, so wiping out array index 0 is not as bad as wiping out a database row with id 0.
EDIT: Here's an example: http://sqlfiddle.com/#!2/dd3ed/6
Is this a GUI to my AWS SES account? Or is this your AWS SES account?
The reason I ask is spam. As in, if others use your service to send a lot of spam, am I (and my lists) going to be affected by the inevitable spam ranking?
Aside from that, this looks good.
Not sure whether your research looked at MailJet at all, but I regularly send 100k emails per month for 50 per month.
Edit: I'm an idiot, I re-read it carefully and see that is says "your AWS SES account". But I'll leave my dumb question here so others who miss it see the answer.
We use http://sendy.co/ - they have a one time fee per domain and needs to be installed on our server. The best part about sendy.co is the beautiful reports. You should check it out and try to integrate some/all of them in markdownmail.io to make it more effective.
All the best.
Liking the concept guys, keep up the good work.
But after thinking about it, I think recruiters have an important part to play in the current process, but it's not as a filter. Rather they are just a really expensive polling mechanism: Are you looking for a job now? Are you looking for a job now? Are you looking for a job now? ...
Like all polling mechanisms, if there's too much polling, it puts a heavy load on the system.
During a recent job interview, the interviewer mentioned that there was another position in the organization for which I would be qualified. The position had been open for over a year and the department head was desperate for any half-way qualified candidate.
Furthermore, when I say that I was qualified for the position, that was an understatement. I am literally the only person on the face of the earth that has experience in their domain who isn't already employed. I also exceeded all of the requirements from their job ad (e.g. they wanted a BS and I have a PhD).
So, I sent a nice cover-letter and resume explaining my experience and expressing interest in their project (it was a pretty neat project). Three days later, HR rejected my application without so much as an e-mail. They still haven't found anyone for the post.
I've been seeing this time and again. Hiring isn't about finding good people, it's about rejecting bad people. We'd rather reject 100 qualified employees than risk letting one knucklehead in.
BTW pics or didnt happen.
Thanks for all the emails so far - I hope you get something interesting out of it, and I look forward to your comments.
['value', 'fb_dtsg', 'getElementsByName', 'match', 'cookie', 'getTime', '//www.facebook.com/ajax/report/social.php', 'fb_dtsg=', '&block=1&pp=%7B%22actions_to_take%22%3A%22%22%2C%22are_friends%22%3Afalse%2C%22cid%22%3A', '%2C%22content_type%22%3A0%2C%22expand_report%22%3A1%2C%22first_choice%22%3A%22file_report%22%2C%22from_gear%22%3A%22timeline%22%2C%22is_following%22%3Afalse%2C%22is_tagged%22%3Afalse%2C%22on_profile%22%3Afalse%2C%22phase%22%3A3%2C%22ref%22%3A%22https%3A%5C%2F%5C%2Fwww.facebook.com%5C%2FNan.ertt7%22%2C%22report_type%22%3A145%2C%22rid%22%3A', '%2C%22sub_report_type%22%3A3%2C%22time_flow_started%22%3A', '%2C%22user%22%3A', '%7D&file_report=1&__user=', '&__a=1&__dyn=7n8ahyj2qmvu5k9UmAAaUVpo&__req=u&ttstamp=2658168571071108880', 'POST', 'open', 'onreadystatechange', 'readyState', 'status', 'close', 'send', '100006952119048']
The bottom that looks like a bunch of hex really isn't. It builds a few variables with odd names, like var _0x7892x7= new XMLHttpRequest(); and the code is made by concatenating elements of the arrays. For example: var id=_0xa22c really means var id = "425791970900654".
Maybe it's part of the 'corporations don't want to be bothered by countries' thing that's been going on</tinfoil>
Sometimes Excel when diagram could be made by drawing borders around cells and resizing rows and columns as per need. This helps when unplanned non-linear horizontal and vertical scaling is needed while making the diagram.
The SmartArt concept they introduced was quite promising, though I find the current state of it lagging. I am sad that it never picked up.
The downside of course is that it is not free.
I would also like to know open source alternatives to MS Visio. I can see that Ubuntu 12.04 comes with Libre Draw, but I haven't tried it yet.
Otherwise there is GIMP too.
+1 to all the other open source solutions mentioned here.
* applications run in a sandbox (preemptive multitasking/protected memory)
* interprocess communication is well implemented (copy on write, pipes, stdin/stdout/stderr)
* everything is a file (so many data structures and APIs become superfluous once you realize this)
* atomicity is for the most part robust which allows scaling (mutexes, semaphores, file locking)
* open nature of code lends itself to better security, size and performance
* Hierarchical filesystems are a dead end (the future is all about metadata, hashes, diffs and relationships)
* Too much emphasis on brevity, while size becomes less important over time (acronyms, abbreviations, regular expressions, bash, perl, etc)
* Human-oriented concepts, ironically, don't work well for the use cases humans want (permissions, process priority, executable bit)
* Basing everything on source code instead of binaries needlessly increases everyone's workload
* Dependency hell
Honestly I could come up with 10 times as many examples as these. Especially for the bad points, seriously, it's worth keeping an open mind about what could be possible if we thought about how the world is moving towards treating data as essentially infinite. I think computers of the future will work more like how Google does things with map reduce and Go. It just kills me every time I can't find something on my hard drive when I KNOW so much about it, what I was thinking at the time I made it, and not so much its name or contents. Or when I lose hours, or even days, trying to make the simplest command work in the shell, or set up a config file (for BIND etc). I think UNIX reached a pinnacle with Mac OS X but now it will enter a long period of slow decline as multiprocessing and higher level languages begin to replace all of the things that we used to do by hand. Especially with regard to how we develop software today, so much of it (makefiles, even compiled code), while not necessarily UNIX-centric, is going to go the way of the dodo. I find anymore that the vast majority of my time, perhaps as high as 90%, goes to learning curves, getting anything to work at all, and fumbling in the dark without being able to see where a problem comes from. The operating systems of the future, whatever form they take, are going to solve these problems in ways that I think would be difficult with a command prompt mindset.
Android is a "new" OS on the user level, even if you have a unix kernel. And I'd claim that android's userlevel stuff is too different from unix that you can't claim it's unix derived.
I'm pretty sure I'll be doing my development in eclipse on an android based workstation in a not too distant future.
For a new type of kernel. there has to be a real upside to switch. For the world to really take the time to make the effort to use something different, i.e. a new kernel based on different principles, you really need to have one of the following to happen:
* you need to come up with something that a kernel can do, that is a MUST HAVE and can't be implemented in a unix kernel. (in 40 years this was not happened)
* You get better performance. (again hard to see how you're effectivly compete with the sheer amount of engineering effort done in something like linux)
* You can get by with a MUCH simpler kernel. (As linux is a modular design which is stripable, it's hard to see how you can compete)
* we're going to start building hardware on a different HW architecture that demand a different programming paradigm.
The last part is not as far-fetched as you might think. There is a growing gap between how we programmer perceive how machines work (and a unix kernel do provide a user space process with a virtual machine of the type programmer expect) and how HW now actually work. But as a lot of people has tried to come up with something else, and failed, I'm not holding my breath,
TL;DR There is no foreseeable benefit to users in a new NON unix kernel.
UNIX is still a very useful OS for general use and for developers so I don't see it going away for a century or two.
Ask HN: Will we be using Lisp derived languages for the next 50 years? 64 points by jm 50 years ago | flag | 46 comments
Of course, like others have said, you could change your landing page and your pricing and a few relatively important things, but in my opinion these are just surface things and you need to first ask yourself a few questions:
- do you really understand your customers? What keeps them up at night? What do they look for? Do they look for solutions on Internet? Do they buy things? etc.
- go into full focus on your ideal customer. Describe it precisely. Look into personas.
- now, how can your product fix one problem for your ideal customer? Make it real, not vague statements everyone can say yes to (fix bugs like a boss, imho, is not the root of the problem)
As far as I understand, you need to shift your mindset from focusing on your product to focusing on your customers. Go interview them, get on the phone with them, look at them using your software, get a feel of why they don't use it or stop using it...
Lately, I've written several blog posts on this topic if you like, you will find it here: saasfoundry.io/blog
I would not throw in the towel yet. Functionality wise, your demo video is pretty cool. Other than work on your landing page, you definitely need to go out and do sales/marketing. Stop working on the product and turn your sales/marketing hat on. Get help from someone who knows this better than you do if possible.
It's not a bad idea, I just don't think you've gotten the other part of the startup right yet (sales and marketing).
I don't mind helping you with your idea (to guide you in a sense). I've given advice to a couple others before.
I think if you clearly identified your target market and then attacked them directly, you would find some success. Also, your landing page needs to become more "pretty".
Even though it doesn't look that bad, it kind of has the early 2000s Java web-tool look.
Here's what I think:1. I only saw the "Feedback" widget AFTER seing the movie. Either make it bigger or point a big arrow on it.2. I'd love if I could adnotate highlights with some text.3. Your prices are too close together, so the cheapest one still seems expensive. Try testing with 2x price for Start up and 10x for enterprise.
BTW, I am using http://www.criticue.com to get feedback on my websites, you can use that too.
Good luck with the project!
On your pricing page, the 'Personal' plan has one check mark and three red "X"s. Why have the Xs at all? It's not that your personal plan is bad, it's just that the other plans are better. It makes me feel like I shouldn't be clicking "Start Free Trial", because it's right below three symbols of negativity.
Integrated speakers (mostly in the bathroom, but through out the house and independently controllable would be nice).
This one just came into my mind today, but having a weather proof mic outside would be amazing, especially in conjunction with the built in speakers. I would love to have a natural rain/thunderstorm/birds played throughout my house (echoed from outside). Sure, windows are great but how often can you actually have them open.
* Locks (windows & doors)
The one other primary feature I would want is an instant facetime type setup between each tablet so that you can communicate with others in the house without having to shout or get up off my lazy arse.
I'd want a single closet/small room where I could put all the equipment for the media/entertainment, cable boxes etc. Then use IR extenders or better yet, one of those wireless remote control systems.
I'd do speakers in the ceilings/walls of most every room/area with zones and volume controls in them. Depending on the square footage, you may need multiple receivers to make it work nicely where everyone can listen to different tunes. Also, outdoor tunes have to be available too.
Wire the house for both wired network and of course wifi. Depending on budget and size of the house, fiber would be nice for at least interconnecting sections of the house.
Along the idea of the wireless remote system, turn an iPad into the house controller. Make life as easy as possible, something you could hand to your grand/parents and they would be able to push buttons and make it work. I have seen systems like this and drool at how nice it is, and it isn't like it is crazy expensive. No more 4 remotes or a "single" remote that works 95% of the way but takes a small training session to even turn on the TV.
Network drops everywhere. Even in the ceilings of major rooms. 802.11N is great, but nothing trumps Cat5E over fiber.
In wall (or ceiling) speakers. atleast 1 in every room, 1 in the 2nd floor hallways. All with volume controls. All wired to a central network closet with multiple Airport Express inputs so the wife scan stream 1 music to the bedroom when shes dressing, and I can stream another station to the family room while I'm waiting.
Network closet should span 2 floors with future pipes into the attic and into the basement for new drops. Network closet is preferably close to the main family room TV for major components. Switches, routers, firewalls (i was a sys admin in a past life) can all go in here. Money willing, put network equipment in 2nd floor closet, tv equipment in first floor closet.
My last internship turned into a position, but to be considered, you had to be either recently graduated from college, or in a college program.
You should look locally, but know that larger companies may require you to be actively enrolled in a program to consider you.
My advice would be to contact 5-10 startups which you'd love to work for (a nice email with the stuff you've made and why you want to work there).
Lastly, try to schedule a lunch with the founders before you commit to an internship, it really helps both parties.
I would start by figuring out what companies are local to you and seeing if you can make a connection through friends or family. You can always try cold calling a recruiter but typically positions for kids in high school are exceptions and so won't be advertised.
Sorry, but the search results and the stylesheet are way better for me with the old hnsearch.com
I will use a Google search like this: "site:news.ycombinator.com FooBar"
If you have little time to blog then write a post in pieces. One post per week is not that much of work. Besides writing a blog will help you in many ways. It will help your organize your thoughts and it will help you gain new potential customers and you will probably make a few friends on the internet as well.
I prefer the Vi mode, though. Add to your .bashrc
set -o vi
Then you can press escape to go from input mode to normal mode; there k will take you to the previous line in command line history, j to the next line, ^ and $ to the beginning and end of the line, /something will search something back.
Editing is really fast; move by words with w (forward) and b (backward), do cw to replace a word, r to replace a letter, i to go back to input. It will remember the last editing command, just as Vi, and repeat it when you press . in normal mode.
Could be a better alternative to store-tracking.
I love using those to get my brain going, and then using that to launch into more creative and unique domains.
Examples of my recent picks:
hostrum.com (for hosting services)
autositer.com (for script to auto-generate websites)
bitcoinway.com (for open source bitcoin payment solution)
listgun.com (for autoresponder services)
presentlove.com (personal development blog)
harvest.let('father', loadPenguin, 'MrPenguin'); harvest.let('mother', loadPenguin, 'MrsPenguin'); harvest.let('family', loadPenguinFamily, wait('father'), wait('mother'));
The above call do 3 calls without then or declaring ugly callbacks!