Here are some tips to help you keep your coins safe from attacks like these and other dangers in the digital—and physical—world.

Information in this blog post is also covered in my visit to the Coin Chat podcast episode 22.

One of the backbone technologies of the internet is something called “Domain Name Services” (DNS). DNS is like a phone book for your computer. When your computer needs to talk to a site like “example.com” it asks DNS for the IP address, which is a string of numbers (four in most cases) such as 192.168.10.121. Someone hacked into some DNS servers so that the phone book returned the wrong address. This was most widely reported on—ironically enough—Google’s “Honest DNS” servers.

Know Who You Are Dealing With

Certificates of Authenticity

The first clue that something was wrong was when some browsers indicated that the SSL certificate didn’t match the site. Modern browsers use a technology that allows a website to prove it is the one it claims to be; for instance to ensure you are talking to your bank and not someone pretending to be your bank. It is very difficult (though not impossible) for hackers to get an imposter certificate and the hackers in this case did not have that certificate. Those that fell prey to this attack either ignored the warnings or used a browser that did not display the warnings prominently.

Before you enter private or important data into a web site make sure the site presents a valid certificate.

Initiate Contact Yourself

This technique only works if you are on the site you think you are. A common technique is to create a website with a name that is very similar to the site you are trying to reach, for instance www.myetherwa11et.com. If you aren’t paying close attention you might not notice the difference. Through clever tricks the illusion can be even more convicing. For high-security websites never trust a link someone gives you; always enter the link by hand. If a stranger comes up to you, hands you a phone and says “your bank is on the line, they need some information from you” it’s very likely you would walk very far away. Clicking on a link in email or Telegram is the equivalent.

Notice that I haven’t linked to My Ether Wallet in this post. I don’t want you to click on that link because…well…read the previous paragraph.

If you do a search on Google or another search engine be aware of the “ad” links. Anyone can pay money to show up at the top of this list and frequently these are used by scammers to direct you to a less-than-noble site. Always click the non-ad links to ensure you are going to the most relevant and highly-rated site.

Use Security Tools

An additional step you can use to protect yourself in the crypto world is install a browser extension like Cryptonite or EAL. These extensions can warn you of known phishing sites and are an extra layer of protection. Don’t rely solely on these extensions as they can take some time to propogate bad sites; if a site seems phishy you should just walk away even if the extension doesn’t say it is bad. You can find links to these extensions at the top of the MEW site.

Also make sure you are running malware-detection software on your system. Malware can make illegitimate sites look on the up-and-up and can steal information right from your browser.

Host It Yourself

Alternatively, MEW allows you to download the site to run locally on your own computer. You can keep this copy in a folder on your machine and transact with it there. This will ensure that network attacks like the ones above won’t affect you.

Safeguard Your Keys

First, the standard warning:

Your private keys are the only way to access your coins: If you lose your private key you lose your coins. If someone else gets your private key they can get your coins.

Your wallet does not “store” coins. The coins exist on the network in the blockchain and can be accessed by anyone, anywhere, at anytime as long as they present a valid key. This “proof” of access is all that is needed to take your funds. The network treats this proof as authorization from you directly and does not provide a mechanism for undoing the transaction. There is no 1-800 number you can call to get them back.

Loose Lips Sink Ships

You should never enter your private key or recovery seed into any computer or web browser. This is the least secure method of interacting with the blockchain and is the attack vector used in the above hack. Within seconds of entering a private key on the copycat site users had all of their funds stolen. A piece of malware or a rogue browser extension can also steal this information.

The TSA—a government agency with “security” in its name—posted photos of the master keys used to unlock travel padlocks allowing anyone to build a duplicate key. If you have a TSA lock it is only slightly better than using a ziptie to lock your luggage. If your private wallet key falls into the wrong hands you’ve just opened the door to your funds.

Avoid storing your private keys in the cloud, such as Dropbox or 1Password. If your keys exist in digital form there is likely a way to trick you out of them or malware that can grab them. If the entirety of your crypto holdings is behind your Gmail password “s3cr3tp@ssw0rd” then chances are your coins are going to be stolen. Use a secure password, such as those generated by Make Me a Password. For even more safety, generate the password offline.

Hardware Wallets

At the same time that people were getting their funds stolen from the fake MEW, others were using the exact same malicious site without issue. These people had hardware wallets and did not need to enter their private key into the website. In the presence of a malicious third party they made it through unscathed.

If you have an amount of cryptocurrency large enough that you’d be angry at yourself for not spending $100 to protect it then you should invest in a hardware wallet. A hardware wallet holds your private key and doesn’t share it with anyone and resists almost every attempt to access it, digitally and physically. When it comes time to make a transaction, the software sends the transaction to the hardware wallet for confirmation and signing. The signed transaction is then returned to the software to be posted on the network.

Once a transaction has been signed it can not be tampered with—doing so invalidates the signature and the transaction will be rejected by the network. Hardware wallets give you an opportunity to review the transaction before signing it so you can confirm the address in the transaction is the address you wish to send the funds. There can be no clever sleight of hand to make you see one thing and sign another.

Always make sure you confirm the address on your hardware wallet matches the intended address. All sorts of tricks can swap out the address before you sign it but there are no tricks to change the address once you’ve signed the transaction.

Any of the major hardware wallets will work, it’s mainly a matter of features and convenience. I personally have a Trezor, a Ledger, a KeepKey, and a Digital Bitbox.

An alternative to the hardware wallet is to keep your key offline and only enter it into a secure computer. This process is called offline signing and is supported by many cryptocurrency wallets.

Backups

The converse to “don’t share your keys” is “don’t lose your keys.” The network only accepts your private key as the means to accessing your funds. If you drop your mnemonic seed in a puddle your coins are gone. There is no tech support to help you.

In the world of disaster recovery, one is none and two is one. If you only have a single copy of data or a single failover then if you lose that instance you are done. If you have a single backup (two copies) and you lose one then you still have a copy—but now one is none. For your private keys you should have at least three diversified copies. If you have your seed written on three pieces of paper and they are all in the same filing cabinet they will all be wiped out in the same fire.

An example of a good backup strategy is to have a hardware wallet, a copy of the keys in paper (or metal) form in a location with frequent access, and a copy in a more secure place such as a bank vault or a lawyer’s office. Check regularly that all copies of your key are present and undamaged. If you discover one of the copies to be damaged replace it immediately.

In addition to geographical diversity you should diversify the technology. Having copies of your key on three Ledgers means you are out of luck if the software is no longer available or Ledger-specific malware renders your devices inoperable. For an alternative to a paper wallet I recommend the use of a device like Cryptosteel or you can get a cheap metal punch kit and make a keychain from punched metal discs.

Encryption

If you need to store your key with someone and you don’t fully trust them you can encrypt your mnemonic with a password. General rules apply: use a strong password and don’t forget it! You will want to keep this password in a safe place or two.

There are also more advanced options like erasure coding (m-of-n encoding) and multi-sig wallets (n-of-m signing) that allow you to spread the secret out across multiple people such that some number of parts must be used to get the whole secret. For instance you can give part of your key to three of your friends (e.g. every first and second word to Doug, second and third to Sarah, and first and third to Taylor). In order to steal your funds two of your friends would need to collude against you. Harsh.

Operational Security

Finally, to keep your coins secure, keep your mouth shut. If people don’t know you have crypto they probably won’t try to take it from you. If you tell people how and where you store your keys it is easier for them to go after it. If you use a unique and secure password to lock your keys people can’t look over your should when you log into Facebook.

Social Engineering

The easiest way to hack someone is to use social engineering techniques. You can get 1000 people to click on a fake email long before you can crack a WiFi password or break into a “firewall with 256 bit encryption” (thanks Hollywood). Arm yourself against these kinds of attacks by understanding how social engineers can manipulate you. Set an internal alarm to go off anytime someone asks for personal information, your password, or your keys. Don’t trust any web site, no matter how official it looks, until you’ve confirmed the address and certificate.

Plausible Deniability and Duress Mode

Many hardware wallets and some secure systems have a duress mode or plausible deniability. If someone has a knife to your throat and is making you unlock your wallet you can enter the duress password and it will unlock a wallet that you’ve filled with a trivial amount of money. The bulk of your coins are safe.

Be Inconspicuous

Your lambo with the “BTCMOON” license plate is probably going to draw attention. The amount of effort and money you spend on protecting your coins should be proportional to the value of your portfolio and size of your notoriety. It’s not worth spending $100 to protect $20 of Ethereum, no matter how vocal you are about your holdings. If you’ve been HODLing since 2012 you probably should invest is a good security and disaster-recovery plan even if no one knows your crypto secret; doubly so if your stage name is DJ Dogecoin.

Conclusion

The security of your coins rests solely in your hands. Crypto is not yet at the stage where safety is foolproof. Do your research, be on the lookout for suspicious sites, and keep your private keys private. Don’t trust random strangers bearing gifts. Not every fork is a friend.

Were you hit by this phishing attack or any other? Do you have any tips I missed? Leave them below.

It was 2011, I was working at a large company and the Group Product Manager (GPM) had just committed to the VP of another team that we would finish our project in three months….and we had previously scoped the project to take six. I circled my pod and explained that since we had a limited amount of time we had to pick apart the specifications, identify the top-priority (P0) features, and work on those first. It was when I reviewed the checkin requests later that week that my jaw dropped.

The development team had gone into full-on panic mode. All the progress we had been making towards Agile development melted away as people fell back to what they were comfortable with. Huge multi-page specs were written and developers settled down to waterfall the heck out of this project.

One of the developers on the team had chosen as her very first task to implement the footer links on the bottom of the page. Not an entirely unexpected starting task as the web page needs a place to link to legal text, support, etc. What shocked me was the footers were implemented dynamically: An XML file defined the footer links and was used to generate the page content. So instead of modifying an HTML page we could modify an XML file. This was a lot of effort for something that was not in the critical path of the user experience.

Most software has a purpose. When a user sits down to accomplish a task there is a clear point at which the user considers their effort a success. At an ATM, the user is attempting to withdraw money and once they have bills in their hand they are done. Everything else is a distraction; if the user had their druthers they would walk up to a machine and it would magically spit out money¹.

The site we were building allowed customers to submit applications to a catalog. The customer’s task was complete when they had submitted their application for approval. Not a single step in the submission workflow involved the footer links. The meta success-criteria for our team was an internal beta that showed the submission process end to end—no one external to the company was going to see these footer links!

Any distraction that delays the customer from reaching the critical success criteria is probably inconsequential and very likely an annoyance to the user, making it a very good candidate for postponing as long as possible. On the same project I sat in two-hour design-review meetings with a couple of dozen people talking about how the hint bubbles were going to appear and disappear next to the forms. A man-week of effort lost arguing about pixel placement.

In some cases the distractions are necessary or beneficial. For ATMs, you are required to enter your PIN; this distraction is for security purposes and one people are willing to tolerate to keep their money safe. I remember when some ATMs started returning your card before giving you the cash. This added “distraction” actually acknowledged the propensity of a person to focus solely on the success criterion (receiving money)—people were leaving the ATM without their card. By swapping the order of operations the software could ensure a greater overall success of the transaction. As it turns out, there are multiple success criteria for ATM withdrawals:

• Get money
• Keep money secure
• Leave with card

What features are in the critical path of your users? What features are getting in the way of your users completing their goal? Stop working on the latter and spend your time on the former, you’ll be surprised how much more time you have. A user can use an ugly application that works; a beautiful application with no functionality is simply a piece of art which—unless you are a museum—isn’t going to make you a whole lot of money. And if you get pushback from a product manager just ask the magic question:

“Are you willing to delay the release for this feature?”

For more on this look into methodologies like Lean Software Development and ideas like Minimum Viable Product (MVP). They are very good at deferring all but the most critical tasks.

¹ For simplicity sake I am referring to a single use case. Of course there are many more reasons why people use an ATM.

A bug in our system is not just a nuisance, it means people don’t get their paycheck. It may mean someone misses their rent payment. It may mean someone can’t buy food for their family. Bugs in our system affect people’s lives.

Now the software you are working on may not have the same level of impact as ours but it doesn’t mean that lives aren’t impacted. People deal with buggy software everyday. Think how much of your brain is reserved for knowing the workarounds of software that doesn’t behave as it should. Reducing your bug count could be collectively add years to people’s lives.

In practice I spend more time on tests than I do on the code. The exception is when I am in exploration mode and the key outcome is learning. One line of code may spawn half a dozen tests, because of this writing tests has to be both quick and easy while still allowing me to come back later and understand what is going on.

I am very partial to Behavior-Driven Development-style (BDD) tests. These tests code the behavior of the system—not how a particular piece is coded—and these behaviors are usually specified as business requirements. BDD tests generally have three parts: Arrange, Act and Assert (also known as Given/When/Then).

Why Are You Testing?

One time, during a presentation on readable tests, I asked for a volunteer to share their test code. My boss at the time offered. Looking at the tests I said, “Hrmm, ConstructorTest1, it looks like the purpose of this test is to get to 100% code coverage.” Code coverage is a laudable goal, but using code coverage as your driver doesn’t result in the best tests, you are merely finding the shortest path to exercise code that was already written. On the other hand, when you set out to test behavior you are thinking about code in the context of how it fits into the system.

For instance, if I’m writing a method that will generate a payment instruction I’m not wondering “can the method handle a negative number without crashing?” I’m thinking “what should the behavior be if a negative number is supplied” and “how does that behavior change if I adjust the context?” This will often lead to discussions with the product owner, such as “do we want to allow refunds using this API by sending negative payment values?”

Incidentally, if code-coverage is your goal then you should work towards test-first development. If you write the minimum amount of code necessary to get the test to pass then you end up with near 100% code coverage automatically and that’s a beautiful thing.

Structure of a Test

When our team writes tests, we follow a fairly standard layout. Here is an example test in C# using [Machine.Specifications] (MSpec):

[Subject(Foo, "CheckIsValid")]class FooTest_CheckIsValid_WithValidData {    const string ValidInput = "non-empty string";    Establish a_profanity_filter = () => {        MockProfanityFilter = new Mock<IPofanityFilter>();        MockProfanityFilter.Setup(m => m.HasProfanity(Moq.It.IsAny<string>()))                           .Returns(false);        TestSubject = new Foo(MockProfanityFilter.Object);    }    Because of_passing_valid_data = () => ActualResult = TestSubject.CheckIsValid(ValidInput);    It should_return_no_error = () => ActualResult.ShouldEqual(May.NoValue);    It should_check_profanity_filter = () => MockProfanityFilter.Verify(m => m.HasProfanity(ValidInput));    static Mock<IProfanityFilter> MockProfanityFilter;    static Foo TestSubject}

As a quick introduction to MSpec, it uses lambdas to define the Given/When/Then (Establish/Because/It) of the test. The Subject attribute defines the test and also groups test output hierarchically. You can read this test as “Foo:CheckIsValid, when given a profanity filter, because of passing valid data, it should return no error and it should check profanity filter.”

At the very top of the test we define class members that define key context for the test. In this case, we put the input we are going to be using. Both by the name and the value we help describe what valid content is. It’s not a complete list as we will spend more time defining and testing invalid input in subsequent tests. At the bottom of the class are incidental members, variables that handle bits of the test but aren’t giving us a lot of information. It’s more important the detail of assigning TestSubject a value than seeing how it is declared.

In the Establish section we define the context for the test, including setting up any of the test subject’s dependencies. In this case we create a mock profanity filter that always returns false, valid input is also not profane. At the very bottom of this setup the test subject is created and we can see all parts that are passed to it.

Note that the entire setup is present in this class. There are no base classes or common code hiding several directories away. Everything needed for the test is visible in a single class on a single screen.

Some people like automock or other helpers here. I tend to avoid things like that. To me it’s more important that I can see all of the dependencies and how they are defined rather than rely on magic that I can’t see. It also lets me see when the subject starts getting too large to be easily testable. If I can’t setup all of the dependencies in just a few lines of code chances are the class or method is doing too much.

Once we’ve defined the context we then have the Because clause that does the actual work. In most cases this is a single line of code acting on the subject, here we call the method we’re testing with the input and capture the output.

Finally, we have the assertions. There are the It clauses and each one defines a different result we expect to see. There is only one assertion per It so that any failures are unambiguous. Also, we only assert that which we care about for the behavior. If the subject writes a log message we don’t check it, unless the log message is part of the business logic (i.e. logging an audit message when an account is deleted).

Let’s look at another example from Javascript (actually Livescript). This is from a Hexo plugin that gets picture data from Unsplash and drives the awesome pictures that front every blog post. I have removed a lot of the cases here for brevity.

describe \parseUnsplash ->  sut = parseUnsplash  const default_crop = \entropy  specify "it should return undefined for a local file" ->    expect(sut "foo.jpg").to.be.undefined  describe "when using long form specification" ->    specify "it should return undefined for a different site" ->      expect(sut "http://example.com/").to.be.undefined    specify "it should match http" ->      expect(sut "http://unsplash.com/?photo=37121").to.not.be.undefined    context "without a specified crop" ->      const expected_id = \0987      result = sut "https://unsplash.com/?photo=#{expected_id}"      specify "it should return id" -> expect(result.id).to.equal expected_id      specify "it should return default crop" ->       expect(result.crop).to.equal default_crop    context "with a specified crop" ->      const expected_id = \1a2b3c4d      const expected_crop = \face      result = sut "https://unsplash.com/?photo=#{expected_id}##{expected_crop}"      specify "it should return id" -> expect(result.id).to.equal expected_id      specify "it should return expected crop" ->       expect(result.crop).to.equal expected_crop

Here LiveScript, Mocha and Chai combine to offer a very succint syntax (the -> operator defines a function which is passed as the second parameter to the specify, describe, etc functions). We don’t have much in the way of context, so we just set up the subject under test (sut) which is simply an imported method. At the beginning we set the value for default_crop since it holds true across the specification. First we have a simple test for the local-file case. We then jump to the long-form version of specifying the Unsplash image, which in this case is a URL. There are several cases nested under this section. For the case “with a specified crop” we put the pertinent data for the case at the top of the method, like we did with C#.

Each describe entry gives us another nested context that is exposed in the test results.

  unsplash    parseUnsplash      ✓ it should return undefined for a local file      when using long form specification        ✓ it should return undefined for a different site        ✓ it should match http        without a specified crop          ✓ it should return id          ✓ it should return default crop        with a specified crop          ✓ it should return id          ✓ it should return expected crop

This allows you to build up rather complicate contexts that build upon the earlier, simpler contexts. You can also see how the specification reads as plain English:

when using long form specification, without a specified crop, it should return default crop

Note that for simple contexts I omit the “when” clause altogether: “it should return undefined for a local file.” Sometimes the context is so simple that it doesn’t need a separate clause. Experiment with your test structure and find what works for you and what doesn’t.

That repository actually shows off a bunch of different patterns I use when trying to simplify the tests and make them readable.

Deciding What to Test

Almost as important as how to structure tests is what to test. In the next installment I’ll talk about how I go about deciding this and how to divide the behavior into testable chunks.

Recap

These are the top principles I use for effective automated testing:

• Use BDD to test the behavior instead of the code
• Present the entire context; avoid hiding test setup in other locations
• Structure tests so the important stuff is up front
• Avoid “magic”

Any comments or questions? Leave them below, I’d love to hear what works for you.

“Let’s create an implementation of the Transaction Store implemented in memory.” It was kind of absurd but it was also exactly the right thing to do. It took almost no time to build and test and we didn’t have to worry about migrations, ORMs, etc. We were a long way from production and all we needed was a place to store transactions so that we could work on payment processing.

By deferring the decision about how to store the data we were able to focus on the data itself and how that data was used. We were in a much better position to answer all of the questions once we had a better picture of the shape and usage of the data. This also allowed us to discuss the merits of each solution in the context of the actual problem instead of everyone focusing on their favorite data-storage technology.

Something similar happened several years ago on the Microsoft Store team. The product needed an engine to calculate discounts on the site. A vendor had quoted us a couple of months and they weren’t willing to commit to a date that we wanted (plus the design was bonkers). With one month before ship and a blank slate I sat down to tackle the problem myself.

The first task was asking the business team to give me the top five discount types they wanted to handle. Dollar off, percent off, linked discount (i.e. discount on B with purchase of A), promo code, etc. I sat down and started knocking them off the list one by one. I didn’t pre-plan the schema, just relied on good tests and simple migration strategies.

Evolution of the schema:

• Dollar off: [ SKU | Discount ]
• Percent off: [ SKU | Discount | Fixed/Percent? ]
• Linked discount: [ SKU | Discount | F/P? | Required SKU ]
• Promo code: [ SKU | Discount | F/P? | Required SKU | Promo Code ]

By the end of the month I had implemented the top 4 or 5 discount types they wanted giving the business team several months of runway. If I had tried to design a schema to handle all of the discounting types the business team could imagine before writing code it’s unlikely I would have had a plan in the first month.

This wasn’t without its problems. Due to lack of time and resources the “user interface” for the business team was a spreadsheet that they uploaded to the website. And in later iterations the design suffered from combinatorial explosion: when a discount was “buy a copy of Windows and get 10% off a mouse” you had to deal with multiple SKUs for Windows (in the “Required SKU” column) for each mouse SKU. But it worked and every discounting type that was asked for could be handled with the existing schema (promo code for a linked discount? we can do that) or a simple change to the schema.

Side note: solving the combinatorial-explosion problem turned out to be pretty simple. We were building a new UI to enter discounts (to replace the spreadsheet method). The business team wanted to be able to select product families instead of per SKU. Rather than rip apart the data store we built a UI data store with enhanced information and then projected to the existing data store. This meant we didn’t have to change any of the code in the discounting engine and we were able to decouple changes in the UI from changes in the engine and vice versa.

Many people are nervous about delaying decisions until the last possible moment. There is comfort in an up-front design. In my experience, however, by delaying decisions until they absolutely have to be made you have the most information available to make the decision, which can only be a good thing. So many of my “last-moment” designs turned out much more elegant than I could have come up with at the start.

One example I like to use is a dApp that came out last year called Moon Cat Rescue. You use your browser to search (mine) for cats and then you can rescue them, name them, and sell them. Something happened on the way to release, though.

Are you getting paid for this?

Nope. We intended to collect ether from the sale of the genesis cats. As it turns out however, a fix we made during our QA process led to those funds being locked away forever. But that is okay.

If you destroyed your entire business model would you be “okay” with it? Looking at the contract source it’s pretty clear what the bug is, when a genesis cat is created the contract owner isn’t added to catOwners[catId]. When the cat is later adopted—transferCat(catId, catOwners[catId], msg.sender, offer.price);—the funds go to 0x0 which has so far racked up 5.4 ETH in the contract. An automated test that included adoption of a genesis cat and its subsequent withdrawal would have caught this.

Another example shows that even when you are up to no good a good test can help you out. Honeypot contracts are fun little Easter eggs hiding in the Ethereum network. Because many smart contracts have bugs people will go out looking for exploitable contracts. Honeypots are contracts set up to look exploitable but actually trap the “exploiter”‘s money and hold it for the creator. One such contract had a subtle little bug in it.

I won’t detail the inner workings too much except that it takes advantage of shadowing. The bug in question is on line 89: _addr.call.value(_wei); Can you spot the problem? It’s a confusing aspect of the Solidity syntax (but one familiar to Javascript programmers). The statement is missing the final parenthesis to actually make the call: _addr.call.value(_wei)(); Imagine the surprise when the contract creator attempted to withdraw his 1 ETH bait only to find a bug in the contract locked their funds forever.

My final—and favorite—example is this contract posted to Reddit. My analysis showed that the contract was vulnerable to an attack that could drain all of the funds from the contract. The core of the vulnerability lay in an extra equals sign: members[msg.sender].isPermitted == false; Sometimes a single character is all that stands between you and financial ruin.

Testing is an important component to maintainable software; but even when you don’t expect to maintain the software past initial release, automated testing can help you prevent last-minute bugs from creeping in.

Or don’t, I’m happy to keep collecting bug bounties.

interface ICapacityProvider {  bool GetAndUpdateCapacity(string customerId);  void SetSettings(CapacitySettings settings);  CapacitySettings GetSettings();}class CapacityApi {  bool GetCapacity(string customerId) {    return this.customerService.IsEnabled(customerId)        && this.capacityProvider.GetAndUpdateCapacity();  }}class CapacitySettings {  public Duration Period { get; set; }  public int UserLimit { get; set; }}

I stated that I found a few things “weird” or suboptimal and we proceeded to discuss it. This discussion drew out a few ideas regarding code organization that I want to share here.

Intention in Naming

I had an objection to the naming of GetAndUpdateCapacity. Usually when I see a name with “and” in it I immediately check it for a violation of the Single Responsibility Principle. In this case the method checks to see if there is capacity and if so records another usage. Since this behavior is so closely connected it didn’t make sense to split the work into two methods.

The implementation of GetAndUpdateCapacity at its core is:

if (tokens.Count < capacitySettings.UserLimit){  tokens.Add(now);  return true;}return false;

Since clear intention is an important principle in our team we are careful to make sure that names convey what the code is doing. The team felt the name accurately described what was happening and that it was important the name captured the update so that callers wouldn’t be surprised when state was mutated. I agreed.

And also disagreed.

ICapacityProvider is an interface. The team had decided to use the Strategy Pattern and therefore the expectation is that the interface can be implemented by many different algorithms, many of which would not actually update anything, e.g. by querying a tally of events made by some other process. By including “AndUpdate” in the name the interface was signaling an implementation detail that was not true. This is mirrored by the fact that ICacapityApi‘s method is called GetCapacity instead of GetAndUpdateCapacity—the fact that updating was happening was not passed along the hierarchy.

In actuality, the team was really trying to signal that the code was not idempotent. However “AndUpdate” doesn’t really signal that as there are plenty of update implementations that could be idempotent. In reality, the language is missing any sort of contract or signal that a method is idempotent (short of adding our own [Idempotent] attribute) so an attempt was made to signal that intention through naming.

Finally we settled on the name TryRegister. Still not a great name but it better indicates what is happening. The “try” implies a Boolean success/failure result and “register” is a better clue of what is happening—recording the use of the system—that doesn’t necessarily imply an “update” is happening.

Advanced SRP

The next thing I had an issue with was having {Get,Set}Settings on the same interface as TryRegister. For context, the team was in the process of making the settings configurable, in this case CapacityApi was getting matching members that would be called by and REST endpoint to receive the settings. Their justifications for putting settings get/set on the capacity provider were:

• This had to do with capacity.
• The capacity provider needed access to ISerialization anyway.
• The capacity provider was the best class to know about settings.

I asserted that putting the settings there violated the Single Responsibility Principle for a couple of reasons. By choosing the Strategy Pattern it meant that there could be multiple algorithms to determine if we had capacity. Likewise, it’s reasonable to assume that there could be multiple strategies for storing the settings. However, the strategies for capacity and the strategies for settings did not correlate so the coupling was unnecessary. Changing a capacity strategy also meant settings might be fetched from a different place ncessitating a change in behavior unless care was taken to maintain identical settings in each location. Furthermore, some algorithms might not use the settings at all!

Additionally, some strategies for retrieving settings might not have a corresponding method for configuring them. They may be hardcoded, pulled from something like Zookeeper, or calculated based on instance count. This would lead to some providers having SetSettings throw NotSupportedException which would violate the Liskov Substitution Principle.

In fact, we were preparing to change where we stored the settings to a different location that allowed us to update them more easily. That meant that we would be passing in a second ISerialization interface which further showed that settings were orthogonal to the algorithm, not to mention the confusion and difficulty with having two parameters of the same type when dealing with an Inversion of Control container.

Strategy for Strategies

A final part that I felt could be changed is the actual implementation of CapacityApi. One thing I found interesting was this piece of code:

return this.customerService.IsEnabled(customerId)    && this.capacityProvider.GetAndUpdateCapacity(customerId);

We had chosen a strategy for capacity but were calling into another service to do the initial check on the user. I proposed that the first half of the expression was itself a strategy for determining capacity. It was—to my first point—a strategy that didn’t update and—to my second point—didn’t use the settings.

The first expression was rewritten as:

  class ServiceEnabledCapacityProvider : ICapacityProvider {    bool GetAndUpdateCapacity(customerId)=> this.customerService.IsEnabled(customerId);  }

And here is the new implementation of CapacityApi:

public class CapacityApi : ICapacityApi {  // multiple ICapacityProviders registered with IoC container in order of priority/effort  readonly IOrderedEnumerable<ICapacityProvider> capacityProviders;  // Constructor omitted for brevity  public bool HasCapacity(string tenantId, string userId)    => this.capacityProviders.All(p => p.TryRegister(tenantId, userId));}

Simple, succinct, and very testable.

Over-Engineered?

One question came up as to whether this solution was overkill. That’s of course subjective and difficult to answer. We certainly didn’t have plans for additional algorithms so worrying about naming and factoring wasn’t important from a YAGNI point of view. The team initially chose to use a provider, though I made the case that CapacityApi could have easily handled the contents of CapacityProvider without straying too far from the Single Responsibility Principle.

Ultimately, I let principles and the context guide how far I go into engineering. Since we chose the strategy pattern it meant that splitting the algorithm from the settings was necessary. But maybe it wasn’t necessary to move the IsEnabled check into a separate provider.

Never Perfect

There are many ways in which this particular design can fall apart and in other contexts this may be overkill or under-engineered. Having multiple simultaneous providers means that if two providers update usage, the first may update a usage only to have the second declare no capacity in which case the first provider is over-stating its usage potentially denying users later that would otherwise be let in. This is something we have to watch and potentially add a two-phase check/commit process increasing complexity.

We ended up with three separate single-method interfaces. For half-a-dozen lines of “real code” we created six files, plus tests. Part of this can be blamed on the language, but we also could have gotten away with putting those 6 lines in a single class and been done with it. As functionality is split off into separate files it decreases readability as you have to piece together all the pieces of the puzzle to have complete context.

In the end, design is a trade off between conflicting principles an often times it can come down to a coin flip. My goal isn’t perfection but to have a mindfulness of the principles, understand the tradeoffs, and stop when it’s “good enough.”

Styles of Development

Solo Development

When I first started in software the most common method of working was solo development. It still is the most common method of working. Depending on the methodology a developer would get a specification and go off to write code. In Waterfall the developer would hibernate in their cave for a few months, come out, rub their eyes, and hand off a chunk of code. Modern methodologies have the developers syncing up more often, but the bulk of the work is still done in isolation.

One of the biggest problems with solo development is that in the time between synchronizing with other developers a solo dev can get off track. With daily syncs it can’t get too far track but it is still a tax on development. Without daily syncs it can get worse. I once had war room practically in tears when I reported that two parts of the product absolutely refused to work together. I was the first person to use the feature—we were triaging to get to code complete by this time—and the two halves were so far out of alignment that no workaround could fix it.

Pair Programming

In pair programming two developers work together. One developer is the driver and has hands on keyboard. The other developer is an observer that reviews code as it is written. The two often discuss approaches and solve problems in real time. The role switches often.

One of the benefits of pair programming comes from two sets of eyes looking at the problem. Each developer has a unique perspective and unique skills allowing for more options in solving the problem. In addition, the observer is not burdened with the ask of assembling the code and can keep an eye out for mistakes that can arise when a single developer is focused on the task. For instance, intending one thing and typing another.

Mob Programming

Mob programming takes the concept of pair programming even further. If two sets of eyes is good, more sets of eyes must be better!

Mob in Practice

At about the same time every morning the mob starts to form. Three to four developers jump on voice chat. One of the developers shares out their screen and then the mob gets to work. For the next four hours or so the team is focused on a single task.

Many times the driver is making progress fine on their own. The observers can just observe or split off into a chat about pasta. Other times the driver needs some help and the team can discuss solutions. If the driver starts straying off course or falling into an anti-pattern the observers can jump in.

Observers are watching the code that is being written, thinking about the problem, and keeping the driver motivated. The observers are also available as a reference either from memory or being able to look up reference docs quickly without needing the driver to switch contexts.

Tooling

We use Google Hangouts for voice communication and Teamviewer for screen sharing. Teamviewer has a feature where participants can click on the screen and the presenter can see which is important for pointing out lines of code. Teamviewer’s voice capabilities weren’t as good as Hangouts.

We experimented with various tools, including those that allowed multiple developers to write the same code at the same time. They just didn’t work well.

Code Reviews

Our code-review guidelines require at least 3-sets of eyes in order to merge: the author and two others. Mob programming allows us to get those eyes as the code is being written. There is usually a code review performed as the code moves from a mob branch to a feature branch or to master but those are mainly to make sure nothing was missed as a group. It’s unlikely that any significant changes in direction need to be made at this point.

Motivation, Team Rapport, and Enjoyment

Everyone has good days and bad days. Sometimes you get up and you just can’t get your brain in gear. You stare at an empty IDE and can’t get past public static void main() {}. When you’re in a mob, you can let someone else drive. You can be an observer. On those days when you think, “I can get away without a test here,” you have a team member encouraging you to write it, or willing to take over and write it themselves. The mob works at the level of the strongest individual for that day, lifting others up to their level.

When a team is communicating everyday for hours at a time them team gels more quickly. Companies exert effort to have team-building exercises to help everyone know each other, why not a team-building exercise that involves writing production code. My team communicates more than any other team I’ve been on–even teams in open workspaces sitting 4 feet from each other–and the cohesion shows. Everyone on the team is familiar with all aspects of the code so there are no silos of information.

Writing code in a mob is just more fun. It becomes a social exercise. In an experiment we went back to solo development to see if we were faster (read on to find the answer to that) and what we found was, faster or not, people had much more fun working together.

Realities of Mob Programming

It Isn’t Slower

In the lifecycle of code, only a small part of the cost is in the initial writing of code. The majority of costs comes from bug fixing, maintenance, change requests, and the like. Code written in a mob rarely needs to be rewritten when developers diverge from each other. Code written in a mob tends to have fewer bugs. Code written in a mob has better design.

When we did our solo experiment we noticed that while the initial code was written faster—of course, we could generate up to 4x the lines of code than mobbing—overall time took longer as developers had to take time to do code reviews, respond to code review feedback, and fix bugs. The code tended to vary wildly in quality and design, and broad knowledge of the code was lost. We were faster as a group.

There are times when we split the mob for speed. Spikes are a good example; we each tackle an potential solution—solo or in pairs—and report back. The code is then re-implemented as a mob with full quality. Tasks such as filling out logging or writing repetitive tests can be broken up since the core design has been decided and all that’s left is filling out code.

It Doesn’t Work for Everyone

If there is one pre-requisite for mob programming I would have to say that there needs to be a lack of ego among the developers. Mob programming can’t be a contest of “I’m right” or else work won’t be done. My team respects each other, is willing to discuss options, and is willing to be wrong. We all have the view that you can learn something from anyone. By putting the product first and ego last it is possible to work together.

Of course different personality types can perform better or worse in mob. If you are someone who has trouble communicating with people and prefer to talk to machines it may be difficult to adapt to the social nature of mob programming. I have varying levels of social skills depending on how hard I’m thinking; when I am deep in thought socials skills are paged out and my responses can be short and lacking empathy. My coworkers know this and give me leeway. Usually, though, the cognitive load is spread across the mob so deep thought generally isn’t required.

Another aspect that can be tricky to manage is different types of thinkers. I’ve found that there are some people that prefer to see deep and narrow on a problem and some that prefer to see shallow and wide. Some want to know the whole picture while others can work in the abstract and fill in details as needed. Sometimes this difference in styles will clash with one side screaming “get on with it!” and the other side screaming “I don’t have enough information!”

I’ll explore some of these issues with more detail in a later post.

First, I’m a very curious person. As a kid, computers always outpaced my curiosity so I continued to chase it, like a greyhound chasing the mechanical rabbit. I was hungry and eager to learn more.

I think more than anything else the one thing that has brought me to where I am right now is the phrase “there’s got to be a better way.” It became a mantra of sorts at Microsoft where I saw dysfunction all around me. I was always on the look out for new technology, tools and patterns that would help me do my job better and more efficiently and help me avoid traps. I had the thought that if only syntax was different or if there was more syntactic sugar all our development problems would be solved.

While there are aspects of programming languages that can improve the development experience and reduce the chance of bugs (see generics in .NET compared to Java) almost any modern language will suffice for writing good code. The more important factor is how code is written and that is where I believe the answer lies. I have built a philosophy around writing code that focuses on maintainability and sustainability so the choice of programming language is only a small part of the system. Sure it’s better if a language has Inversion of Control built into the language, eschews global variables, and prevents bad patterns; those aren’t necessary. I don’t need a language to force me to write good code because now that I know the principles I don’t (usually) want to stray from them. And language syntax isn’t going to help you learn why principles are important.

That’s not to say if someone built a programming language from good, solid programming methodologies that it wouldn’t make it better code. It would be like bumpers helping keep your code within the walls of good design.

Ultimately, my search for a “better way” to write software has slowed. Sure there are things that could be better. But with the team I’ve built and the process we’ve developed the job of writing software is enjoyable and the quality of code we create is very high. Now the “problems” to solve are around inter-team dynamics and external relationships. I mean, there has to be a better way, right?

Based on that experience—and others like it—I have focused on learning why software development goes awry. I have researched and experimented with the how-tos of designing software and building teams so that the development process is smooth—and how to handle it when things goes wrong.

Here are some key points that define sustainability for me:

• The team should work at a pace that they can sustain indefinitely.
• Code should be maintainable so that a given pace produces similar (or better) output.
• People can’t write software faster; they either write less software or spend more effort writing software in a given day.

I will be exploring these topics in more detail on this blog.

My mission statement has remained relatively unchanged in the last 20 years:

Software is broken and I want to make it better.

I do this by learning new techniques, improving the way I write software, and sharing my knowledge with others. I build teams so they are set up to write great software and I push against the usual suspects that prevent developers from doing so. This blog is an extension of that. I hope you enjoy it and learn something from it.