Yesterday, this blog post documented how a bug accidentally cost Carlo $6500 in a few hours because his AWS keys were compromised. Please read the whole post.
Synopsis: He was using the GitHub Extension for Visual Studio. He published a new repository that he *thought* would be private, but due to a bug, he created a public repository. That repository contained his Amazon Access Key. Scanners found it and created lots of resources.
First, some good news and a task for all of you: This issue was reported on the GitHub project for that extension (https://github.com/github/VisualStudio/issues/62) That page shows the quick response by the team involved for that extension, and a fix has already been implemented, and deployed to the Visual Studio Extension Gallery: https://visualstudiogallery.msdn.microsoft.com/75be44fb-0794-4391-8865-c3279527e97d
A task for all of you: If you are using the GitHub for Visual Studio Extension, update it now. You’ll get a notification from Visual Studio that there is an update available. Install it. Now. (This extension does not auto-update, so you will need to perform this task.)
One fact I learned yesterday was that there are organizations that constantly monitor Github commits to see if any contain credentials to AWS or other cloud providers. That is the price of success. A tremendous amount of code is added or modified on github.com every day. It’s an attractive target.
As soon as a public repository gets created, or updated, that information will be captured. Be careful.
Clearly, hindsight is 20:20, I’m writing this to share a few tips that might help you. I’m also interested in hearing what techniques you use to manage these kinds of secrets. My experience is with the open source projects for Humanitarian Toolbox. Those projects are deployed on Azure, and are all Open Source projects on GitHub. That means we must manage to keep secrets out of the repository. Here are the recommendations we use:
The default web.config files that are in the open source repositories are set for a localDB on the developer’s drive. There are no keys for our deployed resources.
Instead, we use the Azure Portal to configure any and all secrets for our sites. (click on the image for a larger view..) This does mean that we, as an organization, must protect our Azure portal settings. And, folks like me must make sure to obscure the secrets in images like this.
We deploy our applications directly from GitHub source control. The staging site deploys from the master branch, whenever we merge pull requests. If you are one of our contributors, you can see your changes live as soon as we merge the pull request.
For release builds, we use the Release branch in GitHub. One of the commit team must merge changes from master to the release branch for those to be live. We do that after our partners have had the chance to validate the changes on Master.
For my projects that are not planned to be open source, but will remain private, I use Visual Studio Onilne. (https://www.visualstudio.com/en-us/products/what-is-visual-studio-online-vs.aspx) It supports Git, and has some pull request support for teams. (I’d like a model where I could use Pull Requests even with my shared repositories, but I haven’t found that feature yet.)
VSO is designed around a team model, not an Open Source model. It’s less likely (well, almost impossible) that I accidentally make a public repository on VSO.
The key takeaway for me is that the era of Open Source requires all of us to learn some new skills and take care of new potential risks. These security breaches can and do happen to experienced people. Teamwork, care, and multiple levels of defense are needed.
Most importantly, professionals did their job: I’m truly impressed with the speed and professionalism of the response to this issue by team members from both Microsoft and GitHub.
Last weekend, Humanitarian Toolbox held a very successful coding event at That Conference. We got quite a bit done on both the crisis checkin and the All Ready applications. Thanks to the organizers of ThatConference, and everyone that attended during the weekend. I’m always impressed by how many developers join us there, and by how much they contribute.
OK, enough of the public service announcement. I wrote this post to help with a common issue: Pull Requests vs. Commit Privileges.
All the Humanitarian Toolbox projects use the Fork & Pull model for development. It enables us to keep the core contributor teams small, and yet enables anyone that wants to contribute to make changes and submit them. I, sadly, didn’t announce that clearly enough to all the volunteers when we started our event. Many of the volunteers thought we were using the Shared Repository Model.
That meant that later in the event, I had a number of developers come to me with issues because they could not commit to the main repository. That’s because they didn’t have the rights.
Announcement: Don’t fix this by just giving commit privileges. It’s not necessary.
Announcement II: Don’t fix this by trying to copy your changes and merging by hand. It’s not necessary either.
Thankfully, this is really easy to fix, once you understand how git works. It does also require using the git command line. The git command line isn’t that hard, and it’s important to learn as you do more with git. If you ever run into this issue, follow the instructions below to fix it.
Before I explain the steps to fix the problem, let me describe what happened, and why it’s a problem. Look at the image below. It’s a portion of the main window in my Github for Windows application. You can see that I have two copies of Crisis Checkin on my local machine. The top one is the clone of a fork that I made in my account. The bottom one is the clone of the master HTBox repository. These two copies have different remote repositories.
If I run ‘git remote –v’ in both directories, you can see the difference. Here’s the output from my fork:
C:\Users\billw\Documents\GitHub\TheBillWagner\crisischeckin [master]> git remote -v
origin https://github.com/BillWagner/crisischeckin.git (fetch)
origin https://github.com/BillWagner/crisischeckin.git (push)
Note the difference when I run the same command in the main repository:
When you execute a ‘git push’, you’ll send your changes to the git repository identified by origin (by default). If you cloned the main repository, git will try to push to that main repository. If you forked, and then cloned your fork, git will try to push to that forked repository.
The goal is to push the changes you made in your desktop to a fork of the repository where you have commit rights. The drawing below shows the relationship between the repositories, and the commands used to create each one.
To save your work, and create a pull request, you’ll need to create a fork, and push the work you’ve already done to that fork. The specific commands are as follows:
I usually create a fork from the github.com website. (Just press the “fork” button in the upper right corner). That creates a copy of the main repository in your account. This copy is also located on the github servers, not on your drive.
This fork is where you want to push your changes.
Now, you need to configure your fork as a remote repository for your working directory. Looking at the image above, you want to push from your desktop (where you have made changes) to the fork (the upper right repository).
You add a remote by typing:
‘git remote add fork https://github.com/BillWagner/crisischeckin.git’
Replace ‘fork’ and the URL with a name you want to use and the URL of your fork. I use ‘fork’ as the name, because it’s easy for me to remember.
You can add new remotes as long as the additional remotes are related to the origin remote. (Unless you start forcing git to do things. That means you can’t accidentally push your changes to crisis checkin to a fork of the Roslyn project (for example).
Now, your local copy is configured with two remotes: The source for the application (owned by HTBox) and your fork (owned by you). These two remotes are named ‘origin’ and ‘fork’.
Now, you need to push your changes from your working area to your fork. That’s just a git push, and specify the fork as the remote:
‘git push fork’.
By adding the extra parameter to the push command, you specify the remote repository where you changes should go.
It’s that easy.
Unless it isn’t. If it has been a while since you cloned the original repository, you may need to sync your working directory with upstream changes. That’s documented here. It’s variation of what I just described, but you should be able to follow it.
After you’ve pushed your changes to your fork, you can open a pull request to discuss your changes and get them merged into the main repository.
After I’ve finished this work, I will often make a new clone of my fork for a given repository. That way, the default remote (referred to by ‘origin’) points to my fork, rather than the main project.
I hope that helps.
For the past year, I’ve been creating Github repositories to store the code demos for my public presentations. I’ve found it had quite a few advantages. Now, some other speakers are starting to follow the same procedure (or even extending it). Earlier this week fellow MVP Joe Guadagno asked me about how it works, and then used it for his next presentation.
He also started a broader conversation about the technique on twitter. A number of other speakers chimed in with their thoughts. Twitter has great immediacy, but doesn’t let me explain how I do this at length. In this post, I’ll discuss why and how I use git repositories for code presentations.
I’d grown very dissatisfied with the common practice of having a zip file for the starter project and a zip file for the finished code. Way too much information was missing. My best presentations show attendees more than just what to do. I also discuss alternatives that don’t work correctly, and why. These small side steps are missing when the only assets are “before” and “after”. Missing is “why choose this path?”
What was missing was any roadmap to “tried this, found these problems and changed directions.”
I can preserve all that history using a source control system. In fact, I’d been using my own internal source control for presentation demos for years. I could practice by rewinding the history, replaying the checkins and seeing each step along the way through the demos.
Github has now made that possible for anyone attending the presentation. It’s public, and I can represent each step of every demo as a branch in the repository.
My process starts after I’ve thought through the demonstration that I want to show during the talk. This hasn’t changed at all: what do I want people to learn, and how will I explain those concepts?
Once I have the story, and the arc of the demo, I’m ready to make the repository. I run through my practice sessions (using git at each step) and the I’m ready to record.
I make the “start” project for the demo. Maybe this is direct from File: New Project, or maybe it’s a starter project that already has some features.I’ll create a new git repository, and make regular commits as I get to the initial state. Once I’ve reached the initial state, I make a branch. For these repositories, every branch starts with a 2 digit number, so that my talk attendees can easily follow along. The first branch is likely “01-Start” or something similar.
I continue to build the demo, by following my script. I use commits for small-grained changes that I want people to follow, if they are working on their own. I use branches for larger-grained changes that I discuss during the talk. You can see each section of the talk by checking out each labeled branch in turn. You can follow each commit to see all the smaller changes that make up each new feature.
One very powerful feature that I use for talk demos s to be able to show what happens down the wrong path. For example, I may make an async demo where I’ll explore the problems that happen from using async void methods. Letting the audience see those mistakes is important for their learning. Using Git, I can explore dead ends, and leave those branches as examples of problems. I can back up and checkout an earlier branch, then show the correct path. Attendees can follow the straight-line path, or check out the dead ends for reasons why certain paths are avoided.
Once I’ve finished running through the script of my demos, I check the repo and the commits and branches in order one last time. I make sure each of the branches can be checked out in order and that each branch correctly represents the next checkpoint in the demo.
Having a repository does not mean I don’t write code during the presentation. Before the talk, I checkout the starter branch. There are two rules I follow during the talk:
You may notice that these two rules contradict each other. I do show some of the changes as I move forward through the demo. But, I don’t type everything out. Often, I prefer doing a ‘git checkout’ to move the code to the next step.
If I don’t get any questions, comments, or feedback on the discussion, the demo goes forward from its starting branch to its final branch. I’ll mix typing some of the changes (or using the Visual Studio tools to make large-scale changes, depending on the demo), with using git to move forward by checking out the next branch.
Because I’ve done some typing, git complains when I want to roll forward from branch to branch.
One important tip is to remember to do a ‘git checkout –f’ so that you don’t have to stop, rewind anything you typed, and then move forward.
I can also go off script to answer questions or respond to different feedback. This changes the demo, but I can get it back on track by checking out the next branch.
I don’t have hard and fast rules for when I type vs. when I roll forward in git. My heuristic is that when attendees will learn from seeing the code typed, I type it in. When that transformation is not instructive to the topic, I roll forward. I’m not certain I always get this right, but I like the guidance so far.
At the end of the talk, I tell attendees the URL of the repository, and tell them to explore.
Github provides several tools to continue the conversation with attendees after the talk. I’ve used all of these after different talks.
This process is still somewhat new, and I’m sure I’ll continue to refine and change how I do it in the coming years. However, I really like where it is heading.
What do you think?
All of these projects are Open Source (using the Creative Commons license for content, and the MIT license for code). If you would like to contribute, visit our GitHub Repository. Or, if you have questions, comments, or ideas for improvement, please create an issue for us.