Bill Blogs in C# -- OpenSource

Bill Blogs in C# -- OpenSource

Created: 2/9/2016 4:47:17 PM

I’m continuing my discussion on proposed C# 7 features. Let’s take a brief look at Slices. The full discussion is here on GitHub. Please add your thoughts on that issue, rather than commenting directly to me. That ensures that your thoughts directly reach the team.

Let’s start with the problem that Slices are designed to solve. Arrays are a very common data structure (not just in C#, but in many languages.) That said, it’s often that you work with a subset of an entire array. You’ll likely want to send a subset of an array to a some method for changes, or read only processing. That leaves you with two sub-optimal choices. You can either copy the portion of the Array that you want to send to another method. Or, you use the entire array, and passes indices for the portion that ahould be used.

The first approach requires often copying sub-arrays for use by APIs. The second approach means trusting the called method to not move outside the bounds of the sub-array.

Slices would provide a third approach; one that enforces the boundaries to the proper subset of an array, without requiring copies of the original data structure.

Feature Overview

The feature requires two pieces. First, there is a Slice<T> class that would support a slice of an array. There is also a related ReadOnlySlice<T> class that would support a readony slice of an array.

In addition, the C# Language would support features designed to create those slice objects from another array:

 

Person[] slice = people.GetSlice(3, 9);

byte[] bytes = GetFromNative();

There is quite a bit of discussion about the final implementation, including whether or not CLR support would be needed. Some of that discussion moved to the C# Language Design Notes here.

This post is somewhat light on details, because the syntax, and the full feature implementation is still under discussion. You can look at one implementation here: https://github.com/dotnet/corefxlab/tree/master/src/System.Slices

I will write new posts as the feature list and the syntax and impementation becomre more concrete.

Created: 1/28/2016 5:12:15 PM

This is the first of a series of blog posts where I discuss the upcoming feature proposals for C# 7. At the time that I am writing these posts, these are all proposals. They may change form, or may not be delivered with C# 7, or ever. Each post will include links to the proposal issue on GitHub so that you can follow along with the ongoing disussions on the features.

This is an interesting time for C#. The next version of the language is being designed in the open, with comments and discussion from the team members and the community as the team determines what features should be added to the language.

Ref Returns and Local are described in the Roslyn Repository on GitHub, in Issue #118.

What is Ref Return and Ref Locals?

This proposal adds C# support for returning values from methods by reference. In addition, local variables could be declared as ‘ref’ variables. A method could return a reference to an internal data structure. Instead of returning a copy, the return value would be a reference to the internal storage:

 

ref PhysicalObject GetAsteroid(Position p)
{
    int index = p.GetIndex();
    return ref Objects[index];
}

ref var a = ref GetAsteroid(p);
a.Explode();

 

Once the language defines ref return values, it’s a natural extension to also have ref local variables, to refer to heap allocated storage by reference:

    ref PhysicalObject obj = ref Objects[index];

This enables scenarios where you want pass references to internal structures without resorting to unsafe code (pointers to pinned memory). Those mechanisms are both unsafe and inefficient.

Why is it useful?

Returning values by reference can improve performance in cases where the alternative is either pinned memory, or copying resources. This features enables developers to continue to use verifiably safe code, while avoiding unnecessary copies.

It may not be a feature you use on a daily basis, but for algorithms that require large amounts of memory for different structures, this feature can have a significant positive impact on the performance of your application.

Verifying Object Lifetimes

One of the interesting design challenges around this feature is to ensure that the reference being returned is reachable after being returned. The compiler will ensure that the object returned by a ref return continues to be reachable after he method has exited. If the object would be unreachable, and subject to garbage collection, that will cause a compiler error.

Effectively, this means you would not be able to return a ref to a local variable, or a parameter that was not a ref parameter. There is a lengthy discussion in the comments GitHub that go into quite a bit of detail on how the compiler can reason about the lifetime of any object that would be returned by reference.

Created: 9/1/2015 11:35:22 AM

Yesterday, this blog post documented how a bug accidentally cost Carlo $6500 in a few hours because his AWS keys were compromised. Please read the whole post.

Synopsis: He was using the GitHub Extension for Visual Studio. He published a new repository that he *thought* would be private, but due to a bug, he created a public repository. That repository contained his Amazon Access Key. Scanners found it and created lots of resources.

First, some good news and a task for all of you:  This issue was reported on the GitHub project for that extension (https://github.com/github/VisualStudio/issues/62)  That page shows the quick response by the team involved for that extension, and a fix has already been implemented, and deployed to the Visual Studio Extension Gallery: https://visualstudiogallery.msdn.microsoft.com/75be44fb-0794-4391-8865-c3279527e97d

 

A task for all of you:  If you are using the GitHub for Visual Studio Extension, update it now. You’ll get a notification from Visual Studio that there is an update available. Install it. Now. (This extension does not auto-update, so you will need to perform this task.)

A bit of background on how this happened so fast

One fact I learned yesterday was that there are organizations that constantly monitor Github commits to see if any contain credentials to AWS or other cloud providers. That is the price of success. A tremendous amount of code is added or modified on github.com every day. It’s an attractive target.

As soon as a public repository gets created, or updated, that information will be captured. Be careful.

What habits could have prevented this?

Clearly, hindsight is 20:20, I’m writing this to share a few tips that might help you. I’m also interested in hearing what techniques you use to manage these kinds of secrets. My experience is with the open source projects for Humanitarian Toolbox. Those projects are deployed on Azure, and are all Open Source projects on GitHub. That means we must manage to keep secrets out of the repository. Here are the recommendations we use:

No Credentials in *.config

The default web.config files that are in the open source repositories are set for a localDB on the developer’s drive. There are no keys for our deployed resources.

AzurePortal_thumb

Instead, we use the Azure Portal to configure any and all secrets for our sites.  (click on the image for a larger view..) This does mean that we, as an organization, must protect our Azure portal settings. And, folks like me must make sure to obscure the secrets in images like this.

Deploy Directly from Source Control.

We deploy our applications directly from GitHub source control. The staging site deploys from the master branch, whenever we merge pull requests. If you are one of our contributors, you can see your changes live as soon as we merge the pull request.

For release builds, we use the Release branch in GitHub. One of the commit team must merge changes from master to the release branch for those to be live. We do that after our partners have had the chance to validate the changes on Master.

Consider Visual Studio Online

For my projects that are not planned to be open source, but will remain private, I use Visual Studio Onilne. (https://www.visualstudio.com/en-us/products/what-is-visual-studio-online-vs.aspx) It supports Git, and has some pull request support for teams. (I’d like a model where I could use Pull Requests even with my shared repositories, but I haven’t found that feature yet.)

VSO is designed around a team model, not an Open Source model. It’s less likely (well, almost impossible) that I accidentally make a public repository on VSO.

What next?

The key takeaway for me is that the era of Open Source requires all of us to learn some new skills and take care of new potential risks. These security breaches can and do happen to experienced people. Teamwork, care, and multiple levels of defense are needed.

Most importantly, professionals did their job: I’m truly impressed with the speed and professionalism of the response to this issue by team members from both Microsoft and GitHub.

Created: 11/24/2014 3:55:52 PM

The preview of Visual Studio 2015 is public, and the Roslyn APIs are stabilizing.  A group of language MVPs have begun working on a series of code analyzers and code fix tools that will help you write better code.  To facilitate this work, we’ve created an organization on Github, .NET Analyzers, that contain the repositories for the analyzers, code fix projects, and refactorings that we’ve created.

We’ve got seven code repositories in place already. There’s also a Proposals repository that has potential ideas for new code fix or analyzers projects.

The code repositories are varied, both in completeness and in scope. Some are for a single refactoring. Others will grow to a full suite of analyzers.  Some are nearing completion. Others haven’t had any code uploaded yet.

Mark Michaelis and I started this group while at the MVP summit. Since then, I’ve been very impressed with the response. In particular, Sam Harwell, Tugberk Ugurlu,  and Adam Speight have been working very diligently on several of the analyzers and code fix algorithms. There have been many others that have contributed and offered suggestions and help. I’m singling those three out because they have really spent significant time working on this.

We’d love to see other people get involved as well.

If you want to join us, check out the organization on GitHub. Look at the issues in the code repositories and contribute. Make your modifications, and submit a pull request. We are generally following Github flow, but that’s not a firm requirement. If you are new to Github, just ask one of us via Github messaging, and we’ll help walk you through it.

If you have an idea for something you want to implement, please add it as an issue to the Proposals repository. Mention that you want to work on it. We’ll help you define the requirements, and suggest if the new feature should go in an existing repository, or if it should be a new repository.

If you have ideas, but aren’t sure how to implement them, add them as issues to the Proposals repository. We’ll discuss it and prioritize it with you. If you want to implement it, but need help, just ask.

Special note for the VB.NET community: We’re interested in having support for important VB idioms as well. However, we need the VB community to help us. Most of us are more involved in the C# community than the VB.NET community.

We want to see this grow and become an indispensable set of extensions for .NET developers. Please join us. Let’s see what we can build together.

Created: 11/13/2014 12:50:17 AM

Today, November 12th, Microsoft made some major announcements about the .NET Ecosystem.

I’m going to concentrate on four points: Open Source, Cross Platform, Deployment, and Community Involvement.

Open Source

The core .NET Framework is now Open Source. No longer will you choose “Microsoft or Open Source?” The entire Microsoft .NET Development stack is moving to Open Source. From the core CLR on up. I remember NDA meetings with a very small set of people many years ago starting to push for a moment like this. It has taken a long time, and this movement has slowly garnered more and more people. Before I go any further, thanks to those folks that worked at Microsoft back then and started this ball rolling.

It’s really Open Source: It’s being distributed under the MIT license, and the team has committed to accepting community contributions. You can grab the source, build it, use it, fork it, and do anything else allowed by the standard MIT license. It’s a great day.

Cross Platform

By “do anything that’s allowed by the standard MIT License”, that includes running .NET on platforms other than Windows. That explicitly includes Linux and Mac platforms. In fact, to support these platforms, Microsoft will be releasing official distributions for both Linux and Windows.  

Let me repeat this: Microsoft will be releasing official distributions of .NET for Linux and Mac, in addition to the traditional Windows platforms.

Deployment

As part of the work to create Open Source and Cross-Platform .NET distributions, Microsoft is also breaking the link between the Windows OS updates and .NET. You no longer need to run your .NET applications using the framework in the Global Assembly Cache (GAC), on a machine wide basis.

Instead, you can deliver the .NET core framework (or just the parts you use) with your application. This means you can run .NET based server applications that use different versions of the framework on the same machine.

There’s a huge advantage for this: In a data center, you can upgrade individual application on individual machines without forcing the DevOps team to upgrade machines in the Data Center. You can even run different applications with different versions of the framework on the same machine.

It follows that you can run new versions of the framework in Azure before the team has rolled out new images with updated bits.

Just deploy your needed version of the framework locally with your application.

Community Involvement

Finally, the big finish: Visual Studio 2015 Community Edition. This edition is similar to the previous Professional edition, and it’s now free for independent developers. (Enterprises must still license Visual Studio for commercial use). This is a much more useful edition than the previous Express editions: you can build web sites, mobile applications, and windows applications all with the same edition. You’re not limited to just one style of application. Also, this new community edition supports plug ins and extensions.

I’m excited about this as a great product for students I teach. Too often students were using the trial (time limited) trial, or the express versions. Those just weren’t great tools to keep learning and keep growing. This new version is.

Now what?

This is a huge set of announcements, and it’s only the beginning. Microsoft just made a set of huge bets in cross-platform Open Source. I know I have some biases, but .NET is the most open, most viable cross-platform toolset available today.

When someone asks if you have a cross-platform, mobile, or Open Source strategy, you can answer: “Yes, it’s all in .NET.”

Exciting times.

Created: 4/9/2014 3:00:03 PM

By now, I’m sure you’ve heard the big news: There’s a new public preview of the Roslyn C# and VB.NET Compilers.  And, even bigger news: The Roslyn Compilers are Open Source.

This is an important milestone for these projects. The C# compiler is now written in C#; the VB.NET compiler is written in VB.  (See below). They also now do have full support for all the features in C# version 5 (and then some).  The rest of this blog post walks through some of the important ideas in the Roslyn ecosystem.

TL;DR version: Go to https://roslyn.codeplex.com/ and get started.

Using the Roslyn Compilers (and IDE updates)

You can get the Roslyn compiler from a NuGet package, or by signing up and downloading the End User Preview. You will want to do this before you explore the Roslyn Source. That’s because the Roslyn compiler source makes use of new language features only available in the Roslyn compiler. That means you can’t build the Roslyn source with the compilers that ship with VS 2013.

Once you have the end user preview. You’ll find quite a few new features in the IDE. Most of these are covered in the //build talks, available online. My one recommendation: train yourself to look for the light bulb in the left margin of source. There’s lots of new information there.

If you want to dive deeper into Roslyn, get the SDK Preview. That provides templates and examples for writing extensions based on the Roslyn Language services API. One design goal for the Roslyn compilers is to provide the same language services to the compiler, the IDE, and 3rd Party add-ins. Those APIs let you view program source at its syntactic and semantic tree level, and manipulate those trees. If you are interested in analysis tools, or Refactorings, that’s where you start.

Learning about the source (and more about the languages)

The Roslyn source is hosted on CodePlex. It is a git – based repository. You can clone it using your favorite git tools. In my case, I cloned it from the command line using posh-git. Once cloned, I added it to my list of local repositories in GitHub for Windows. That’s worked well for me.

Once you’ve gotten the source, follow the instructions here to build it. Congratulations, you’ve built a compiler.

Feel like learning more, get involved in the discussions.  First, read the language design notes for C# (or VB if that’s your favorite) and then check the Language Feature Implementation status. The language design notes will give you the history and background for the current language features. It’s very illuminating. Personally, I’m thrilled that the teams have the confidence to share this level of detail with the public. You’ll learn an incredible amount by reading about the alternative implementations and the thoughts and discussions around each decision.

For many of the proposed new features, discussions are ongoing. In particular, the discussion on the null propagating operator is rather lively. If you have an opinion, join in.

Do you have ideas?

Well, we all do. And here’s where life gets very tricky. The Roslyn compilers are licensed under the Apache v2 license.  You, me, and everyone else on the planet can fork them and add our own new language features. On the one hand, it’s a fantastic move to release the compilers as open source. On the other hand, it could lead to chaos.

On the plus side, academics (and their students) can teach and learn compiler construction by working with a world class commercial compiler for two of the most popular programming languages on the planet.

Also on the plus side, community members can prototype ideas for language features and submit them for review. That will do a lot to keep the C# and VB communities vibrant and involved.

And, by looking at the compiler source, you’ll better understand how to build extensions and addins that leverage the Roslyn APIs.

But there is a dark side.

Part of the success of VB.NET and C# is that they are consistent languages. The Microsoft and Xamarin compilers both track the standard very closely, and have very few divergences. That’s a positive for the language.

I worry that it’s possible to have several dialects of C# in the future that all behave differently. If that happens, this global experiment to open source Roslyn will have been a mistake.

Rather than worry, here’s my recommendation to anyone looking at creating new language features: Get involved in the feature discussions first. Talk with the team. They are already active in the discussion forums. Get buy-in for the new feature.

If that happens, this is a marvelous new era. If community members create and discuss great new features, and those features become part of the standard C# Language, the global experiment to open source Roslyn will come to be seen as one of the defining positive moments in computing history. That’s what I want to see happen.

Current Projects

I create content for .NET Core. My work appears in the .NET Core documentation site. I'm primarily responsible for the section that will help you learn C#.

All of these projects are Open Source (using the Creative Commons license for content, and the MIT license for code). If you would like to contribute, visit our GitHub Repository. Or, if you have questions, comments, or ideas for improvement, please create an issue for us.

I'm also the president of Humanitarian Toolbox. We build Open Source software that supports Humanitarian Disaster Relief efforts. We'd appreciate any help you can give to our projects. Look at our GitHub home page to see a list of our current projects. See what interests you, and dive in.

Or, if you have a group of volunteers, talk to us about hosting a codeathon event.