Title</th>	Author</th>	Second Author</th></tr></thead>
NEED</td>	Abbott, Elena</td>	</td></tr>
Pack of Her Own</td>	Abbott, Elena</td>	</td></tr>
The Revenge of Captain Vessia</td>	Allen, Leslie</td>	</td></tr>
Temptation at Randy's</td>	Alter, Emily</td>	</td></tr>
A Sea of Silence</td>	Atkinson, Kay</td>	</td></tr>
Nibbled Bookmark</td>	Avant, Melody</td>	</td></tr>
Galaxy: The Prettiest Star</td>	Axelrod, Jadzia</td>	</td></tr>
Confluence: A Person-Shaped Story</td>	Bakkalian, Nyri</td>	</td></tr>
Grey Dawn**</td>	Bakkalian, Nyri</td>	</td></tr>
Any Other Name**</td>	Bambust, Ela</td>	</td></tr>
Horns in the Library</td>	Bambust, Ela</td>	</td></tr>
Nexus Alpha</td>	Bambust, Ela</td>	</td></tr>
Dulhaniyaa</td>	Bhatt, Talia</td>	</td></tr>
Nevada**</td>	Binnie, Imogen</td>	</td></tr>
Rhapsody</td>	Bragg, Molly J</td>	</td></tr>
Transistor</td>	Bragg, Molly J</td>	</td></tr>
Magica Riot</td>	Buchanan, Kara</td>	</td></tr>
Princess Fox and Iron Wolf</td>	Chaos, Liliana</td>	</td></tr>
The Price on Her Head</td>	Clay, Suzanne</td>	</td></tr>
Ghost Flower**</td>	Conwell, Jessica</td>	</td></tr>
Dreadnought</td>	Daniels, April</td>	</td></tr>
Sovereign</td>	Daniels, April</td>	</td></tr>
She Who Earned Her Wings</td>	Denarr, Vaela</td>	Iannandrea, Micah</td></tr>
Seasons of Fools</td>	Doerner, Morrigan</td>	</td></tr>
Catnip</td>	Durav, Vyria</td>	</td></tr>
Hatchling</td>	Durav, Vyria</td>	</td></tr>
Wyrmheart</td>	Durav, Vyria</td>	</td></tr>
Sundered Moon</td>	Fae'rynn</td>	</td></tr>
Stray Dogs (1st book in series)</td>	Faust, Freya</td>	</td></tr>
The World Within</td>	Finn, Dani</td>	</td></tr>
Scales of Seduction</td>	Gray, Rien</td>	</td></tr>
Seure the Tempered</td>	Gray, Rien</td>	</td></tr>
Valerin the Fair</td>	Gray, Rien</td>	</td></tr>
(kimmy)***</td>	Greaves, Alyson</td>	</td></tr>
Cora Christensen</td>	Howsden, Tanner</td>	</td></tr>
JUNKER SEVEN</td>	Kelley, Olive J.</td>	</td></tr>
Ransom to Love</td>	Keto, Chloe</td>	</td></tr>
Huntress of the Night: In Cold Embrace**</td>	Leonard, Gwen</td>	</td></tr>
Make Room for Love</td>	Liao, Darcy</td>	</td></tr>
Not Just Best Friends</td>	Liao, Darcy</td>	</td></tr>
Beauty's Beast</td>	Lynn, S.T.</td>	</td></tr>
Cinder Ella**</td>	Lynn, S.T.</td>	Veldura, Tami</td></tr>
Mer Made</td>	Lynn, S.T.</td>	</td></tr>
Ghostkin</td>	Mellor, Ellen</td>	</td></tr>
You Kiss Like a Rocket</td>	MsJuniper</td>	</td></tr>
Revolution of Wild Magic</td>	Quill, Katie</td>	</td></tr>
The Red Satin Collection</td>	Renarde, Giselle</td>	</td></tr>
Brainwyrms**</td>	Rumfitt, Allison</td>	</td></tr>
Imposters</td>	Sandwich, Cassie</td>	</td></tr>
Fake It</td>	Seabrooke, Lily</td>	</td></tr>
Not Like Before</td>	Seabrooke, Lily</td>	Ramsden, Jacqueline</td></tr>
Us</td>	Soler, Sara</td>	</td></tr>
To Own the Libs</td>	Storm, Zoe</td>	</td></tr>
Wayward Fairy</td>	Sutton, Mackenzie</td>	</td></tr>
Blood of Gorgon</td>	Tang, Selene</td>	</td></tr>
Shadow of Gorgon</td>	Tang, Selene</td>	</td></tr>
As Happy as Pirates or Burglars</td>	Topaz, Jemma</td>	</td></tr>
Captains of a Tyrant Fate</td>	Topaz, Jemma</td>	</td></tr>
Our Monsters</td>	Topaz, Jemma</td>	</td></tr>
The Brute of Greengrave</td>	Topaz, Jemma</td>	</td></tr>
Pride, Pain, and Petticoats volume 1</td>	Trusity, Abby</td>	</td></tr>
Pride, Pain, and Petticoats volume 2</td>	Trusity, Abby</td>	</td></tr>
Against Fearful Lies (The Amelia Temple Series Book 2)</td>	Valentine, Vivian</td>	</td></tr>
Beneath Strange Lights (The Amelia Temple Series Book 1)</td>	Valentine, Vivian</td>	</td></tr>
Best Friends' Promise</td>	Waldrom, Issy</td>	Sutton, Mackenzie</td></tr>
Dawn Weaver</td>	Waldrom, Issy</td>	</td></tr>
Esper and the Witch</td>	Waldrom, Issy</td>	</td></tr>
Heart's Claim</td>	Waldrom, Issy</td>	</td></tr>
Soul Flames Volume 1</td>	Waldrom, Issy</td>	</td></tr>
Soul Flames Volume 2</td>	Waldrom, Issy</td>	</td></tr>
Soul Flames Volume 3</td>	Waldrom, Issy</td>	</td></tr>
Soul Flames Volume 4</td>	Waldrom, Issy</td>	</td></tr>
The Grace of Sorcerers</td>	Ying, Maria</td>	</td></tr>
The Gunrunner and her Hound</td>	Ying, Maria</td>	</td></tr>
The Hades Calculus</td>	Ying, Maria</td>	</td></tr>
The Might of Monsters</td>	Ying, Maria</td>	</td></tr>
The Ruin of Beasts</td>	Ying, Maria</td>	</td></tr>
The Serpent of the House of Hua</td>	Ying, Maria</td>	</td></tr>
The Spy and her Serpent</td>	Ying, Maria</td>	</td></tr>
The Vela</td>	Yoon, Ha Lee</td>	Chambers, Becky; Huang, S. L.; Solomon, Rivers</td></tr> </tbody></table> () - Collection of stories</li> () - Needs to be double-checked</li> (*) - Includes rape or other sexual assault by a man</li> </ul> Sharing Trans Femme Lit 2024-11-04T00:00:00+00:00 Co-authored with Aurora Foo</a></em></p>* We just finished reading "How to Read a Trans Fem Writer"</a>. The article is written generally for readers from outside the queer (and specifically the trans femme) community who might not be used to engaging in this type of art, and has lots of good tips for how to engage with our art. We'd encourage folks to read the essay and reflect. This is especially important because of the volume of negative comments many trans femme authors get on their work. Helping readers have a keener eye and not fall into patterns and expectations that are hurtful to those authors is important.</p> As someone inside the trans femme community, it also brought up additional questions about the art that we create and our responsibilities we have to each other.</p> The first that needs to be said and repeated: we should make the art we want to make. It's vitally important. Having articles like this one help people navigate the art that we're creating.</p> Having said that, there's a conversation we should have amongst each other about how we as a community can better share our art.</p> Trans femmes, as a community, experience a profound amount of collective trauma. As individuals also, ourselves included, have more than our share of trauma. Art can stir up that trauma because it reaches inside of us. Art that does so should let us know the topics it will cover so that we can be better informed as readers.</p> Recently, Soph finished reading (kimmy)</a>. It's brilliant. It's clever. It's everything folks lauding it say it is. And it's also horribly triggering and in its current version has an incomplete content warning list. Aurora saw firsthand the effect it had, following along on Soph's reading journey. A book with the themes and content that (kimmy) has needs to throughly warn its readers, to ensure people aren't re-traumatised by its content and can make an informed decision about choosing to read it. That was not the case in our experience, and Soph took emotional damage as a result of reading the book.</p> Additional (kimmy) warnings</summary> Repeated rape, forced incest, nonconsensual mind control, and ego/identity death. </details> We then went on social media to see people breathlessly praising this work without warning anyone about its contents. It's undoubtedly brilliant, but it's also painful, and discusses at length things that could be potentially triggering for a lot of trans femmes out there. Going on about how everyone should read it, without stopping to warn folks reading your post to take care, feels irresponsible.</p> To reiterate: we firmly believe we should make the art we want to make. Make it messy if you want to make it messy. Make it challenging. Make it dark. Give it all the sharp edges you feel you need to in making the art that feels honest and true.</p> Once you do, let folks know what those sharp edges are. As readers and promoters, it's the responsible thing to let other folks know so they can make better informed choices. It is more than responsible, though, it's the kind thing to do. As a community, we need to take care of each other, and when we share these often brilliant works of art that explore traumatic content, we need to ensure that people go into them with informed consent.</p> We need to be better at not uncritically</strong> promoting work that has the potential to re-traumatise our peers. That's not to say we shouldn't share our praise and love for our art - we absolutely should! We just need to be better at balancing our (perfectly valid) enthusiasm for works with making sure the people we broadcast our thoughts to don't develop an incorrect picture, only to be blindsided by potentially triggering content.</p> We need to be better, because our work deserves it. As a community, the creation of trans femme art is vitally important. It can be a cathartic experience to create, and it can be a carthartic experience to consume. However, we must remember the context in which we exist as a community, and ensure we discuss our art with care and consideration for our fellow sisters.</p> Following new paths ahead 2024-06-04T00:00:00+00:00 Hi all! Wow, the last month has been quite the rollercoaster. In that short time, I announced June publicly and, on a sad note, also had its funding pulled.</p> The end result was a feeling of whiplash that I needed to recover from. After taking a break, here are my plans going forward.</p> Nushell</h1> Nushell has been a pride and joy for many years, though for many months I've very much been in the background while others led the project. After watching my friend Andreas step down from Serenity</a> (EDIT: please note, Andreas and I are no longer friends), I knew it was time for me to do the same for Nushell.</p> Effective today, I'm no longer part of the Nushell administration team and have handed those responsibilities off to other members of the team.</p> June</h1> After announcing June and then losing funding for it, I took a step back from it.</p> There are some hard truths about programming languages and open source. It's a difficult path to travel, and it takes a long time. A good language can take 6+ years to mature to the point it's ready for users to trust it. Open source itself requires that you're ready to weather the burden of maintenance.</p> When I look at this, in light of having to self-fund the project, I know that I don't have the focus nor the energy to pull it off. That's not a knock against June. More, it's an admission about understanding how I work - especially in the light of being autistic and having to listen to where my drive is. The thing is, you can't fake drive. If you try to, you can easily burnout.</p> With this in mind, I'm archiving June</a> and setting it to not-maintained. I'm also doing likewise for another project called Truffle</a> that I released along the same time.</p> Having done this, I'll be able to let my creativity come back and go where it wants to go. This will be healthier for me and also will let me continue to make art that makes me, and hopefully others, happy.</p> For those of you who believed in the June message - thank you! I would love to see motivated people take it and fork it and carry it on. I'd also love to see other languages spring up which evolved the ideas further.</p> The search for easier safe systems programming 2024-05-08T00:00:00+00:00 I've been involved in the Rust project in some form or another since 2016, and it's a language I'm very comfortable using. Many Rust programmers could say the same</a>. But, if we take a step back and are honest with ourselves, we'd admit that the road to getting to that level of comfort was difficult.</p> I taught Rust professionally for two years. Watching the faces of people trying to learn Rust for the first time reminded me just how hard this language is to learn.</p> After two years of that, I wanted to answer a question I wasn't entirely sure had an answer: Is it possible to make an easy-to-use, easy-to-learn, and easy-to-teach safe systems language?</em> Could I put my career working on programming languages (TypeScript, Rust, Nushell, etc) to use and find a solution?</p> Enter June</h1> For the last year and a half, I and my recently-added collaborator Jane Losare-Lusby have been working in secret on a safe systems language that could be learned about as quickly as one can learn Go. I think we might have something worth exploring.</p> Changing how we think of memory</h2> In Rust, we think of each piece of memory as having its own lifetime. Each of these lifetimes must be tracked, sometimes leading to rather complex code, complex error messages, and/or complex mental models of what is happening. The complexity of course comes with the benefit of being highly precise about each and every piece of memory and its reclamation.</p> Using a Rust example:</p> `struct Node { data1: &Data, data2: &Data, data3: &Data, } </code></pre> Rust developers will spot right away that this is an incomplete example. We need two more things: Lifetime Parameters and Lifetime Annotations. Adding those, we get:</p>` `struct Node<'a, 'b, 'c> { data1: &'a Data, data2: &'b Data, data3: &'c Data, } </code></pre> The concept count for this example ends up being pretty substantial. Counting them off, we get:</p>` Lifetimes</li> Lifetime annotations</li> Lifetime parameters</li> Ownership and borrowing</li> Generics</li> </ul> When I showed examples like this to my class when I taught Rust, I had to walk them through each of those concepts first before I could show the full example.</p> The question then is: can we make this easier?</p> What if memory was grouped?</h2> What if instead of having to track every piece of memory's lifetime separately, we let groups of related allocations share a lifetime?</p> Effectively, this would mean that a data structure, like a linked list, would have a pointer pointing to the head which has a lifetime, and then every node in the list you can reach from that head has the same lifetime.</p> There are some benefits to this approach, as well as a few drawbacks. Let's take a look at the benefits first.</p> Benefits of grouped allocations</h2> Exploring grouped allocations, we noticed some immediate benefits. The first is that we could treat all user-defined values as pointers, and these pointers could also represent their own lifetimes (without needing lifetime parameters). This makes the code feel quite a bit lighter:</p> `struct Node { data: i64 next: Node? } let n = new Node(data: 123, next: none) </code></pre> Since all user data is pointers, we can use the name of the type to mean "pointer to this structured data".</p> The next thing we noticed is that both lifetimes and inference for lifetimes becomes significantly simpler.</p> Let's take a variation of the example above:</p>` struct Node { data: i64 next: Node? } fun do_this() { let n = new Node(data: 123, next: none) } </code></pre> We can infer that the allocation that creates new Node(...)</code> has a lifetime and what it is. Because this allocation never "escapes" the function - that is, it never leaves the function in any way - then we can call its lifetime "Local".</p> As we'll find out, each of the lifetime possibities is a readable name that we can show the user in error messages. It also makes things significantly easier to teach.</p> Let's look at another example to see a different lifetime.</p> struct Stats { age: i64 } struct Employee { name: c_string, stats: Stats, } fun set_stats(mut employee: Employee) { employee.stats = new Stats(age: 33) } fun main() { mut employee = new Employee(name: c"Soph", stats: new Stats(age: 22)) set_stats(employee) println(employee.stats.age) } </code></pre> A bit of a longer example this time, but let's focus on this function:</p> fun set_stats(mut employee: Employee) { employee.stats = new Stats(age: 33) } </code></pre> What's the lifetime of this new Stats(..)</code> allocation? In this example, we do see the new pointer escape the function via a parameter. We can also give this a readable lifetime: Param(employee)</code></p> In all, we have three lifetimes an allocation can have:</p> Local</li> Param(xxxx)</li> Return</li> </ul> Any data structure you want</h2> Another big advantage of grouping our allocations is that we no longer have to worry about a drop order. This means we can think of the whole thing as dropping all at once. For large structures, that can be a speed-up over languages with a required drop order.</p> Additionally, we get another major benefit. We can now create arbitrary data structures.</p> struct Node { data: i64 next: Node? } mut node3 = new Node(data: 3, next: none) let node2 = new Node(data: 2, next: node3) let node1 = new Node(data: 1, next: node2) node3.next = node1 </code></pre> And just like that, we've made a circular linked list. Creating a similar example in Rust is certainly more of a challenge.</p> But, something fishy is going on here.</p> To make the above work, we're using shared, mutable pointers. This is explicitly forbidden in Rust. Why is it okay here?</p> The dangers of shared, mutable pointers</h2> Rust disallows holding two mutable references to the same memory location and for good reason. Well, multiple reasons actually.</p> First, having two copies of a mutable pointer where two separate threads each hold a copy means we have the possibility for a race condition. This can leave us with incoherent data that's difficult to debug.</p> Second, even if these two multiple pointers are limited to the same thread, we get what we might call "spooky action at a distance". The modification of one pointer is then visible to the holder of the other pointer, which might be far away from the source of the mutation.</p> For us to reasonably use shared, mutable pointers, we need to tame both of these. The first issue, the race condition, is easy enough: we prevent sending shared, mutable pointers between threads. This limits them to a single thread.</p> The second issue is decidedly harder. There have been many attempts at ways of handling this through rules enforced by the type system.</p> In June, we're trying something a bit different. We'll let developers use shared, mutable pointers, but then offer a "carrot" to opt-in to restrictions around using them. The carrot ends up pulling from a classic technique of software engineering: encapsulation.</p> The full power of encapsulation</h2> In traditional encapsulation, programmers make a kind of "best effort" to hide implementation details from the world around them. Keeping private state private grants the benefits of better code reuse, ease of updating implementation details, and more.</p> But as often is the case, if that kind of rule isn't enforced, over time APIs get designed where internal implementation details leak out.</p> Something very interesting happens if we don't allow this to happen. If an encapsulation can be checked by the compiler, and the compiler enforces that no private details leak, we have what you might call "full encapsulation".</p> These kinds of encapsulations wouldn't allow any aliasing of pointers into them. They'd have their internal pointers fully isolated from the rest of the program.</p> Once we have this, some new capabilities start opening up:</p> We can "fence off" our shared, mutable pointers, making it possible to create single-owner encapsulations that can be sent safely between threads.</li> We can lean people in the direction of cleaner API design, as now we have a way to truly keep private implementation details private.</li> We can handle some of the drawbacks of grouped allocations.</li> </ul> What kind of drawbacks, you might ask? It's high time we talked about them.</p> Drawbacks of grouped allocations</h2> If we go back to our earlier example and look carefully, we'll notice something:</p> fun set_stats(mut employee: Employee) { employee.stats = new Stats(age: 33) } fun main() { mut employee = new Employee(name: c"Soph", stats: new Stats(age: 22)) set_stats(employee) println(employee.stats.age) } </code></pre> The question is: what happened to the new Stats(age: 22)</code> allocation?</p> Remembering that June is a systems language, we can't say "the garbage collector handled it" because we have no garbage collector. Nor can we say "the refcount hit zero, so we reclaimed it" as we don't use refcounting. As a systems language, we can't allow hidden or difficult-to-predict overhead to happen.</p> It's not actually leaked either, as even the memory it occupies will be reclaimed once the entire group is reclaimed. For all intents and purposes, though, it's lost to the user until the group is no longer live. It's a kind of "memory bloat" that happens if we group our allocations.</p> To handle this, we'll need some way of recycling that memory. I say "recycling" specifically because in June we can't free the memory, as the group is treated together as a single entity where all the allocations in the group are freed at once. If we instead recycle the memory, we can reuse that same memory while the group is live.</p> Techniques to do this have been around for decades, and often people use "free lists" to keep a list of nodes that have been recycled, so they can be reused when the next allocation happens.</p> The problem with free lists is that they aren't safe. If you're not careful, you'll create a security vulnerability and/or an incredibly hard bug to find.</p> Instead, we need to build in a safe way of recycling memory into the language.</p> Safe memory recycling</h2> Using the idea of full encapsulation from earlier, we can create "fenced in" sets of pointers that we know aren't shared with the rest of the world. Once we have them, it's possible to track the pointers inside. These pointers can get a "copy count", so we know how many copies are live at any point in time (not dissimilar from a refcount, though this has no automatic reclamation).</p> Once we have a copy count for each internal pointer, we can give developers a built-in recycle</code> command.</p> let x = new Foo() recycle x </code></pre> Recycling would start at the given pointer and would check the pointers reachable from it. Each pointer it finds that it can recycle would go into the safe free list.</p> You might wonder "why not do this automatically?". There are a couple reasons:</p> The operation is linear time based on your transitively-reachable pointers. This means you may incur a noticeable overhead when recycling</li> Because of the first point, it's important to make places where this occurs visible</li> </ul> If this sounds like a kind of manual garbage collection, you're right. My collaborator Jane calls this "semi-automatic" memory reclamation. You ask once, and when you ask you get a kind of highly focused mark and sweep for that single pointer and the pointers reachable from it.</p> Note: this feature is not yet in the reference compiler. We're hoping to implement it in the coming weeks.</em></p> More work ahead</h2> We have a way of simplifying lifetimes, making for readable code that people from various languages should be able to understand and use. We can also give clear, easy-to-understand lifetime errors when they arise.</p> Having safe memory recycling gives us a way to keep groups and still offer things like delete</code> in a linked list abstraction. It's convenient but not so automatic that we lose the visibility into the costs of memory management.</p> That said, there are still some challenge ahead that will need to be solved in the language design and tooling. For example, how do you know when the program is bloating memory? We'll need some way of doing a memory trace when the program is running to detect this and warn the developer.</p> I see this in a way as a more incremental/prototype-friendly way of development. June is always memory safe, but the first version of a program may not be as efficient as it could be in terms of memory usage. That's a process we often go through as developers. First, we "make it work" before we "make it good".</p> In June, we keep it lightweight as we keep your programs memory safe, and then we provide tools and support for incrementally improving code.</p> Future possibilities</h2> Relationship to Rust</h3> June has a real opportunity to be a good complement to Rust. Rust's focus on embedded and system's development is a core strength. June, on the other hand, has a lean towards application development with a system's approach. This lets both co-exist and offer safe systems programming to a larger audience.</p> An even better end state requires Rust to have a stable ABI. Once it does, June will be able to call into Rust crates to get the benefits of Rust's substantial crate ecosystem. We're looking forward to collaborating on this in the future.</p> Going beyond OOP</h3> OOP has for decades been the way many applications are written, but it's not without its flaws. Many OOP languages allow programmers to freely break good rules of thumb, like the Liskov substitution principle, or to create a mess of interwoven code between parent and child classes that's difficult to maintain.</p> We're currently investigating other ways of making code reuse easier, more modular, and more composible. We're not quite ready to talk about this, though we hope to soon.</p> Exploring related work</h3> Over the years, there have been a number of memory management techniques tried</a>, including many that lie outside of the ones commonly found in languages today. We'd like to explore these more deeply to see which, if any, may help June.</p> Acknowledgements</h2> We had the help of dozens of experts in various fields as we brainstormed the initial design for June, and for their contributions, we're thankful. We'd especially like to thank the collaborators who went above and beyond with their time across multiple brainstorming sessions to help June grow to where it is.</p> Andreas Kling</li> Doug Gregor</li> Jason Turner</li> Mads Torgersen</li> Mae Milano</li> Steve Francia</li> </ul> Also, special thanks to our private beta testers for testing out June and giving us feedback.</p> Checking it out</h2> Documentation on the June language and the June reference compiler are now available via the June repo</a>.</p> Please note: the reference compiler is pre-alpha quality.</p> Worst part of being trans 2024-03-25T00:00:00+00:00 I'm sitting here, reading a book, thinking about the road ahead for me as a trans woman. It feels like I can already tell what's likely the worst part of being trans, at least so far.</p> You might think it's the bullying. I definitely experienced some of that.</p> Or maybe it's friction with the family. I definitely experienced that, too.</p> For me, though, the worst part is time.</p> Estrogen does what it does. And then you wait.</p> I got hair replacement surgery. It takes months to even start growing. And then you wait.</p> Start a series of electrolysis, going regularly. And then you wait (and keep going until it's done).</p> Schedule surgeries. And then you wait.</p> Recover from those surgeries. And recovery takes time. So you wait.</p> The trick is to learn how to be good at waiting. I'll admit, I kinda suck at it. I've been checking my head everyday to see if I can see changes. I don't recommend this.</p> For a while there I was measuring my chest almost every day. I don't recommend this, either.</p> I guess the lesson I'm trying to learn here is to ignore it. I'll still very likely be on this path tomorrow. And the next week. And the next month. It's perfectly okay not to watch the grass grow and just be myself and see how things are coming along later. Just find a rhythm, don't stress.</p> Still, that desire to have the magic button you can press and then wake up fully-transitioned is so strong. Or a button that if you press it you can jump ahead 6 months each time you press it.</p> We get the days we get. It's up to us to make the most of them. And, if I'm honest with myself, it's up to us to not fill them with stress about what happens today or tomorrow.</p> I'm learning. I'll get there. But it's going to take some practice.</p> One year on HRT 2024-03-23T00:00:00+00:00 Today marks one year on HRT. A year ago, I remember nervously sitting across from my doctor as she typed up the script that I'd pick up minutes later. I remember the fear of getting started, the trepidation of putting new chemicals into my body with unknown consequences. Along with that fear, a kind of hope also dared to show its head. What if HRT was good for me? What if it helped?</p> A year later, I can look back and say without any hesitation that going on HRT was one of the best decisions I've ever made.</p> In this post, I share my reasons why. I don't mean for this post to be any kind of medical advice for anyone else. Rather, I'd like it to be a record for other folks who are curious about the journey.</p> For me, HRT has affected a lot of how my body and mind work, so I'm going to try to create different sections to cover the different kinds of changes I see.</p> Mind changes</h2> Mental calm</h3> I mentioned in Happy Pride, I'm trans</a> that the early changes to the brain were eye-opening. I felt a sense of pleasant calm. The background static in my brain went down in volume, and I felt a kind of openness. I wasn't sure at the time if this would continue, and I'm happy to report it has.</p> If HRT only did this part and nothing else, it would be well worth it.</p> Mood</h3> I've been tracking my mood everyday for over an year now. It's great data to see that my mental health is indeed quite a bit better. I still have bad days, to be sure, but my overall mood is quite a bit better than it was before I started HRT.</p> Access to emotions</h3> This one gets talked about quite a bit in regard to HRT. You cry a lot more. Like, a lot</em> more. That's not a universally true thing, but does seem to be quite common.</p> I now keep a box of tissues where I'm reading or watching tv because at any point something might catch me. Crying can be a few tears or it can last for many hours or anything in-between.</p> More than just feeling sadness more clearly, I feel everything more clearly. If I want to sit and reflect on how I feel about something, I can. Those feelings are things I can access and explore. When I was working my way to coming out as a trans woman, I spent a lot of time "peeling the onion" while exploring my inner world. It meant feeling my way through some pretty uncomfortable feelings, but I have no doubt HRT was making things visible that I wouldn't have been able to easily see and feel before.</p> On a related note, the biggest of the emotional changes for me has definitely been the head-to-heart connection, so let's talk about the heart changes.</p> Heart changes</h2> I've asked around to a few of my trans femme friends if they've experienced what I'm about to talk about in this section. To date, I haven't met anyone else who has. There are a few theories here that'll share at the end of the section, but first let me just write down what I experienced.</p> At about the 6 month mark, I started consuming trans and sapphic book content. Much of what I read were romances. It was like I couldn't get enough of them.</p> I didn't think much of it, it was just hitting the right buttons, so I was going with it.</p> Then, I noticed I was feeling this tug in my heart as I read the stories. At first, it was noticeable but not terribly strong, and then it got stronger. I started pulling for these women to get together and felt joy when they did.</p> This went on for months. It was like having a very personal conversation with my heart.</p> You see, for my adult life up until taking HRT I'd dated dozens of people, but if I'm being really honest with myself, I have to admit I never fell in love with any of them. I'll go a step further and say I'm not even sure I had the capacity to. I remember years ago wondering to myself if I was aromantic. That maybe I just wasn't wired for the romance part and that staying in a relationship long-term was something that wasn't natural to me.</p> Fast forward to October of last year where I'm lying on the couch, bawling my eyes out at some novel I just finished, going through tissue after tissue because my heart just felt like it was overflowing.</p> Something had changed.</p> In the past year, I had other firsts. My first crush on HRT felt nothing like I've ever experienced before. It was like a bowling ball compared to what I had before, which was more like a marble. The intensity, the overwhelm, and the sheer hyperfocus had me begging for some kind of escape by the end. When I compared notes with my close friends, their help amounted to "welcome to being a girl and having a girl crush. Remember to breathe."</p> Something had definitely changed.</p> Somehow, my brain and heart had "rewired". In chatting with other trans folks, we came up with two possible reasons why it was happening:</p> HRT is having some kind of mood-altering effect that is giving me access to my romantic side in a way I could not access before.</li> </ol> or</p> By transitioning, I'm now more comfortable in my body. As a result, I'm open to feeling romantic in a way I wasn't before. This is because I can actually share my real self with others.</li> </ol> Whether it's #1 or #2, or even some unknown #3 option, I don't know.</p> Sexuality changes</h2> I'd been warned that this could happen.</p> First off, this year I went through an expected dip in sex drive. Right after starting HRT and blocking my T, my sex drive vanished. It was almost completely gone for about 8-9 months.</p> When it came back, I was different. Everything about my sexuality felt different.</p> How do I even explain. Let me explain this by talking about the obvious one first.</p> When the libido returned, I'd lost all interest in anything other than queer relationships. Before HRT, I was pansexual. I could happily enjoy various straight and queer content. Afterward, I just had no interest in cis straight anything.</p> As I explored a bit and let things settle down it became clear that I'm a lesbian. I might find other things interesting, but when I asked myself what I wanted, I had no doubts.</p> As a side note: it reminded me of being in my 20s watching L Word for the first time going "wow, why does this make so much sense to me?". You poor egg you. At least now I know why.</p> Other things related to sexuality have also changed. I won't go into a lot of details here, but I'll mention that being attracted to another person isn't largely a visual thing anymore. It is much more sensual and about the connection to the other person and wanting to feel closer to them.</p> Relationship changes</h2> Oh I love this part. My connection to my friends has deepened noticeably. When we spend time sharing heart-to-heart about something, we can go to a much more satisfying place, making room for uncomfortable emotions and feelings to be talked about.</p> In short, the level of intimacy possible is deeper and more nourishing.</p> Romantic relationship changes</h2> I'm writing this section as a bit of a placeholder, as I'm not yet ready to fill it in. I'm curious if other things have changed. Before HRT, I'd spent the last few decades monogamous. Now, with the connection to my heart being so different, I'm not sure where I fall. To know, I'll have to experiment.</p> The more fundamental shift is being ready to be in relationships again. I'd checked out of them for the three years leading up to HRT so that I could figure out what was going on. Coming to the conclusion that I'm trans and needed to start that journey explained quite a lot of things.</p> What will relationships look like now? I'm eager to find out.</p> Body changes</h2> Yeah, this is a big one for a lot of folks. Estrogen does whatever it does, and you get whatever it gives you. Everyone seems to be different. Here's what I got:</p> Skin: Wow, very little psoriasis now. Also more and more my skin is getting softer and nicer to touch.</li> Breasts: Currently 38D in the American system. They are perky. I'm eager for them to fill out a bit. Also, I'm regularly tender. It's something I have to be careful of.</li> Hips: At around the 11 month mark, my hips shifted. This changes your gait, giving you more swing in the hips as you walk. It also changes your body to make your hips a bit more pronounced.</li> Temperature regulation: What temperature regulation? E giveth and E taketh away.</li> </ul> Voice changes</h2> Haha, if only HRT helped with this. It sadly does not. I have been practicing regularly for the last 6 months, which has helped a bunch. But wow, it's work work work.</p> Passing</h2> An earlier draft of this post didn't include this section, but after writing it and having breakfast, I wanted to say something here. HRT does not allow me to be read as a woman. It's done many things to make parts of me more feminine, but I've never once been correctly gendered in public.</p> For some trans women, they're lucky in that regard. After a few months on HRT, their already-feminine features are enough with a bit of help to appear feminine enough to be regularly gendered correctly.</p> I'm on the other end. My face is a bit androgynous naturally, but things like my baldness, and perhaps other characteristics like shoulders, will lead me to be read as masculine.</p> My goal in this regard is to be a trans woman. What I am as a trans woman. To be myself, basically, and to grow and change to be more myself. To do that, rather than to try to fit into a box that wasn't made for me. If at some point in the future, I happen to fit into such a box, then it will be a happy accident.</p> Worth it</h1> HRT has been without a doubt thoroughly worth it. I'm healthier. I'm heading in a healthier direction. My connections to my friends has gotten deeper. My connection to my inner world has likewise also gotten deeper.</p> I can't imagine ever going back. I feel like I stepped through a portal and behind me a set of doors closed. In front of me, a set of new doors opened.</p> I can't wait to explore the doors ahead.</p> Books read in 2023 2023-12-31T00:00:00+00:00 2023 was the year I managed to rediscover my love of books and reading. Once I got turned on to the array of trans and sapphic books available, it was like I couldn't get enough.</p> Below is the list of the books I read this year, sorted by author, loosely in the order I read them.</p> One note before I begin: a lot of the books below deal with trauma of some form or another. If you're sensitive to this, please look into the book you're interested in first.</p> Stephen King</h2> Before we get into the trans/sapphic stuff, the first author I turned to this year was Stephen King. This is largely because in recovering from online addiction</a>, I wanted to read through stories of other people dealing with addiction. I ended up picking three Stephen King books.</p> Misery</a></h3> This is the main book King cites when he mentions books he's written about addiction. According to him, this book is about his struggle with cocaine.</p> Perhaps so, but the interplay at the heart of this book is definitely the reason to read it. It's so tense, and it works really well.</p> Christine</a></h3> Addiction to independence, maybe? I remembered this story fondly though after reading it I think I may have preferred the movie more.</p> The Shining</a></h3> Addiction to spirits. Interesting book - I think this was the first time I'd read it, despite having seen the movie a handful of times.</p> April Daniels</h2> Now we get into the trans and sapphic books. April was the first trans author I read. She wrote two books about a trans superhero. After I read those, I was hooked and kept looking for more.</p> Dreadnought</a> and Sovereign</a></h3> A two book series about a trans superhero. April doesn't shy away from some harsh topics and situations, giving the stories quite a bit of grit in parts. The end of the second book, Sovereign, definitely had me wanting more in the series.</p> Sakaomi Yuzaki</h2> She Loves To Cook, and She Loves To Eat (Vol I-III)</a></h3> I didn't read much manga this year, but this was a fun read that deals with aro(?) connection between two women and how they discover who they are and what they want together.</p> Meredith Russo</h2> If I Was Your Girl</a></h3> Coming of age story about a trans girl. Not bad though I think I prefer Russo's follow up book, which is...</p> Birthday</a></h3> If people are looking for a good young adult (YA) coming of age/coming out story with a bit of a romantic twist, this is one of the books I'd hand them. It's solid, deals with some real issues, and pulled me along to the end.</p> Julie Anne Peters</h2> Luna</a></h3> I... did not like this one. It doesn't feel like a relatable trans experience. The main character, Luna, feels drawn from details that matched more closely to maybe female impersonators rather than being trans.</p> Still, it's no doubt remarkable both for its age and for the fact it was a trailblazer as the first mass market YA book with a trans main character.</p> Vyria Durav</h2> The Hatchling</a></h3> This book initially felt like it was for a YA audience. All up, not a bad first effort, but truly overshadowed by Vyria's follow-up effort...</p> Catnip</a></h3> My god, this book is so good. It's like a warm bath in trans-affirming goodness. I've read it twice so far and no doubt will read it again soon.</p> Molly J. Bragg</h2> Heart of Heroes series (Scatter, Transistor, Aether)</a></h3> This three book series is tied together with recurring characters though each book is very distinct tonally and thematically from each other. Of the three, I think I liked the first one the best because of the focus and the mind-bending romance/erotica.</p> Transistor is also notable for having a trans main character.</p> Blood of the Basilisk</a></h3> Loved this one. Slow burn sapphic romance with an adorable dragon makes this one a clear fave of the year.</p> Mail Order Bride</a></h3> My god, this book is cute. Wholesome sapphic romance in space. I finished it and was so sad that it was over I just started re-reading it immediately. I wish Molly had more with these characters.</p> The Inevitable Singularity</a></h3> I... wasn't so sure about this one. It has elements of romance, though the main threads revolves around prophesy, predestination, and a kind of psychic-meets-pseudoscience fabric woven throughout. At times I found it difficult to read, both because of the density of what was being described but also because giving too much of the future plot away ruins a bit of the book's magic.</p> Tiffany E. Taylor</h2> Coven</a></h3> Please avoid. I... yeah, it just reads too much like it was written in the 90s without any of the benefit of modern thought. Was not my thing.</p> Akwaeke Emezi</h2> Pet</a></h3> There are a small handful of masterworks on my list, and this is one of them. I've recommended it to friends and family.</p> I think</em> to get the most out of it you need to have been brought up religious, but I don't think that's a requirement. It's written with beautiful prose with a lovely handling of trans-affirmation in the near-future.</p> Casey Plett</h2> A Safe Girl to Love</a></h3> A collection of short stories. I found them to be a bit too focused on 20-somethings by a 20-something to be really relatable.</p> A Dream of a Woman</a></h3> Seven years later, and her powers as an author are noticeably sharper. Her themes more on point and more mature. It feels like she's reaching to learn and yet be honest with the journey.</p> The stories in this one are also much more relatable. The first story in the collection, Hazel & Christopher, is worth the price of the book alone.</p> Alison Rumfitt</h2> Brainwyrms</a></h3> Not a fan of this one. It felt too much like a trans remix of Things Have Gotten Worse Since Last We Spoke. Some good ideas here and there but tried far too hard to be edgy for me.</p> Silvia Shaw</h2> Rand books 1, 2</a>, and 4</a></h3> Yes, you read that correctly. I accidentallly read these out of order. In my defence, the fourth book says "Book 1" on the cover and is indeed the first book of a new series in the same world as the original series.</p> These are solidly plotted, well-written sapphic fantasy novels. I'm looking forward to getting back around and reading the 3rd one when the mood hits. Of the ones I've read so far, I think the 4th book (Daughter of the Light) was the best.</p> Issy Waldrom and Mackenzie Sutton</h2> Best Friends' Promise</a></h3> Trans sapphic vampire romance. It's exactly what it says on the cover. An enjoyable ride, and one that I recently re-read and still enjoyed.</p> Ela Bambust</h2> Any Other Name</a></h3> Loved this one. Magic school meets Beetlejuice in a trans/queer-affirming way? Yes, please. Tightly plotted and well-written.</p> The Principles of Non-Euclidean Romance</a></h3> One part Groundhog Day, one part eldritch-flavoured romance? I have no idea how to describe this. I liked some parts more than others, though I think preferred the tighter feel of Any Other Name over this one.</p> Rosemary Kirsten</h2> The Steerswoman</a></h3> Wow, we actually took a quick detour away from trans and sapphic books for a minute to read this one because friends kept asking us what we thought.</p> I think my short summary is: it's well done. It's nice that it has some clearly autistic spectrum characters in it, and their unique view of the world matters deeply. I just found myself wishing that there was more queerness among the characters.</p> Maria Ying</h2> The Grace of Sorcerers</a></h3> This took me a couple tries to fall into, but after I did was richly rewarding. The prose is stellar, and the characters build as you progress.</p> Andrew Joseph White</h2> The Spirit Bares Its Teeth</a></h3> My god. Here's another masterwork. That's not to say it'll be an easy read for anyone. Andrew is steeped in the dark Southern history of literature, and it's pretty clear. I'm actually speechless as I write this, as I truly have no idea how I might do this book justice.</p> If you can make your way through it, I think the book richly pays for your investment. Especially, after the end when you find out what the book is based on. That killed me. I just walked around the house in a daze.</p> Kai Cheng Thom</h2> Falling Back in Love with Being Human</a></h3> I laughed. I cried. I... honestly just wanted to take Kai out for coffee after reading this. Just to sit across the table from someone else who wanted to marry being honest about pain with desperately looking to find ways to forgive and heal.</p> The book is made up of letters to various people who had done harmful things in Kai's life, including to people like Jesus and JK Rowling. At times poetic, it flows from the heart with a keen edge.</p> Mel E. Lemon</h2> Violet Moon</a></h3> Slow burn werewolf sapphic romance. I'm looking forward to reading more in the series as they become available.</p> Zoe Storm</h2> The Bradford McKinley Chronicles (books 1-4)</a></h3> Zoe is one of my go-to authors for coming of age/coming out trans books. These are the kinds of books I would hand other newly-out trans folks, or folks who are questioning whether or not they're trans.</p> The 4th book in the series especially is quite good. It deals more directly with dysphoria and how to work through it, and for that reason alone will be something I recommend to others as they deal with their own dysphoria.</p> The Enchanted Forest</a></h3> A collection of five trans-focused short stories. Some of these stories, like the vampire-themed one, really feel like they want to grow to be their own larger work.</p> Avery Arden</h2> Blood Mistress (1</a>, 2</a>, 3</a>)</h3> So you say the other things I have listed so far are a bit tame? You want something a bit spicier? Yes, come feast at this three novella set (which I think of as one novel). Spicy sapphic vampire erotica with heart. The third book especially definitely fills out this world and makes it more alive, which I enjoyed.</p> Oh, and it actually emphasises consent. Thank goodness.</p> SD Simper</h2> Carmilla and Laura</a></h3> I loved this. More vampire goodness. Honestly, it feels like the third masterwork in our list. It so deftly allows the readers to watch the characters grow, come into their own, and discover themselves. It's erotic but also has so much heart to go along with it. Definitely one I'm looking forward to re-reading.</p> May Peterson</h2> Calyx Charm</a></h3> I saw this book called out by a few folks as one of their faves. I was quite a bit more mixed on it. I struggled with the narration style through a lot of the book, as the plot would slip from present day, to past, to possible future without much indication as to where you were.</p> That said, it has by far some of the hottest trans erotic scenes I read this year.</p> Elena Abbott/Lily Hardt</h2> Pack of Her Own</a> and Mate of Her Own</a></h3> Elena's werewolf series with strong trans representation and sapphic romance/erotic elements. I quite enjoyed this, though I'd recommend spacing them out a bit. Elena isn't afraid of dealing with different aspects of trauma in her work, and I definitely noticed feeling overwhelm after I finished the second book.</p> Need (as Lily Hardt)</a></h3> BDSM and trans? Yes. Dealing with internalised trauma? That, too. This book is erotic but it's really more about healing.</p> That's wrap</h2> And that's it for my list. If any of you have reading recommendations that fall along with this list, let me know! I got turned on to a lot of authors that I'll be watching in the coming year(s).</p> Becoming Sophia 2023-12-08T00:00:00+00:00 I started thinking this year what a "Year in Review" might look like, but to be honest, I've had one bright theme running through it that I kept coming back to: being trans. This post then will be some reflections of the year with that in mind.</p> Getting clean</h2> The year started off with me realising I had a massive YouTube addiction</a> and facing it head-on. This helped me put to rest a struggle I'd had for 12 years. I could tell that I'd been using the addiction to numb something painful, but since I kept numbing, I wasn't able to figure out what it was.</p> My goal after getting clean was to figure out what I'd been so desperate to numb.</p> Looking in a mirror</h2> Between January and March, I did a lot of soul-searching. Actually, that was a big theme this year. I tried whenever I could to reach far back into my life and connect to younger versions of myself, to ask them what they needed, and to see if I could meet them in the present day.</p> One way I started doing this might sound a bit silly: every day, I turned my phone to selfie mode, looked at myself, and repeated "I love you" until I could feel something. At first, I just felt stupid, but I kept going because I knew that if I wasn't going to be my own advocate and my own support, I was just going to fall back into addiction and lose sight of actually trying to take care of myself.</p> I didn't feel any changes from doing this daily practice, but I kept going. I figured if it did anything, it'd take a long time. While I didn't feel any different, in hindsight I can safely say that it was doing something. It was getting me to see myself as a person worthy of protecting and supporting.</p> That plus doing the "eat the frog" practice of making a list of things you're avoiding/afraid of and then one by one going after them let me slowly move away from hiding from myself, to working on myself.</p> Hormones</h2> By March, I'd started hormones</a>. I'll let the post talk about the benefits of going on hormones for me, but it opened my eyes to feeling noticeably better than I ever have. It wasn't just in my head, either. I had been using moodflow</a> to track my mood every day for months before starting hormones, and you can watch the shift happen around March. My mood significantly got better and stayed better after March.</p> COVID</h2> This July was the first time I've ever gotten COVID, and it hit me pretty hard. I spent weeks barely able to get off the couch. At one point in those weeks, I got up and made it to the laptop at the kitchen table and quickly typed up a list of things I wanted to do with my life. When I was done, there were only two items on it:</p> Have a healthy transition</li> Finish "secret project"</li> </ol> I'm not talking about #2, yet, so let's talk about #1. I was just shocked looking at it I only really cared to focus long-term energy in two places, especially when my energy was so low (at the time). As I started recovering, I dove further into transition. How to do it, but also, searching to understand what I wanted. Up to this point I thought "oh, I'll just get on hormones" and not think too hard about what happens after that. After getting COVID, I knew it meant far more to me than that.</p> Rekindling my love of books</h2> On a whim, I start picking up trans YA (young adult) books. I couldn't tell you exactly why, other than I wanted something to read and seeing what was available sounded fun. This sent me deep down the rabbit hole. Now, dozens of books later, I can say that I've had quite the journey with not only trans YA books but trans books generally. I've laughed, cried, and saw parts of myself that I'd tried to keep safely hidden since I was 5-6 years old.</p> Before I move on, here are some of my current recommendations (this set is all YA or YA-adjacent):</p> Birthday, A Novel by Meredith Russo - a read a few trans coming of age/coming out books, and this one stood out to me.</li> Dreadnaught and Sovereign by April Daniels - Two book series about a trans superhero.</li> Pet by Akwaeke Emezi - Trans protagonist, near-future with a fantastical twist.</li> Any Other Name by Ela Bambust - Magic school meets a bit of Beetlejuice. Bonus: see how many digs against JKR's world you can find 🙂</li> Bradford McKinley Chronicles by Zoe Storm - Easy to read set of coming out/trans-focused books. The 4th in the series is surprisingly good.</li> </ul> Love and support</h2> In October, a dear friend of mine came to visit. By this time, I knew I was a trans woman but was only out to a few friends. She took me shopping, gave me loads of hand-me-down outfits, and just generally was a dear support during the week she was here. While she was here, I woke up one night at 2am, after finally breaking through my dysphoria enough to know that it was time to come out publicly. The led to me pouring my heart out in a letter to my immediate family.</p> The letter</h2> Copied below is the letter I sent my family. I put it here for other trans folks, and potential trans folks, as data they can use in their own journeys.</p> "I'm writing an email to you all to put everything in one place about what's going on with me. I hope you don't mind the length of the email, but there's a lot to talk about, and I want to give you the full story.</p> I'll also say before I start that I hope you all will read this with an open heart. This stuff took me a long time and a lot of work to figure out. I'm sharing them because I don't want to hide. I know in our family we don't often talk about "the hard stuff", but in my experience not talking about it makes things so much worse.</p> I've always known I was different from other people. A bit of a black sheep. I didn't really understand why, I just knew that how I thought about things always felt out-of-step with people around me. One of my earliest memories of this goes back to being 5 or 6. I was at church, playing house with a girl my age. After we were done she was so happy to have pretended to play the wife role, and I just felt... strange. The chaperone there said how cute it was that we could play so well together and I remember just looking at them, wanting the praise, but also being struck that these weren't my dreams.</p> I remember learning to sew, knit, and crochet in California. I remember Dad saying he didn't want me to be a sissy. I tried to live up to what he wanted and make Dad proud. But it's hard when what comes naturally to you brings disdain and ridicule. You learn to push it down, to hide who you are from the world. So that's what I did.</p> It was in my teen years when it first hit me hard. It felt like someone had punched a hole through my heart. I could feel it inside me, a deep pit that felt like it had no bottom. I thought it was just teen stuff. They tell you that depression and anxiety is normal for teens, so I thought maybe I'd grow out of it. Little did I know that this hole would follow me for the next 30 years.</p> I'd grow older, go to college, date and watch my friends form serious relationships, get married, and have children. But I never went that path. When I would date, that hole in my heart would eat at me. I knew each relationship was going to end, regardless of how nice the person I was with was. I didn't understand it, and at the time I just thought I hadn't found the right person, yet. So I just kept looking and breaking hearts as I went.</p> It's not that I didn't try to figure it out. I spent years in therapy with different therapists, I went to churches and talked to pastors, I meditated, you name it. I just kept going still thinking when I met the right person it would all fall into place.</p> I kept moving, literally, and at each place I kept looking.</p> I had a therapist one time tell me that "you can keep running, but you know it will follow you wherever you go, right?" I didn't really appreciate at the time how right she would be.</p> In January of this year, I was talking to a dear friend and off-handedly mentioned that I watched a lot of YouTube. She asked how much, so I went and added it up and then stared at the result. For the three days prior, I'd average eight and a half hours a day. And that didn't count the other content, like Netflix, that I'd watched. I couldn't remember how long it'd been like that, but it quickly sunk in that it had been months, and likely years. I knew I had a problem. I could feel that I wasn't using it for entertainment anymore. I was using it to numb myself so I wouldn't have to feel. I saw in myself what I'd seen in my friends who struggled with alcohol and drugs. The desire to drown out something that I was afraid to face. I quit and then immediately started reaching out for help. My doctor connected me to addiction services, who connected me to a counsellor who could help me. And he did. He got me to slow down and start facing those things I was afraid to face.</p> Working to face my fears eventually landed me back in the doctor's office trying to address mental health issues that had been a constant part of my life. The symptoms matched a cluster of symptoms often associated with dysphoria, specifically dysphoria associated with being transgender. In March of this year, I mentioned this to my doctor, and then asked if we should explore treating it. She heartily agreed and put me on my first prescription for hormone replacement. Within two weeks I knew I'd made the right choice. I could feel the improvements almost immediately. My brain was clearer. I had less background noise. I felt so much better that I thought the hole in my heart was starting to heal.</p> But what did it mean? At first I was content just to enjoy my improved health, but then I started to ask questions. Why was estrogen so much better for me? Why did testosterone hurt me?</p> So I started to dig. Like putting a puzzle together without knowing what the puzzle was supposed to look like. I just started pulling the pieces out, trying to make sense of them.</p> I also was afraid of what was underneath all those layers, too. After all, I'd tried so hard through my life to bury whatever it was and to pretend it didn't exist. What I'd buried though was an enormous amount of pain. I didn't want it to swallow me, so I had to deal with it.</p> I once had a friend of mine ask me "if you had a button that you could press and wake up as a woman, would you?" I didn't have to think about the answer. I blurted out "of course, but everyone would do that" She just gave me a knowing look.</p> I mean, everyone would push the button, right?</p> It took a while for it to really sink in what the ramifications of my answer were. I'll admit it, I can be a bit dim sometimes.</p> As I put the puzzle pieces together it started to paint the picture. I didn't fit in the male world. I never have, and I never would.</p> I looked instead to what brought me joy. Not a surface level joy, but the kind that would touch deep inside. I know that if who I am doesn't spring from that place, it will never be fully authentic. I tried to ask that part of me what it wanted, but at first it didn't answer.</p> I waited and instead asked other questions. As the months passed, the hormones started to have an effect on more than just my brain. My body, too, began to shift and change. I asked myself if this is what I wanted and got a resounding "yes!" from deep within.</p> Trying to listen for that small voice inside takes care and time, so I let it speak, and I would listen when it did. Bit by bit, I saw enough of myself reflected back to me that I knew what I was looking at.</p> I am a trans woman, meaning that the deepest part of me isn't male -- it's female. And the journey I'm on is one of transition, shifting from outwardly being male to outwardly being female. This kind of transition isn't something that happens overnight. Instead, it happens slowly over months and years.</p> If you're surprised by what I'm saying, don't worry, you're in good company. Had someone told me years ago this is what would happen this year, I'm not sure I would have believed them. Truth be told, I wasn't really ready then to face it.</p> But now I am, and I'm going to lean into being who I am. I hope to have your support. I know the path ahead won't be easy, but I can tell you from the deepest part of me that it's the right thing to do.</p> I'm not entirely sure how to sign this email, as I don't know what my name will be. So, I'll sign it with a name that means one of my favorite things in the world: wisdom. Who knows, perhaps the name will stick.</p> -- Sophia"</p> Speedrunning wisdom</h2> It's now December and Sophia, as a name, stuck. Sophia June Turner is what I moved to, and I've begun the many (many) processes to change my name in all the places I can.</p> The bigger change, though, was how I viewed my transition. I know who I am. I know where I want to go. I told some friends that I went from "hormones are helping" to "how do I speedrun transition" in eight months. That's not far from the truth.</p> Here are some on-going and upcoming things I'm working on:</p> Regular conversational voice practice. I'm able to hold femme range and head voice, but I still need to work on femme cadence and inflection. Yesterday, I managed over 6 hours of practice, which is a huge step up from where I was a month ago, when I barely had the courage to practice at all.</li> Next week, I'm having my first large effort electrolysis with a place that uses the modern electrolysis machines. The new machines should move a bit quicker, and in theory should be less painful, than the old machines. All up, I have 6 hours of electrolysis planned over 3 sessions. We'll see if I can take it.</li> Next month, I have my first major surgery: hair transplantation. They're moving follicles from the back of my head to the front and top using a class of techniques that fall under FUE. As part of this, they'll be sculpting a more feminine hairline as well.</li> I'm still trying to figure out how and when to get back to the States for my official name change, but once I figure that out I'll kick that process off.</li> </ul> The road ahead takes work</h2> There's a lot of work ahead of me as I transition, but I think of it like walking a long distance. I can do it. I'll have to rest from time to time. I'll hit obstacles I have to find a solution for. But, I'm going to keep going. That's the promise I made myself. The joy I feel every time I hear my name, my pronouns, or feel a change in my body as I go are just signs along the way I'm heading the right direction.</p> The case for Nushell 2023-08-30T00:00:00+00:00 Recently, I had a chat with some of my friends about Nushell and why they stuck with traditional shells like bash/zsh or the "new" hotness like fish rather than using Nushell. After chatting with them, my brain kept bubbling away at the state of how folks were using their terminals and the end result is this blog post.</p> In this post, I make the case for really taking a hard look at Nushell and also for generally asking the question: "can the state of shells be improved enough to overcome the inertia of sticking to what you know?"</p> The lay of the land</h2> Let's take a look at some of the offerings out there that people are using everyday.</p> Bash/zsh</h3> Bash, originally a set of improvements on the Bourne shell, has grown to be the default shell for almost all Linux distros. That's generally people's first experience when they hit the terminal. It's what they see when they log into a remote machine. It's reached the definition of ubiquitous.</p> I also throw 'zsh' in here as well. Apple's macOS switched from bash to zsh, an operationally similar shell but created a bit more recently.</p> Bash at this point has become so well known that people often confuse support for bash-isms as part of the POSIX standard, but we'll talk about that later.</p> Pros:</strong> it's everywhere. Learn once, run anywhere.</p> Cons:</strong> as a language, bash/zsh feels a bit too retro. It doesn't offer any of the modern programming language style, tool support, etc folks would be used to from other languages. In truth, bash was never really meant for writing the kind of large scripts that people are maintaining today.</p> Example for loop in bash:</p> #!/bin/bash for i in `seq 1 10`; do echo $i done </code></pre> Fish</h3> As fish's website says: "Finally, a command line shell for the 90s"</p> It's enough to elicit a smirk, because you know it's a bit true. The bash/zsh style shells are getting left behind by something that feels a bit nicer, has nicer completions, looks nicer (you can get similar improvements out of bash if you work at it, but fish ships with them out of the box)</p> Fish also bravely steps away from the shell scripting form of bash to something a bit more readable.</p> Example for loop in fish:</p> for i in (seq 1 10); echo $i; end </code></pre> Pros:</strong> the interactive experience of fish does feel quite a bit nicer that bash/zsh out of the box. Scripting is a bit nicer.</p> Cons:</strong> As it says on the tin, it's a shell for the 90s. It ain't the 90s anymore.</p> PowerShell</h3> Coming into 1.0 at 2006, PowerShell is one of the first shells to really draw a line in the sand to say "enough, we're going to do things differently". The unix style of pipelines, where commands communicate via text to each other was replaced by a .NET engine that passed objects between commands.</p> The impact wasn't immediately obvious but as devops folks (and others) discovered what was possible when you have ways to work with data directly a fanbase grew.</p> Example of a for(each) loop in PowerShell:</p> foreach ($i in 1..10) { echo $i } </code></pre> PowerShell came with an opinionated design that focused on verb-noun naming, improvements to shell syntax, and a vast set of functionality drawn from the .NET ecosystem.</p> Pros:</strong> it's a structured shell - you can actually work with objects rather than text. Powerful set of tools and capabilities taken from .NET.</p> Cons:</strong> I'll go ahead and say it: PowerShell was never really designed to be a language first. The verb-noun convention forces a style that feels very awkward coming from other languages. Worth a mention: earlier versions of PowerShell were Windows-only and modern crossplatform support lacks some of the features of the earlier versions.</p> Other shells</h3> When I was coming up, there were a lot of other shells, including the csh/tcsh family. Having said that, I don't know anyone who is using any of the other family of shells. Bash/zsh and to some extent fish really dominate the developer mindshare.</p> Hold up, we really need to talk about POSIX</h2> We really need to take a minute and talk about POSIX before we continue. A lot of folks have leveled "but it's not POSIX" as an argument against using Nushell, but I'd like to turn that around and ask the question:</p> "What's so good about POSIX?"</p> Most folks when asked would likely point to it as a common ground that code can be ported to. In reply, I'd like to quote a few bits of the POSIX standard.</p> The following are reserved words in the POSIX standard for shell scripting</a>:</p> case</li> do</li> done</li> elif</li> else</li> esac</li> fi</li> for</li> if</li> in</li> then</li> until</li> while</li> </ul> Yes, really. fi</code> and esac</code> are a joke that never found their end. No one would design a language that did that with a straight face these days.</p> Let's take a quick look at the number of flags common Unix commands ship with. These are on my macOS system, so ymmv.</p> command</th> number of flags</th></tr></thead> ls</td> 45</td></tr> man</td> 15</td></tr> ps</td> 29</td></tr> </tbody></table> If you look through the flags of ls</code> to see why it has so many, notice how many are configuring what ls</code> is displaying. In a real sense, this is going against the underlying philosophy of unix pipelines. Rather than composing a pipeline to get the display you want, you're learning a language of flags for each command to configure the display.</p> Let's talk about exit codes. Actually, wait, I already did that</a>. As I point out in the post, the standard says:</p> "The value of status may be 0, EXIT_SUCCESS, EXIT_FAILURE, [CX] [Option Start] or any other value, though only the least significant 8 bits (that is, status & 0377) shall be available from wait() and waitpid(); the full value shall be available from waitid() and in the siginfo_t passed to a signal handler for SIGCHLD. [Option End]"</p> </blockquote> Sure - we all still use 8-bit machines while flipping through a manual we printed trying to find the exit code description to understand why a command failed.</p> I hear you: "but, look, it doesn't matter how archaic this stuff is, if we all agree to use it things keep working."</p> I dunno - these arguments just don't hold up. We'd still be using C as our main systems language because it's the most documented, most portable, etc. But, by and large, we don't. We're increasingly choosing other languages.</p> The truth is, in 2023 if someone asked us to design a system, we wouldn't design POSIX. If, in 2023, someone asked us to design a shell language, we wouldn't design bash/zsh. This matters.</p> Show us the money</h2> I make some pretty bold statements in the above. There is a heritage of technology that got us to this point. While that heritage is important, it's not without its drawbacks. Are there better ways of doing it?</p> Why structure matters</h3> Before we get into talking about Nushell, let's talk about why structured data matters.</p> In the Unix pipeline way of thinking, text passes between commands. This is very flexible, but has a major problem: both the outputting command and the inputting command have to agree what shape that text will take so the info can be passed. This locks representation</em> to presentation</em>, disallowing commands from evolving their output over time. As we showed earlier, it also encourages a proliferation of flags.</p> That's annoying. Does separating the structure from its presentation help us?</p> > ls \| where size > 10kb </code></pre> I always start with this example when showing off Nushell, because not only is it immediately readable, we didn't have to dig through any flags to figure out what we needed to pass to ls</code> to get that. We also aren't parsing anything from ls</code>. Instead, the data is passed directly to where</code>, which handles it directly.</p> Commands already know this structure, why not make use of it?</p> The same where</code> command works on other things. For example, we can process the output of the ps</code> command:</p> > ps \| where cpu > 40 </code></pre> Or open a csv</code> file and processing its rows:</p> > open fields.csv \| where area > 5 </code></pre> And so on. It's the same where</code> regardless of where the data is coming from. It also gives us the freedom to present the data however we want.</p> Why Nushell matters</h2> Nushell is designed to be a language</h3> I had the good fortune of being a part of some prominent programming language teams, including helping to create TypeScript</a> and helping create Rust's error messages as part of the Rust team in Mozilla</a>. Designing languages to be easy to use, easy to read, easy to scale up, and easy to debug is something I care about and have worked on for many years.</p> To that end, Nushell is designed with an eye towards being readable at a glance.</p> Let's do a for</code> loop in Nushell:</p> for i in 1..10 { print $i } </code></pre> (aside: "but why do variables have dollar signs?". Turns out the flexibility of shell programming allows paths to not use quotes, so it's nice to tell a difference between cd foo</code> and cd $foo</code></em>)</p> This eye towards usable design shows up in many ways. Working with data is improved by not only having structure, but also being able to pattern match against it. Here's an example of pattern matching a list in Nushell:</p> match $list { [$one] => { print "one element list" } [$one, $two] => { print "two element list" } [$head, ..$tail] => { print $"the tail of the list is ($tail)" } } </code></pre> In a way, working in Nushell should feel at home both interactively as a shell and as a full scripting language. We've had folks write COVID reporting software in Nushell, research experiments, even entire shells for well-known database services</a>.</p> Nushell is typechecked</h3> Since Nushell doesn't treat all data as text, you can represent tables, records, numbers, booleans, etc directly in the language.</p> As a result of this, Nushell is fully typechecked. Common errors can be caught early and shown to you before the script even runs.</p> </p> Taking what we learned from TypeScript - the types also feed into another important tool.</p> Nushell has IDE support</h3> The types, autocompletion, and early error reporting feed into an engine in Nushell that knows a lot more about your code. As a result, you can write scripts and then work with them using the IDE support Nushell provides. Seeing errors, jumping to definitions, getting documentation on hovers, etc are all part of the Nushell experience.</p> </p> Nushell has nice errors</h3> In Nushell, we make extensive use of remembering where data comes from, as well as what caused an error. Simple errors, like division by zero, are shown clearly:</p> </p> A more complex error may need to show more to help track down where a mistake came from. Let's say you've accidentally put a string in your list of numbers, and then tried to process it:</p> </p> Nushell has a SQL-like style</h3> </p> When you start using Nushell to compose pipelines, you'll notice that it has a distinct SQL-like style. Data flows through each stage, and you build up what you want to do to it as you add more commands.</p> This gives Nushell a distinctive design that encourages experimentation and exploration.</p> Nushell is, and has always been, crossplatform</h3> An important decision we made from day 1 was to be crossplatform. You can run Nushell on Windows, Linux, and macOS (and BSD, and Android) and get the same experience. You can easily write scripts in a way that they can be run across different platforms. Everything that you learn transfers between OSes without friction.</p> Is Nushell good enough to overcome the inertia?</h2> I distinctly remember going to a SIAM conference many years back and giving a talk on the Chapel programming language</a>. Even back then, it was a clever language. In a couple lines, you could write code that could distribute and process a matrix across a network of computers. Coming from a lineage of array languages, it ate up data parallel processing. The equivalent code in other languages looked verbose in comparison.</p> I went through my talk, hoping I'd done a decent job of conveying the main points, and at the end, someone in the audience stood up and said "but I can do all this in C++".</p> He proceeded to explain that if he could recreate many of the techniques we showed all as part of a C++ library that people could use. At this time, I wasn't sure how to respond other than "but you don't have to, we already built this language" but he couldn't be swayed. If it wasn't C++, he didn't want it.</p> Fast forward a couple years, and I'm standing in front of a JavaScript audience giving a similar talk, this time promoting TypeScript. I remember the kind of politely confused looks on people's faces as I showed off the features TypeScript offered. There was a similar sense of "why do we need to leave JavaScript?".</p> To answer whether Nushell can overcome this kind of inertia, I'll pose two questions:</p> Is Nushell compelling enough for a single person to adopt it?</li> Would adopting Nushell broadly as a community move the needle?</li> </ul> Let's tackle the first question. Time and again, as people try Nushell, they come back with quotes like "this is the most excited I've been about tech in 15 years". It has a fanbase that loves it, and that fanbase is growing. It reminds me of the early days of Rust, just after hitting 1.0.</p> To the second question: would adopting Nushell broadly actually improve things noticeably? Without a doubt. I say this without any reservation. Thinking of our shells as structured, interactive processing engines opens up the doors to a much wider array of things you can do with them. The commands would be far simpler than their POSIX equivalents and would compose far better. They'd benefit from the full knowledge of the data being shared between them. Adaptors could be made to connect to all parts of the system, allowing you full, structured interaction with everything you have access to.</p> In essence, as the saying goes, we'd be building a skyscraper starting from the 15th floor instead of the 1st.</p> It's time to be honest about what Nushell is</h2> It's time I come clean about what Nushell is. Nushell isn't exactly a shell, at least not in the traditional Unix sense of the word. Nushell is trying to answer the question: "what if we asked more of our shells?"</p> Nushell is really an interactive, data-focused scripting language with shell capabilities. It merges these three things into one:</p> A fully-typed scripting language</li> An interactive shell</li> A data processing system (that can also handle large data loads via dataframes)</li> </ul> Rather than these being three separate ideas glued together, in practice it feels like Nushell is treating everything you interact with as data. This allows you to pull together different kinds of data, and know that the same commands will work over them.</p> You might look at that list and think you don't need all that, but the way I might frame it is this: it's nice to have it when you need it.</p> I don't need to do heavy data processing everyday, but it's nice to not have to shift what I'm doing at all when I need to do it. I don't have to download new utilities or switch languages. It's all right there. Need to write a script to load some files and handle some directory processing? Still right there. Need to throw together some web query that outputs the top download results for a github repo? You guessed it, all still right there.</p> This is just scratching the surface, really. Nushell has a plugin system that allows more capabilities to be added based on your needs. We already have plugins that add a variety of additional file formats, querying capabilities, and more.</p> It's okay to have nice things</h2> Nushell was built with a simple idea: working in the shell, writing code, and processing data should be fun</em>. To that end, we work hard to make Nushell feel nice.</p> You can write readable scripts that come with their own documentation, and then come back to them 6 months later and still understand what they're doing.</p> You can sit in the shell and play with pipeline ideas until one grows into a scripting project and then effortlessly transition your experiment into a full script.</p> That's it</h2> That's my case. It's okay to have fun. It's okay to write attractive, well-documented code. It's okay to leave the designs of the past behind when they no longer fit the present day.</p> It's okay to move on to better ways of doing things.</p> Happy pride! I'm trans! 2023-06-26T00:00:00+00:00 Hi, I'm trans. Let's talk about it!</p> Note:</strong> In this blog post, I talk about the medical side of my particular kind of transgender experience. If that's not your thing, feel free to skip.</p> A long time in the making</h2> The first time I looked up anything trans-related was in my early 20s. I didn't know it at the time, but I was actually looking for a way to help address a set of symptoms I was having.</p> For most of my adult life, I experienced a cluster of issues: dissociation, derealization, depersonalization, and others. I thought it was normal to feel disconnected from your body, and for parts of your brain to feel disconnected from other parts of your brain.</p> What I didn't understand, but have come to learn this year, is that I have biochemical dysphoria</a>. In short, this means that the hormone that my body used to regulate, testosterone, was not the correct regulating hormone for me. Think running an engine on the wrong fuel: it kinda works but it's really not good on the engine and you'll have side effects. Switch to the correct one, and you'll notice a big difference.</p> The fix, as things go, is relatively simple. Remove the hormone you're starting from and replace it with the other hormone (using a process often called HRT). If you have the condition, you'll likely start feeling noticeably better rather quickly. If you don't, you'll likely feel worse.</p> The first few weeks of HRT</h2> Before I talk about my experiences on HRT, I want to say that I'm just sharing my experiences rather than giving medical advice. Everyone is different, and there's really no telling how anyone else would respond. I did this through a doctor, as I luckily had access to trans-friendly medical care. It's my sincere wish that in the future everyone</strong> will have that kind of access.</em></p> The early impact of HRT for me was certainly eye-opening. Mental health symptoms I'd had for decades started to lift within the first 2 weeks. I no longer felt internally disconnected. Instead, internally I had more flow, comfort, and a feeling of openness.</p> I think my reaction could be summed up with "what... in the world?" in a very good way. I remember walking around my apartment mumbling to myself "I hope this doesn't stop, because this is great." Luckily, so far, it hasn't.</p> Along with the drastically improved mental health I had two other changes: emotional changes and mood changes.</p> Emotional changes</h3> The emotional side also began to change rather quickly. I think this could be described as someone handing me glasses for the first time. Whereas before emotions felt like they were piecemeal and difficult to understand, now emotions feel clear. I could cry, and smile, and feel each more deeply and more connected to my inner world.</p> I love it. I'm able to speak more of the language of the inner world, which is helping me not only understand myself, but also communicate better with other people.</p> Mood changes</h3> Since November last year, I've been daily logging my mood on my phone. I was hoping to find some patterns or some kind of good data.</p> Without really realising it, I managed to log months of data before starting HRT as well as the months following. With it, I got a pretty clear picture of how HRT affected my mood.</p> Before starting HRT, my average mood around around 6 out of 10. After starting HRT, within a couple weeks, it had risen to 8 out of 10.</p> It wasn't just on the app. Friends who talked to me who hadn't seen me in a while would remark "whatever you're doing, keep doing it."</p> The first three months</h2> While I won't list all the changes, as honestly something</em> should probably be private in this blog post, let me highlight a few more changes:</p> Week 3</h3> At week 3, I had my first big emotional wave. I've never felt something like it before. Imagine being a small boat on a very large sea and the wave effortlessly lifts you up and down.</p> It took a bit of practice (and no doubt something I'll continue to learn) to ride this wave. Along with this, though, I noticed that these swells brought with them a kind of sharp focus when noticing things that are wrong/unjust/etc. You can see and feel when something isn't right, and it can be confusing when other people don't seem to notice or don't seem to feel it as intensely.</p> Week 10</h3> Around week 10 my skin started to change. It changed to have a softer texture which is quite pleasant.</p> Along with the above, my psoriasis started clearing up. Patches which had been there for years shrunk, and are continuing to shrink.</p> The skin changes also got me starting to look into different fabrics and clothing, as the coarse men's clothing honestly doesn't feel all that great on the skin as it gets more sensitive. It's nice to put on something and feel soft and cozy.</p> And that brings us to today.</p> Today</h3> I've just gotten back from my doctor appointment for my 3 month check-in. Clean bill of health, so on we go (thankfully). My doctor was very pleased with how I was taking to the treatment, and clearly I am as well.</p> Questions one might have</h2> Okay, so there are probably a few questions that could come to mind reading this, and no doubt a few others I haven't thought of. Let's see if we can address those.</p> Trans... what, exactly?</h3> Being transgender means being a part of a rather large umbrella. Many folks know that there are trans women and trans men. For me, though, my trans-ness means to leave maleness and go off into the unknown.</p> I don't really have a goal. Hmm, I take that back. My goal is to be healthy, and I try to put the effort in each day to make that happen. What that means may change over time, but I'm not rushing it.</p> Aren't there body changes on HRT?</h3> Yes.</p> And?</h3> Oh oh you mean, what will I do when they happen? Again, taking it a day at a time. I don't have all that figured out, and I'm okay with that.</p> On a personal note</h2> It scares me how political trans-ness has become when, especially for my case, it's medical. It's something I'm working on between me and my doctor. I'm betting that's true for lots of folks: they just want to live their lives.</p> It scares me that receiving medical treatment is difficult for a lot of people, especially seeing just how effective the medicine can be. The changes can be profound, and in some cases, also life-saving. To withhold that feels cruel.</p> I'm sharing my experience here to both be a record for myself, and also to be something other folks can read to learn about what it's like to have biochemical dysphoria and to have it treated. I wish I could share the joy that I have now, the feeling of relief, for being able to address things I've struggled with for over the last 20 years. Do I wish I could have started sooner? Absolutely. Maybe this post will help other folks ask questions and get treatment sooner, too.</p> Why I left Rust 2023-05-28T00:00:00+00:00 There's a lot of speculation about why I left Rust yesterday, so I want to set the record straight in this post.</p> What happened</h2> A short timeline of what happened from my perspective, as this is relevant for later:</p> A request went out to the interim leadership group for potential keynote speakers for RustConf.</li> JeanHeyd Meneide was proposed by myself and Manish as a keynote. JeanHeyd is one of the two editors of the official C standard</a>, a C++ expert, and a good speaker. We felt like he'd be a great invited expert with an outsider perspective.</li> A few days later it was held to a vote. Consensus in the interim leadership group (what the official blog calls the "leadership chat") are of those present in the group when the decision is made. In total five members voted, and all five votes supported JeanHeyd as the keynote speaker.</li> JeanHeyd was asked and accepted the keynote.</li> Before the schedule was posted, it was discussed in a team meeting that JeanHeyd had been selected as one of the keynote speakers.</li> A couple team members had strong opinions/discomfort against JeanHeyd being selected as a keynote speaker, as best as I understand it, because of the content of JeanHeyd's blog post on reflection in Rust.</li> This discomfort was brought to the interim leadership group as a problem that must be fixed immediately. There was no discussion of the ramifications of making a change at this point. There were complaints that this pushback was not sufficient to require change, but this feedback was ignored. The discussion focused on the discomfort of the team members and changing JeanHeyd away from being a keynote speaker as the only solution.</li> Two people in Rust leadership then, without taking a vote from the interim leadership group (remember, JeanHeyd was voted on and selected by Rust leadership), reached directly to RustConf leadership and asked to change the invitation. (note: this used to say one person, but I've now had multiple people independently confirm it was two people.)</li> RustConf leadership decided to wait a week before informing JeanHeyd, giving Rust leadership time to change its mind. The interim leadership group was not informed of this week delay.</li> After the week expired, JeanHeyd was informed of the decision to downgrade the talk. Understandably, he fully declined to speak at RustConf.</li> When I read JeanHeyd's blog post and learned of the declining of the invitation, I immediately resigned from the Rust project.</li> </ol> I'll also note that the above is the best I am able to understand and piece together. A full investigation needs to happen to uncover how and why the above happened. I also reserve the right to edit the above if new information comes to light.</p> Why did I leave?</h2> To my mind, why I left is obvious. I just watched the Rust organization disgrace one of the experts in my field. As an organization, Rust and RustConf offered little resistence to Rust being wielded as a weapon against an individual because of some discomfort of a couple team members.</p> What rang in my mind is: "a system is what a system does"</p> Rust acted as a cruel, heartless entity that did not care about JeanHeyd and treated him as disposable. Easy to offer a place of respect and just as quick to snatch it away. That is what Rust is</em> because that is what Rust did</em>.</p> I left because when I felt JeanHeyd's pain and disappointment at being mistreated and betrayed, my heart broke. I wept because of the cruelty. But I also wept because I helped create the system that could do this to someone.</p> But it was just a downgrade</h2> I shake my head at people that say things like this. Clearly, they are not used to treating people - let alone experts in the field - with respect. Downgrading a keynote that you explicitly invited the speaker to give (and prior to this they weren't even going to speak at your conference) is extraordinarily disrespectful. I asked around to see if anyone had experienced this or even heard of it happening. No one had.</p> Decisions are made with context, never in a vacuum</h2> I also felt the weight of the context of the decision. JeanHeyd isn't just a recent grant recipient of the Rust Foundation. JeanHeyd has important history with the Rust project.</p> It was JeanHeyd who called Rust out for having no Black representation among Rust conference speakers. Rightly so, as both the Rust organization and the conferences had little to no Black representation.</p> When I saw an organization that not only could act so coldly to an expert in the field, but also to one who was a vocal critic of Rust's lack of diversity, it was hard not to see the additional context.</p> Systems have memory and biases. If the people that make up the system don't work to fight against these, they are perpetuated.</p> As my buddy Aman pointed out, the context that this would have also been the first keynote by a person of color at RustConf should not be lost here.</p> </p> It's time for accountability</h2> I read back over how I handled the situation as it unfolded, and I could have done much better. My go-to tool is diplomacy, trying to build bridges, getting information, and finding compromises. But, after reading over what happened and how I acted, it's clear part of the problem is too much</em> diplomacy in leadership. Too many people giving each other the benefit of the doubt. Too many people asking for answers and then being satisfied to let bad decisions slide.</p> So let me do my part to fix my failings:</p> What we need isn't diplomacy, it's accountability</em>. We need people to take responsibility for what happened and to make amends. We need people to step back from leadership who abused it to perform cruel acts against an expert in the wider programming language community. We need an organization that does not act cruelly so that the project it creates and the organization of people behind it can rebuild the trust it has lost.</p> Questions that need to be answered</h2> Rust, as an organization, is guilty of cruelty. It needs to be held accountable. Here are the first questions that come to mind that need to be answered:</p> How was it possible for a decision to be made without a leadership vote?</li> Why was downgrading the talk the only path explored?</li> Why did RustConf leadership go along with this decision and not protect the speaker? Why wasn't Rust leadership notified of the time period in which to change the decision?</li> Who ultimately is responsible for turning some team discomfort into disgracing a public expert? How are we going to hold them accountable?</li> How do we put safeguards in place so that this doesn't happen again?</li> How can Rust not only be held accountable now, but maintain long-term accountability into the future?</li> </ul> Making a keyboard layout 2023-02-22T00:00:00+00:00 It all started innocently enough. I had gotten interested in more efficient typing at first by learning about different keyboards and what they offered. I was intrigued. Except there was one small problem: fancy keyboards are expensive, especially to import into New Zealand.</p> So, I took a different route: alternate keyboard layouts</a>.</p> These promised to help you type better than the standard qwerty layout. I dove in. I tried colemak, workman, and some of their variants. It was clear with practice you could type better with less hand strain. A normal person at this point might have picked something and stuck with it. Well okay, most people wouldn't have even got that far. But I digress.</p> I on the other hand thought "you know what would be cool? Making a layout for myself!"</p> And down the rabbit hole I went.</p> Why do a new layout?</h2> It started from a simple idea that came from playing guitar and later made its way to professional Tetris competition: rolling</a>. The motion you might have used as a kid when you were bored, drumming your fingers on the table. As the fingers fall one after another, you can create high speeds.</p> The trick seemed simple enough. Lay out the keyboard to try to get the maximum benefit from rolling without making the typing that doesn't roll harder than before.</p> While it's still too early to declare victory, I'll tell you that the text you are reading now was written using the layout I describe below.</p> Creating the layout</h2> I created a few programs with a relatively simple goal. I want to be able to generate layouts that matched a set of constraints. Once I had one, I'd try it out, change the parameters. Rinse and repeat.</p> In the end, I had gone through hundreds of layouts.</p> I learned a few things.</p> You need to find a corpus of text close to what you type</h3> Look through the data to make sure it will fit your writing style and that common words are covered.</p> Be careful about using text with high counts of an uncommon word</h3> When I first started solving for a good keyboard, I used the source for Nushell, a codebase I've spent a lot of time in. This was great in theory, but in practice the algorithm tried to fit words like "value" and "span", pushing other words out of the way.</p> Your hands might help you find what works for you</h3> I would try out an idea and often revise after seeing how that idea worked in practice. I tried many shapes to roll and over time simplified it down to a set that felt I could hit with high enough accuracy.</p> It is easy to over-fit your data</h3> Scoring the different moves ended up being a bit of an art. Setting scores high enough, but not too high, took many iterations of trial and error.</p> Show us the layout already</h2> </p> Inas keyboard layout</em></p> Here's the Inas layout (named by the home keys on the left hand).</p> Let's talk about how I scored typing movements, and then talk about some ways we might improve on it.</p> Scoring</h2> I broke my corpus of text up into words, then found 1-, 2-, and 3-letter runs of letters. These runs would then make up the "roll" that would get higher points.</p> I used the following scoring:</p> Typing on the home row is preferred</li> Center row keys are to be avoided (I learned from guitar that the lift of the finger can be easier on it than lateral stretches)</li> Runs of two should be together where possible</li> Jumps between the top row and bottom row should be avoided</li> Runs of three can fall in a row or can be two in a row and then fall by one row (in the code I call this a j-roll)</li> Try to use the pinkies less (unless necessary for the above)</li> </ul> If you look at the keyboard above you can see quite a few patterns that fall out from that:</p> Common three letter runs like "ing", "you", "the", and "can" are easier to type.</li> Longer words that build on runs are also easier: "there", "this", "think", "thing", "your", "than", "care", "other", "another", and so on. Mixing and matching runs while you type ends up being a fun puzzle.</li> Surprisingly, some common words like "and" don't run. I think this comes from how important "ing" is in English. This takes precedence over the "n" key for rolls.</li> </ul> But is it good?</h2> I've been working with this keyboard layout from the perspective of potential speed. It's still too soon to know how fast you can get with it.</p> There are also other considerations. Namely, does this help your hands? Layouts like qwerty over time can hurt your hands. Does Inas help at all with it? The jury is still out. One good iteration on what I have so far would be to score movements that are more ergonomic higher. Inas has some of that on a basic level by preferring smaller movements and preferring the home row. Are there more ergonomic patters to include? Undoubtedly.</p> If you give it a try, let me know how it goes for you.</p> Build your own</h2> If you want to improve what I have or build your own keyboard you can try it out for yourself</a></p> YouTube addiction, one month sober 2023-02-13T00:00:00+00:00 Today marks my first month clean from YouTube</a>. It was the right choice, but that's not to say it was easy.</p> Let's talk about it.</p> Quitting</h2> When I decided to do something about my addiction, I had a few choices:</p> Cut back the amount I was watching</li> Quit cold turkey</li> </ul> They both have their pluses and minuses, but by the time I figured out what was going I was done</em> with being hooked on it. I was ashamed of getting caught up in it so badly. I chose going cold turkey.</p> Would I choose that path again? Good question, let's explore it.</p> Cold turkey</h2> It was pretty soon after I decided to quit that I blocked YouTube on all my devices. For most things, that meant finding the /etc/hosts</code> file or equivalent for the OS, and adding a line to set youtube.com</code> to 127.0.0.1</code>. That effectively shuts you out of YouTube the site.</p> On my phone, I used AppBlock</code> to block the website. I would have preferred to edit /etc/hosts</code> like I did with my machines, but unfortunately that part of the filesystem is read-only, and can only be modified if you jail-break your phone. App it is.</p> Once I blocked it everywhere I could, I tested it a few times to be sure it was working. Satisfied, I settled in to see what would change.</p> Changes</h2> A few things changed right away. Now that I wasn't obliterating myself for 8-10 hours a day, I started noticing a lot of things. One was that on an average day prior to quitting YouTube, I consumed a lot</em> of sugar. Multiple desserts, plus sugary snacks in between meals, that kind of thing.</p> Before, when I was watching hours of YouTube, I'm guessing I would get up and find something to snack on and sit back down. Now, I don't have that kind of "permanently distracted" frame of mind, I notice that I eat fewer snacks and cut way back on the sugar.</p> I also had time to walk more - something I've always enjoyed. I started doing a 5km (3.1 mi) walk every morning. It takes me almost exactly an hour, and it's a nice way to start the day.</p> But the real changes started happening with my mental health.</p> Mental health</h2> One of the things that made me try to find out what was wrong with me was my declining mental health. Part of me wanted to become a shut-in and lose myself, and another part of me wanted to fight this. I felt worse as the weeks passed.</p> Once I stopped YouTube, I knew I needed a few things: I needed to understand why</em> I got addicted in the first place, and I needed to work on healthier habits.</p> This is still very early on for me, but I wanted to mention a few things I did:</p> I talked to my doctor, who put me in for addiction counselling services. Here in New Zealand, you can get five free visits with an addiction counsellor as you begin your recovery.</p> I put in for the services, but I didn't know how long it would take to get into the system, so I also looked around locally for a therapist who could also give some support. This was a private therapist, so there are costs associated with it.</p> The contrast between the two could not have been more stark. The therapist was very gentle and sympathetic. The counsellor... well, let's say they have a specific task at hand.</p> Why are you addicted and what can you do better?</h2> Addiction counselling, as best as I can figure, has a different goal that general therapy. They're trying to hold up a mirror so you can see something very important for you to see: you are not taking care of yourself.</p> More than that - your brain, in your current state, is not your friend.</p> These are some hard truths to swallow.</p> You work on habits you can change: how to make small changes each day, how to think about rewards and how you've been making bad assessments of what rewards are for and what you should do with them.</p> Did I mention that's all in the first half hour?</p> Then we started talking about the "why", in the same way of holding up a mirror and talking through ways of working with this stuff. I won't share the personal details here, but we covered a decent bit of ground. In the end, as best as I could understand, is that while you may have past experiences that caused your instincts to protect you, they're just that. We all try to protect ourselves from pain and danger. This is a good thing, unless it goes too far and we lose the ability to take normal daily risks and live a healthy life.</p> In a sense, the goal of building new habits isn't to only have new habits. It's to start rewiring the brain to process risks, pain, and danger in a healthier way. There will always be risks, like crossing a street, and we re-learn how to work with them.</p> To some folks, that might sound like total common sense. If it does for you, that's great. For me, it got surprisingly easy to create a safe illusion that turned into an unhealthy way to cope. I needed to see my way out of that.</p> Other changes</h2> After talking with my doctor, I also started cooking more. I'm not a particular good cook (actually I'm pretty bad, though getting better).</p> The act of cooking and eating your own food, though, is therapeutic. You can see how healthy (or not) the food you make is. You know the ingredients that go into it. You can make enough for leftovers.</p> By the time I knew I needed to quit YouTube last month, I'd given up on cooking. I might make a pizza here or there, but almost all my lunches and dinners came from delivery food. Now, this is reversed. I'll still get the occasional delivery, but almost all of what I eat these days I make.</p> Cravings</h2> You may wonder how the actual quitting part is going. The short answer is that I have good days and bad days.</p> It's surprisingly easy some days to fill a day with fun things to do, so that I don't even notice it passing. Other days, I might do some work on the computer and then lay in the floor thinking about stuff I want to watch, scheming ways I could justifying watching some</em> YouTube. That voice that says "watch a few hours to scratch the itch".</p> This, I know, would be a bad idea(tm).</p> Content as an addiction</h2> In talking with lots of folks, at this point I'm convinced we've got content wrong. At least, how it's delivered to us.</p> You can take most of them and understand the business model as "retention = profits". The more time you spend on the site, the longer your eyeballs are taking in the ads and related, the more money the site makes off of you.</p> The way sites like TikTok and YouTube give you a feed of content means you can consume it in an unbroken fashion. It reminds me of the people you see at slot machines in casinos, pulling the lever to see what will happen. Maybe you'll get lucky and come across something awesome.</p> The content also keeps you in a kind of trance - a state where you don't have</em> to think if you don't want to.</p> I feel like one of those people complaining about how televisions make us zombies.</p> I chatted with my addiction counsellor about about social media platforms, and I'm sure you're not at all surprised that they're also addictive.</p> I don't want this post to be just</em> about the dangers of these platforms. There's more to it than I can really cover here, and those platforms can be used for good as well. The main thing, for me, is how they interact with our brains. Are we still able to live healthy lives? Form deep healthy relationships? Take daily risks of being our complete selves? Are our brains staying healthy?</p> Reaching out</h2> Probably the most important thing, though, is to directly target what I needed most of all. Addiction is isolating. The particular addiction, YouTube, is a lot of wanting to connect to others in a very safe way. So, to combat this, the goal is to "reach out". To connect to real people. To get my 'cup' filled with real intimacy and connections with others.</p> What in the world that means is something I'm trying to figure out. I know that my cup was very, very empty. Despite having so many friends, fun online interactions, game nights, etc. Those things are great, but they don't meet the need for real connection.</p> I'm signing up for meetups, volunteering, and going and chatting with random people where I can to get face-to-face interactions again.</p> Step at a time</h2> That's it. That's where I'm at. Figuring out what I can, making changes where I can, and learning as I go.</p> To all the folks who reached out with their stories of struggles and recovery: thank you. Sincerely, thank you. The suggestions and words of encouragement have helped. Thanks for that.</p> Here's to the hope my cup, and yours, will be filled.</p> YouTube Addiction 2023-01-13T00:00:00+00:00 Hi, I'm a YouTube addict.</p> I know what you're thinking. That sounds like a joke. That you want a funny punchline to go with it. Me too.</p> First time I noticed</h2> Truth is, though, I've had a problem for a while. I first noticed it back in 2011. Back then, I could feel it growing. I would spend something like 3 hours at night on weekdays and maybe around 10 hours total on the weekends on YouTube. I was mostly watching "let's plays". Long, winding playthroughs of video games with a friendly voice. Each series might, in total, be something like 40+ hours of content. Of which, the most prolific Let's Play-ers have dozens, if not hundreds, of series. From one creator, you'd find another, and another. Already, back in 2011 watching my little niche, it was more content than anyone one person would ever be able to watch in a lifetime.</p> I would nap watching a series, and wake up to that same familiar voice. Episode after episode, I watched myself go deeper into the rabbit hole.</p> At the time, I told my roommate that I thought I might have a problem. He thought the entire concept of being addicted to something like YouTube was laughable. How could someone be addicted to watching some videos on a website? Maybe some of you reading this are thinking the same.</p> Still, it felt like the right thing at the time to try to curb this growing habit. I bought a stack of books on things like meditation and self-help and read everything I could. It helped a little. Meditation helped me to see just how much stress I was taking on. At the time, my roommate and I were interning at Apple. The pressure was high to do well and get noticed.</p> I made a promise to myself as I got into self-help and meditation that I would stop using YouTube as a way to soothe this stress. That I could find other ways.</p> I remember when I slipped back into it. My favorite YouTube content creator - after not posting for over a year - had posted the start of a new series. By this point, I'd been off YouTube for almost a year. I was used to doing other things with my time. And yet, when they posted that video, and I got a notification, I didn't say no. Since then, I don't think I've ever taken more than a couple months break from it. When I slip back into it, it effortlessly returns to being part of my daily routine.</p> What do you mean by addicted?</h2> Yesterday, I watched 572 minutes of YouTube. That's just over 9.5 hours. The day before it was 444 minutes. Just shy of 7.5 hours. The day before that was 510 minutes. 8.5 hours.</p> Sometimes it's background sound to whatever I'm doing. But if I'm honest, most of the time I'm just watching, going from video to video. Not doing anything else. At this point, I have so many interests I'll watch content for. Climbing videos, computing history, endless amounts of video game footage, music videos, concert footage, interviews, wordless videos of food being prepared in the morning in Japanese restaurants. Anything that could catch my eye, I'd watch. It's gotten to the point where I don't really enjoy the video as much as the numb hypnotic state from consuming that much content.</p> It's safe. It's always there.</p> But I noticed something recently. I've been struggling with my mental health during the summertime, a time I traditionally have fewer problems. I wasn't recharging like I should. I was waking up tired, napping during the day, and generally not feeling at all prepared when the day started.</p> I'd tried to dial back on my commitments to see if I could recover. Weeks later, and I'm still not recovering.</p> It didn't really hit me until today, when talking to a close friend, that it might be the amount of YouTube I'm watching. That's when I took my calculator and added it up.</p> A familiar story</h2> I've heard this before. I had an alcoholic roommate years ago describe in detail what it was like for them to be an alcoholic. Why they would reach for it. When they'd reach for it. How they'd plan their life around it. Get the work done and use it to celebrate.</p> I feel like my use of YouTube has basically mirrored their alcoholism at this point. If anything, I spend more time in stupor than they did with alcohol.</p> The truth is, I don't know how long I've been this bad. I keep clearing my YouTube history. Ostensibly, it's so I can get better recommendations. But the truth is probably that I'm also trying to hide how bad it's gotten from myself.</p> My roommate, when they finally came out about their struggles, let us clean their bedroom. I just remember duffle bag after duffle bag of empty bottles. It took so long to clean out their room because there were so many. I couldn't fathom how anyone could drink so much. It had to be hundreds of bottles.</p> I don't know how long I've been averaging over 8 hours a day, but I'm afraid it's been a long time.</p> I remember telling myself years ago that as long as I was getting my work done it didn't really matter what I did.</p> I remember my roommate saying the same thing.</p> Quitting</h2> I always thought that it's easy to quit something. You just don't do it. So naive.</p> This isn't going to be easy. There are so many temptations to pull me back in that it'll be a challenge to try to put this behind me. I mean, I'm a YouTube content creator now approaching 10k subs - a respectable number for a small channel. So much educational content is on YouTube. So much of how I get my name out is via YouTube. So much of what I do is attached to YouTube for good and for bad.</p> It's really all of the above. It's true it has all these nice aspects. But for folks like me, who take it to an extreme, it very politely chews its way into your life and makes it far too tempting to sink deeper and deeper.</p> I don't want to lose the audience I'm building, but I also want to be healthy. I'm afraid for myself and where I'll be five years from now. Will I be worse than I am now if I don't take steps? Will I become more withdrawn, stop working altogether, and live on my savings until I run out?</p> I think about other things, too, like "How will I fill my days if I stop"? I honestly don't know. It's scary to think about having a hole and not being able to fill it with that comfortable "zoning out" feeling.</p> I'm sure some folks are thinking "what if you just limit yourself to 30 minutes a day?"</p> That's just it. If I could do that, I would have.</p> Admitting it</h2> I don't know if I want to do a 12-step program or whatever. I honestly haven't really figured anything out at this point. But the first thing I wanted to do was come clean and talk about where I'm at. If you're someone reading this with similar patterns, you're not alone.</p> I also wanted to write this all down to make a record of where I am. Maybe I'll come back and read this later. Maybe I'll be able to say something like "I remember being hooked on YouTube for over 12 years. I'm glad I cleaned up."</p> Let's hope. Maybe then I'll think up a good punchline, too.</p> The goal is to nourish 2022-12-23T00:00:00+00:00 I just finished watching the movie called Stutz, a movie created by Jonah Hill about his psychiatrist. Through the movie, Stutz gives bite-sized practices drawn from Buddhism, psychiatry, and his own experiments. One of the themes is to accept that connection is the goal (connection with others, ourselves, and our body) and we connect using, among other things, vulnerability.</p> For years, I've thought about a conversation I had with a friend of mine. He had worked with David Schnarch, a well-known psychology that wrote books like Passionate Marriage. In my friends work with David, they worked towards an end goal where yes, we should recover from our pain and trauma, and that will be work we continue with, but that we also have a choice to put light back out into the world.</p> I knew as I talked to my friend about this, that this was unshakeably the goal. I knew that I could put my fears and trauma out into the world, but I also knew that without the light, it wouldn't end up helping myself or others. Not in the sense of sugar-coating my experiences, but rather that there is a light within us that is part of who we are, and that we can share our experiences and</em> share our light at the same time. We don't have to hide our light for others to see our pain. We can show up as ourselves.</p> Showing up, being vulnerable, being willing to fall on my face has always been hard for me. I wanted nothing more than to the gifted kid that never made errors. I wanted people to look up to me. To see me as some kind of spiritual giant. Or great intellect. To do nearly super-human feats.</p> It wasn't until I got older that I started to see those desires as part of wanting to hide. If I do super-human feats, perhaps people will be distracted from the fact that I'm still a very human human.</p> In recent years, I've started to slowly understand how intimacy and vulnerable don't have to come with rejection. Sure, someone may respond poorly to what I'm sharing, which hurts my feelings and makes me want to hide. But I'm also there in the room as I'm sharing. I'm also able to be a light for myself in that moment. When you carry your own light, now others can have their experiences, too, and you can hold space for them.</p> Maybe it's from playing too much Stardew Valley, but I just get this mental image of waking up every morning and watering the crops. It becomes so routine you almost don't think about it.</p> It teaches a subtle lesson. That if you nourish other beings a little each day, you grow as a result. I don't mean giving when you have nothing to give, draining the last bit of blood from your body. I mean just like you breathe oxygen, exercise, and try to eat well, giving a little to others is part of a healthy life.</p> This morning I struggled with derealisation. It's an uncomfortable mental state where everything feels unreal. You may feel like you're watching yourself or your surroundings, but rather than you being the focus, it feels like you're watching a movie. I didn't panic. I just told myself that in the coming hours that it may go away and that I should give myself time to check in a little later and see if it's improved a little. By afternoon, it had lifted.</p> I don't recount that as a kind of "wow, I'm great" moment. Instead, I share it because it's interesting to contrast what it was like 10 years. Months of gray weather in Seattle was too much for me, and I would go through weeks of on and off dealing with feeling like I was in a movie. It drained me. I panicked, worried, and felt like I didn't really have any grounding.</p> While I think part of being able to deal with it comes with practice, I also think there's another important component. Feeling more connected, feeling like you can still live your life and use what you have to connect to other people (even in small ways) meant I didn't have to solve the problem right then or risk losing any hope of love and connection in my life. The love and connection are there, whether or not I could see and feel them. And they'd be waiting for me. And that comes from the regular nourishing of those connections, both with myself and with others.</p> I'm not entirely sure how best to end this post. It's not like there's a magic formula that if you follow it you never hurt and never feel alone. I just wanted to share that having moments of appreciation with other people - even during difficult times - has been an important piece of the journey for me.</p> Autism is everywhere and in everything 2022-07-13T00:00:00+00:00 I've been thinking recently about being on the Autistic spectrum.</p> But first, some trigger warnings: topics include some talk of sex and gender, emotional pain, burnout.</p> I thought it was about sexuality, or gender, or...</h2> I've wanted to talk about this for a long time. Before coming out as non-binary, I was diagnosed in 2020 as being on the autistic spectrum. It hadn't even crossed my mind until a close friend recommended I look into it.</p> As I sat reading my diagnosis I just kinda started to unzip the first outer coat of many coats I'd learned to wear for the first 40 years of my life. I listened to podcasts, read articles, listened to other folks, and just tried to understand anything. Even more fundamental than sexuality, or understanding gender, was this idea that I had be fighting extremely hard to look and appear normal to the world.</p> I was exhausted.</p> Exhausted of opening up to people only to have them leave when they couldn't handle the intensity that comes naturally when I let my guard down. Instead of punishing myself, though, I had a key to understand something fundamental about myself. The way I viewed the world and how the world interacted with me as an experience was profoundly different than it was for most other people.</p> With each thought, I felt like I stripped off another outer coat and set it aside, until I could no longer understand how to communicate with people anymore. I still knew the tricks I'd learned. I still know how to "mask", to use patterns I'd seen other people use to appear more normal. But what I wanted to know was "what was I like when I didn't do that?". In essence, who was I?</p> When I told my ex, who I'd recently separated from, that I had been diagnosed she laughed at me. Rather than getting angry, it confirmed for me what I'd already known - there's just a deep-seated difference in the needs I have and the needs she had. In hindsight, it also confirmed another sad aspect of being on the spectrum: you can have a very large tolerance for people having bad expectations of you because you learn at an early age that you must accommodate others in order to survive. You have no basis for what a good boundary is. Perhaps just as bad, in my case I've found it exceedingly difficult to feel when the other person loves me, respects me, or otherwise. So I kept working harder in the hopes that at some point in the future something will feel right and things would click.</p> For months after my diagnosis, I worked backwards. I wanted to reach an earlier version of myself to see how long I'd been this way. You'll probably be unsurprised to learn I've always been this way. I remembered countless events that now made so much more sense. Times of overstimulation, panic, being unable to understand why my parents were punishing me, feeling completely lost as to why the kids at the new school did things and why they singled me out.</p> In time, as I worked backward, I began to ask questions like "how does sexuality relate to autism?" and "how does gender relate to autism?". Though the research here is fairly new, I came across terms like "neuroqueer" that talked about how autism can "queer" your own perception of your gender and sexuality.</p> Wait. What?</p> I tested it on myself. Was that what was happening? The more I let myself just experience my inner world and not require myself to use images and concepts that I learned from outside, the more I saw sexuality and gender as basically formless goo. A kind nothing concept that carried value for others, but didn't seem to hold much importance to me.</p> When I say I'm non-binary, I mean...</h2> ...gender doesn't make sense to me as an internal experience. I wear the clothes I wear because I own them and they fit. I'm not thinking about what I'm wearing or how I'm presenting myself outside of the house unless my clothes are stained and dirty.</p> Gender, to me, feels like those clothes.</p> What it is like</h2> Sex is nice, but it has nothing on the bliss of a hyperfocus. Losing track of time, where hours are there merely to serve as a dimensional tube to channel more of your topic into you. People are lovely and should be a part of life, but it's sad that many folks will never understand what it's like.</p> But this coin is so easily two-sided. In the computer industry, many a company has noticed that as long as they keep people like me "entertained with hard problems" we're happy. That is, until we burnout because no one reminded us to take care of ourselves.</p> What it's like for me:</p> Imagine not knowing if your partner likes you or if they're just tolerating you. Imagine this going on for years.</li> Imagine being in a tumble dry cycle of ups and downs and feeling like you come out of every social situation worse than you started.</li> Imagine letting yourself enjoy a topic you find fascinating and the joy feels nearly limitless as you dig deeper. Imagine this lasting for days, weeks, months, or even years.</li> Imagine being told you're weird most of your life and finding the only way to feel "not weird" is to completely ignore yourself.</li> Imagine not having the ability to socially read the person in front of you until you've spent many hours with them, or they remind you of someone you knew 10 years ago and you hope that they're similar enough they'll react to the same things.</li> </ul> If this sounds like you, perhaps you're like me. If so, welcome to the club.</p> Autism club</h2> In autism club, we embrace what we are and who we are. There are snacks, heavy blankets, endless libraries, friends to do projects with, calm lighting, soft sounds, and a universal welcome to any and all parts of you. You don't have to mask, and we won't ask you to. Autism club is welcome to all ages. It is our hope that young and old from all walks of life will join this club and bring who they are to it.</p> We make this club for ourselves.</p> Here you can feel your feelings in the way they come out, as intense as they are or even the mildest whisper. You don't have to cut them off. You don't have to ignore the urge to self-stimulate to help direct the energies you feel.</p> Carrying the club with you</h2> I've been trying to learn how to carry autism club with me, wherever I go. Not only so I can be welcoming to others, but I can also be welcoming to myself. The term self-care feels like it almost gets overused these days, but it's still just as important. Checking in and understanding your needs -- even the ones you were taught to ignore -- opens the door to giving yourself space.</p> I, for one, need that space. And I hope as I grow older, perhaps I'll also grow more capable of being a better support to myself and others. Perhaps I'll also find the right people who can be that to me in return.</p> And then?</h2> As I write this, I don't have the perfect thought to end on. Just this idea that everything is ever changing, and that's us too. There isn't an endpoint to this journey because what it means to think and feel and experience the world through the way we process information is so central. I carried it with me into the start of this post. And, as I finish, I'll carry it with me from here.</p> How we judge things 2022-06-23T00:00:00+00:00 I've been thinking recently...</p> The good, the bad, and the average</h2> Do you ever wonder about how we judge things? By this I mean, let's say we go to a restaurant with a friend. The meal is going well enough, then towards the middle of the meal we try our friend's dish. Wham! It's amazing!</p> How do we judge the restaurant? Do we say it's just okay because that's how we felt about our meal? Or do we say it's amazing because of the bite we had of our friend's meal?</p> Say we go watch a movie. The movie is pretty good, but there's this 20 minute section in the last third that just feels like it goes forever. Was the movie good, since we enjoyed the majority of it? Or do we feel the movie was a big let-down because of the drag at its slowest point?</p> Do we judge something by its best moment, its worst moment, or by its average moment? There's also the question to ask - do we judge something by our experience or by the experience of those around us? - but I'm more interested in our subjective experience for now.</p> Noticing how we do this for various forms of entertainment, we can also turn our eyes toward how we judge people. People we've just met. People we're friends with. Relationships we're in and how people act in those relationships. Do we judge people by their best moment? Worst moment? Or average moment?</p> It's not that these are set in stone. For example, in social media perhaps we follow people who we feel on average are entertaining. Maybe some tweets don't makes us laugh, but enough do that we stick around. Let's say something comes out of their past that casts a shadow over them. We may shift from judging their average moment to now judging them by their worst.</p> What I've wondered for myself is this - do I know what style of judgement I'm choosing? And, is that the appropriate kind of judgement for this situation?</p> Imagine you have a friend that's generally fun, but sometimes is cruel. If you judge them by their best moments, you might get a sense of them "having such potential" because you see the high points. If you judge them by their average, perhaps you seem them as "a fun person, with a few quirks". If, instead, you judge them by their low moments, you may think "you know, they actually can be pretty mean".</p> It seems like, in a way, all of these are true at once. It's us, as the judge, who get to say how much weight we put on each type of judgement.</p> Redemption</h2> Stories give us another way to see how we view things as they tend to exaggerate some aspect of the human existence. One aspect of some stories is the "redemption arc". Can we see this in the context of the judgement from earlier?</p> If we judge Darth Vader, we might get three different answers:</p> By their best moment: a hero! By their average: a pretty bad person By their worst: a monster</p> Stories like to tug on our emotions by playing against these different feelings, but what about how this translates to how we interact with people in the real world. Are we able to judge correctly if someone has redeemed themselves?</p> Person or situation</h2> I was recently listening to a podcast, and the speaker mentioned a bit of advice they had gotten: if you are leading a team, do not be afraid to fire people. If working with them is taking energy, and that energy isn't making the product or you stronger, cut them off.</p> It felt so cold, and yet, there's something decisive about it. They're looking at what is happening and judging not just the person but the situation as a whole. Are things going in the right direction? Even if someone is a star engineer on the team, if the situation shows that their average is that the whole team drags as a result of their actions, then perhaps that judgement is more important than their star contributions?</p> A personal note</h2> I was raised to give people second chances. To be forgiving. That forgiveness was part of what you did. In hindsight, this seems to be asking us to focus only on the best moments, or perhaps to focus on the average at the expense of the bad. Instead, we should respect all of our judgements of both people and situations we find ourselves in. To weigh them all, so that we understand them, and then act based on all we see without letting ourselves be pre-disposed to any one style.</p> Scripting languages of the future 2021-10-31T00:00:00+00:00 I've been thinking recently about what the future for scripting languages could look like. We've had a good run with Python, JavaScript, and Ruby for the last few decades. I wondered, what might the next few decades look like.</p> What is the scripting language of the future? How should it work? Here are some thoughts.</p> Scalability</h2> Working on TypeScript</a>, something that became a constant refrain for us was that "TypeScript is JavaScript that scales".</p> I might be a bit biased, but this is</em> actually incredibly valuable. If your team invests time into writing an app in a scripting language, there shouldn't be an impedance mismatch that grows as the app gets larger. For JavaScript, we'd watch teams grow their apps only to have to increase testing and onboarding training to prevent bugs from creeping in. And yet, they still crept in.</p> TypeScript gave tools a chances to catch these errors earlier, even in well-tested codebases</a>. Back in 2012, this was something viewed with a healthy dose of skepticism, and today over third of professional developers use TypeScript</a>.</p> What TypeScript showed is that you could join together the idea of a flexible lightweight (and optional!) type system onto an existing programming language, and do so successfully.</p> The question then is - what if you created a programming language from the start to have this kind of support? No need for transpilers, while also having stronger engine integration and maybe even better runtime errors?</p> It's a tempting mix of possibilities and one well worth being a part of the scripting language of the future.</p> Tune-ability</h2> I've known multiple people who were hired to rewrite a piece of infrastructure originally written in a scripting language. Just like scripts need to be able to scale to larger sizes, scripts need to be tune-able. The choice to go dynamic shouldn't be a hard stop on real performance tuning and maintenance.</p> The work on numpy, scipy, pandas and others for Python is one possible blueprint. Instead of having to leave your language of choice, powerful libraries build up around it to support the high performance needs of different disciplines.</p> There will likely be some resistance to this point, as there seems to be just an inevitable trade-off between performance and the flexibility that a scripting language provides. In all honesty, this feels like an older, and perhaps outdated, way of looking at this trade-off.</p> For years, JavaScript developers have looked to techniques like monomorphism</a> to tune the performance of JavaScript code. They're structuring code in a way that's more "VM friendly". The VM has to do less work to understand the types and structures in the code, and as a result guesses correctly the first time when doing runtime code generation.</p> The ability to tune code can come from other sources, too. Type decorations is one tempting direction to mention, though so far no established languages (that I know of) have used the technique to general type hints to speed up code. Unless type hints are used across hot paths, and with a sound type system, the transformations risks actually slowing performance.</p> There is, however, a large exception. Dataframe systems like pandas structure operations across arrays of known types. While the setup code is all in Python, the code that actually runs is happily in systems-level tuned native libraries. In a way, Python is merely orchestrating these operations.</p> Future languages could take advantage of this pattern in other ways. The associated standard libraries of these scripting languages could encourage more orchestration and less manual processing in the language itself.</p> Easy parallelism</h2> On a related point to the one above: future languages need easy parallelism. By now, we've heard it a million times. The "free lunch is over", tomorrow's CPUs will have a zillion cores and will run at 1hz. Okay, maybe not quite that bad, but still the writing is very clearly on the wall and very much already happening. It's not far-fetched to imagine that today's 8-16 core machines over the next few decades grow to be dozens or even hundreds of cores. Any language that hopes to have a strong footing, even scripting languages, need to be able to handle this.</p> This isn't without its own challenges. As has been quotes a few times recently, Matz - the creator of Ruby - famously said "I regret adding threads"</a>. Having now used a lot of different kinds of parallelism systems in a lot of languages, I can easily see why. Traditional threading, and things like it, honestly feel like goto to me. Just like goto, it's easy to write code that's hard to read and, over time, hard to maintain.</p> Luckily, traditional threads aren't the only game in town. Work-stealing task pools</a> like popular libraries in Rust use are very fast and can be built to work with other abstractions like parallel iteration and reduction operations.</p> Making abstractions like this part of the language gives more opportunity to use more of the machine across all scripts written in that language. All this without, we hope, running afoul of the design choices that make language creators mourn years later.</p> IDE support</h2> This might sound completely alien to some folks, but the future for scripting languages includes strong IDE support. Gone are the days of most scripting being written in a simple text editor. Today, more than 86% of JavaScript is developed in an IDE</a>.</p> Future languages are competing against established languages which have tooling and IDE support polished over many years. Any new language that wants to compete against that will have to swing for the fences.</p> At a minimum, support for the language server protocol</a> these days in a must. Its broad support across editors gives languages much more leverage than ever before to get in front of developers.</p> Looking ahead</h2> There's definitely a large space to explore for scripting languages of the future. The points I list above are important, as are additional points that we'll learn as we go. Perhaps we'll be able to bring the performance that more modern JS engines enjoy to a broader range of languages through an effort similar to what LLVM did for lower-level languages.</p> I, for one, am looking forward to this future and what future languages it will bring.</p> Let's talk about exit codes 2021-10-16T00:00:00+00:00 Before starting a shell</a>, if you would have asked me what an exit code was, I would have quickly replied "0 for success, non-zero for failure" without batting an eye. That's what it meant, right?</p> Today, someone filed an issue on the Nushell repo</a>. In it, they ran git log</code>, exited, and Nushell reported the command to have failed.</p> To know why it happens, we've got to do some digging. First, let's take a look at the standards to understand what we should expect.</p> What the standard says</h1> Bash</h2> One of the first places you'll learn about exit codes is the Bash manual. It says:</p> "A successful command returns a 0, while an unsuccessful one returns a non-zero value that usually may be interpreted as an error code. Well-behaved UNIX commands, programs, and utilities return a 0 exit code upon successful completion, though there are some exceptions</em>." (emph. mine)</p> POSIX</h2> The POSIX Open Group Base standard says:</p> "The value of status may be 0, EXIT_SUCCESS, EXIT_FAILURE, [CX] [Option Start] or any other value, though only the least significant 8 bits (that is, status & 0377) shall be available from wait() and waitpid(); the full value shall be available from waitid() and in the siginfo_t passed to a signal handler for SIGCHLD. [Option End]"</p> On the same page it says: "The functionality described on this reference page is aligned with the ISO C standard. Any conflict between the requirements described here and the ISO C standard is unintentional. This volume of POSIX.1-2017 defers to the ISO C standard."</p> ISO C</h2> Okay, so what's the ISO C standard say?</p> "If the value of status is zero or EXIT_SUCCESS, an implementation-defined form of the status successful termination is returned. If the value of status is EXIT_FAILURE, an implementation-defined form of the status unsuccessful termination is returned. Otherwise the status returned is implementation-defined."</p> "an implementation-defined form of the status unsuccessful termination is returned" feels unsatisfying, but at least we know that non-zero means unsuccessful, right?</p> The bug</h1> The original report was about git log</code>. Let's run the steps the user reports: run git log</code>, quit with q</code>, and then check the exit code. I'll do this in bash, since that's the shell everyone will have access to:</p> > git log > echo $? 141 </code></pre> Sure enough, a non-zero exit code.</p> Bash is reporting that git log</code> exits with exit code 141. This clearly feels intentional. It's not a 0 or 1, but a number specific to the other 254 possibilities.</p> Turns out that some folks dug into this 5+ years ago</a>. They write a bullet list with their findings:</p> Git pipes the logs into less</li> I quit the less process, which sends a SIGPIPE signal (13) to the underlying git process streaming the logs</li> git catches the interrupt and exits prematurely and per POSIX convention returns 128 + the SIGPIPE status ==> 141 to indicate that it was terminated by signal 13.</li> </ul> When failures aren't failures</h1> Why is the user being told about the SIGPIPE in this case? I'm sure there's a perfectly reasonable set of events that take us to this point. Git is calling out to less</code>, and less</code> wants to report the user quitting it ends the pipe prematurely. And yet, it definitely feels to fall short of the spirit of the specifications</em>. Non-zero errors should</em> mean we've exited in an unexpected way. The steps above exit us in about the only reasonable way to exit interactively viewing the log.</p> As it turns out, git log</code> isn't alone. Here's a discussion on Stack Overflow about grep -q</code> returning 141</a>. In it, the responses point out yet other commands, like head</code>, also return 141 in their regular course of action.</p> There are standards, and there are standards</h1> Something folks tend to assume is that the POSIX standard covers a lot of how Unix-like systems work. While this is true at some level, there are cases like this one where we quickly shift to having to lean more on conventions than standards. As the Bash manual says "though there are some exceptions". Using pipes and command composition leads to having to follow one part of the standard that in turn leads to really confusing situations for the user, like the situations listed above. A command like head</code> or less</code> doesn't take all of its inputs. As it closes, it passes along that "hey, technically the pipe I was reading from isn't going to be able to write to its output because I'm closing". From the user's perspective, though, exiting early is exactly what's supposed to happen. It's not actually an error.</p> It kinda makes you wonder - if we could do this over again, how would we do it differently?</p> Retiring 2021-03-03T00:00:00+00:00 Yesterday was my last day at Microsoft. I've been fortunate to have worked in an industry that paid well over my 20+ year career and to have grown up with frugal parents who passed that on to me. Starting the 4th of March, I'll be retiring to work on open source and teaching.</p> So what's next? I have a few projects in mind:</p> Get Nushell to 1.0</h2> We're currently working on the plans for what it will take to get Nushell ready for 1.0. For the last year and a half, Nushell has been quite the passion project for me, and being at the place where we can talk about 1.0 and how to get there has me excited.</p> We'll publish these plans once they're ready so we can get your feedback, but suffice to say there's lots of cool things we'd like to do.</p> Finish the Rust beginner series</h2> I've been enjoying working on the Rust beginner series</a>, where I work through the concepts you'll need as a beginner in Rust. It's more of an overview of the language and how language features fit together. I'm having fun with it, and hearing some good feedback, which is always nice.</p> Work on the Rust intermediate series</h2> I recently asked on Twitter</a> about intermediate Rust content, which inspired me to start an intermediate Rust series. You can find out more information on this series in this blog post</a>.</p> Keep doing Systems with JT</h2> I've been having a blast making videos about various OSes</a>. What started as a "it'd be fun to make some videos on these two hobby OSes" has turned into a weekly video, and I'd like to keep this going.</p> How can we help?</h1> Probably the best thing is encouragement! Say hi on Twitter</a> or come by the YouTube channel</a> or Twitch stream</a>.</p> If you have the funds and don't mind chipping in, I have Patreon</a>, too. This will help me upgrade hardware, improve video editing, and more.</p> If you'd specifically like to help Nushell</a> hit 1.0, I'd be happy to show you around. We've got a discord</a> where you can join us and chat about what's happening and where to jump in. There are lots of places to help with code, docs, designs, and more. We'll also soon be posting the proposal for getting to 1.0, which will include lots of areas that you could help with.</p> Conclusion</h1> I'm both nervous and excited. Though I've planned as best I can and I live as simply as I can, it feels like taking a leap into the big blue unknown. But I'm all for following the heart and seeing what happens next.</p> Intermediate Rust series 2021-02-12T00:00:00+00:00 Hi all!</p> A few days ago I posted on Twitter</a> asking for topics that people wanted to see covered for intermediate Rust content. The response was awesome and it's inspired me to make a series where I go through each topic and do my best to cover it as a video and/or blog post.</p> Here's the current list of topics. If you see something missing, you can drop an issue on this repo</a> or ping me on Twitter</a> and I'll try to add it.</p> Rust features</h2> Associated types</li> Working with lifetimes</li> Pin</code> and Pin<Box<Stream>></code></li> Macros, proc macros, custom attributes</li> Concurrency</li> Async/await</li> Threads/tasks</li> MPSC theory</li> Futures</li> Send/Sync</li> Unsafe</li> FFI, shared mem across FFI, closures/function pointers over FFI</li> FFI to C and C++</li> C interop, opaque structures, repl(c), working with null, alloc/free, bitflag export to C</li> What tools to use for C++</li> std::sync</code> (Arc, Rc, RefCell, etc)</li> Raw pointers</li> Impl trait, dyn trait, and trait objects</li> Fn traits</li> Field sensitivity, field sensitivity + smart ptrs, field sensitivity and closures</li> Why do we cast from Box<Struct></code> to Box<dyn Trait></code></li> How to use [patch]</code> in Cargo.toml</li> std::mem</code>, swap, replace, take</li> </ul> Crates</h2> Tonic</li> Tokio</li> Deeper dive into existing frameworks/libraries</li> Channels/async/par_iter (concurrency vs parallelism)</li> </ul> Projects</h2> Basic SQL database</li> Interpreters and parsers</li> Tokio + io_uring for CLI apps</li> Filesystem + FUSE</li> Seda or actor based project</li> Wasm + wasm bindgen</li> </ul> Patterns and practices</h2> Like http://railstutorial.org but for Rust</li> Functional composition</li> Async functional composition</li> Dependency injection (and alternatives)</li> Design patterns</li> Lessons from Rust in production</li> How to teach Rust to teams</li> Testing patterns and strategies</li> https://rust-lang.github.io/api-guidelines/checklist.html</li> Zero copy processing best practices</li> Error handling. One big error enum or something extensible? Anyhow?</li> Debugging Rust</li> Modelling classic data structures in Rust</li> Logging</li> Shared data structures in long-running apps</li> Profiling</li> Improving compile times</li> </ul> Rust background</h2> How do nightly features stabilize?</li> </ul> Looking ahead</h1> As you can see, there's a lot</em> of area to cover. My plan is to make steady progress on the list above, covering topic by topic until we cover as many of the topics as possible. People were also upvoting tweets with likes, so I may juggle this list around so that I'm sure to cover the hot topics sooner.</p> Excited to dig into this!</p> If you'd like to follow along to enjoy the content as it comes out, you can follow me on Twitter</a>, you can follow my YouTube channel</a> where I post my technical content, and follow me on Twitch</a> for when I stream a topic.</p> Nushell 0.5.0 2019-11-05T00:00:00+00:00 Nushell, or Nu for short, is a new shell that takes a modern, structured approach to your commandline. It works seamlessly with the data from your filesystem, operating system, and a growing number of file formats to make it easy to build powerful commandline pipelines.</p> Today, we're happy to announce the 0.5.0 release for Nu. We've got lots of new features, including some long-requested ones, in this release.</p> Where to get it</h1> Nu 0.5.0 is available as pre-built binaries</a> or from crates.io</a>. If you have Rust installed you can install it using cargo +beta install nu</code> (or if you want all the features cargo +beta install nu --all-features</code>).</p> Nu as a login shell (sophiajt)</h1> One of the key features that landed in 0.5.0 is set of new capabilities that work together to allow you to use Nu as a login shell, completely independent of bash or other hosting shells. For this, we've built in support for querying and updating the environment variables and the path.</p> To get started, you'll need to first copy the environment you're using into the config. Luckily, we've also made some improvements there:</p> > config --set [path $nu:path] > config --set [env $nu:env] </code></pre> Once these values are set, you'll be able to use Nu as your login shell.</p> New variables</h2> As you saw above, we've added a few new built-in variables. These will let you know what the current values that Nu can see are for your environment, path, and the config itself.</p> > echo $nu:env > echo $nu:path > echo $nu:config </code></pre> Adding paths to your PATH</h2> One of the first things you'll notice is that the new $nu:path is structured. If you run the echo above, you might see something like this:</p> > echo $nu:path ━━━┯━━━━━━━━━━━━━━━━━━ # │ <value> ───┼────────────────── 0 │ /usr/local/sbin 1 │ /usr/local/bin 2 │ /usr/sbin 3 │ /usr/bin 4 │ /sbin 5 │ /bin 6 │ /usr/games 7 │ /usr/local/games 8 │ /snap/bin ━━━┷━━━━━━━━━━━━━━━━━━ </code></pre> We've added two new commands: prepend</code> for adding items to the start of a table and append</code> for adding items to the end of a table. With these commands, we can now query out the path, update it, and save it back.</p> > echo $nu:path \| prepend "/my/new/directory" \| config --set_into path </code></pre> Adding variables to your environment</h2> You can use a similar set of steps to add new variables, or change existing variables, in your environment.</p> > echo $nu:env \| insert GREETING hello_world \| config --set_into env </code></pre> Note: the previous add</code> command of previous releases has been renamed insert</code> to remove confusion with mathematical functions.</em></p> On-going improvements</h1> We're continuing to improve the commands we currently ship as part of Nu. Here are a few you might find helpful:</p> Substrings (Flare576)</h2> The str</code> command now supports being able to retrieve a substring from the strings given, so you could return, for example, the first 5 characters and stop after that.</p> > ls \| get name ━━━━┯━━━━━━━━━━━━━━━━━━━━ # │ <value> ────┼──────────────────── 0 │ target 1 │ CODE_OF_CONDUCT.md 2 │ .cargo 3 │ src 4 │ features.toml 5 │ rustfmt.toml </code></pre> > ls \| get name \| str --substring "0,3" ━━━━┯━━━━━━━━━ # │ <value> ────┼───────── 0 │ tar 1 │ COD 2 │ .ca 3 │ src 4 │ fea 5 │ rus </code></pre> Recycling (jdvr)</h2> Ever wish you could rm</code> things, but not forever? You can now tell rm</code> to send items to your platform's recycle bin rather than deleting them forever. As with our other commands, this works across all the platforms that Nu supports.</p> > rm myfile.txt --trash </code></pre> Parameter descriptions (sophiajt)</h2> We're also continuing to improve the built-in help system. New in this release are descriptions for the flags and parameters that the command uses. For example, here's a look at what the help for rm</code> now looks like:</p> > help rm Remove a file Usage: > rm <path> {flags} parameters: <path> the file path to remove flags: --trash: use the platform's recycle bin instead of permanently deleting --recursive: delete subdirectories recursively </code></pre> New commands</h1> In addition to the new append</code> and prepend</code> we mentioned earlier, we've added a few new commands to Nu.</p> Average (notryanb)</h2> Growing our set of mathematics functions for working with numbers, we now have an average</code> command which will take the average of values given to it.</p> > ls \| get size \| average </code></pre> Read (sophiajt)</h2> We've also introduced a new command to load in strings as tables. This new read</code> command will take a pattern that describes the columns, their names, and where they are in each row of the string.</p> > open .editorconfig root = true [] indent_style = space indent_size = 4 charset = utf-8 trim_trailing_whitespace = true insert_final_newline = false end_of_line = lf </code></pre> > open .editorconfig \| read "{variable} = {value}" ━━━┯━━━━━━━━━━━━━━━━━━━━━━━━━━┯━━━━━━━ # │ variable │ value ───┼──────────────────────────┼─────── 0 │ root │ true 1 │ indent_style │ space 2 │ indent_size │ 4 3 │ charset │ utf-8 4 │ trim_trailing_whitespace │ true 5 │ insert_final_newline │ false 6 │ end_of_line │ lf </code></pre> Bugfixes (sophiajt, JesterOrNot, thegedge, andrasio, wycats, notryanb, Detegr, t-hart)</h1> As always, we've had lots of bugfixes. A huge</em> "thank you!" to folks who reported issues, fixed issues, and just generally shared their experience with the shell. It's much appreciated and helps to continue making Nu that much better.</p> Survey</h1> If you haven't already taken it, we'd love to hear your feedback in a quick (roughly 3 question) survey</a>.</p> Looking forward</h1> There are a bunch of areas we're currently working on to make Nu more stable and feature complete. Until this work is finished, please consider Nu to be somewhere in the pre-alpha</em> to alpha</em> quality level.</p> That said, we're excited to merge this work and continue to take steps towards a more full-featured shell. Soon to be coming will be Nu working on stable Rust(!!), some improvements in the Nu internal engine which will make it possible to have better streaming and cleaner commands, and features like aliases.</p> Nushell 0.4.0 2019-10-15T00:00:00+00:00 Nushell, or Nu for short, is a new shell that takes a modern, structured approach to your commandline. It works seamlessly with the data from your filesystem, operating system, and a growing number of file formats to make it easy to build powerful commandline pipelines.</p> Today we're happy to announce the 0.4.0 release of Nushell. The 0.4.0 release</a> marks a continually maturing shell that is now starting to show signs of stability. And, of course, a few fun features along the way.</p> Where to get it</h1> Nu 0.4.0 is available as pre-built binaries</a> or from crates.io</a>. If you have Rust installed you can install it using cargo +beta install nu</code> (or if you want all the features cargo +beta install nu --all-features</code>).</p> What's new</h1> New Colors (wycats)</h2> New colors!</p> With 0.4.0, we're adding some new colors to show off different types of the command, where errors might be happening, and if the command is internal or external. The coloring is just the tip of the iceberg, as it were, and builds from a reworking of the parser to make it more accurate, more stable, and more feature-complete.</p> Streaming table (sophiajt)</h2> Streaming tables</p> Up to this point, we've taken a few shortcuts with how streams worked in Nushell. We knew that a table, in theory, was a stream of rows. In practice, though, this wasn't the case as the code has lots of assumptions about how tables worked.</p> With the 0.4.0, we're taking a big step in the direction of full support for streams. You'll see that now data will start being printed as it becomes available, with tables printing pages of 50 rows at a time (by default). This allows you to get deep listings of files (eg using ls </code>), work with streams from external sources, and more.</p> Better ctrl-c support (sophiajt, landaire)</h2> As part of having streams of tables, it's always helpful to be able to say "stop!" when you need to. With 0.4.0, using ctrl-c to stop the output should be a lot more responsive and will take you back to the prompt.</p> Textview gets VI keys (JonnyWalker81)</h2> The built-in text viewer now lets you use vi keys for navigation thanks to JonnyWalker81's work.</p> Onto the Beta compiler (est31)</h2> For those of you wondering if and when Nu was going to be usable on stable Rust, you need but wait just a little longer. Contributer est31 has been hard at work at moving Nu onto stable Rust and has now successfully moved Nu off of nightly and onto beta. As beta matures and becomes the next release of Rust, we'll be able to move Nu onto stable. This should help people who are working to distribute Nu on various distributions.</p> Brazilian Portuguese book (marcelocg)</h2> Contributor marcelocg surprised us with a translation of the Nu book into Brazillian Portuguese. Obrigado, Marcelo!</p> New commands</h2> from-ssv (t-hart)</h3> To follow along with our from-csv</code> (comma-separated-values) and from-tsv</code> (tab-separated-values), we now have a new way of handle importing content from commands that output their own tables: from-ssv</code>. Like its siblings, it looks for values separated by a separator, using the spaces to show where the breaks between cells should be.</p> match (rnxpyke)</h3> We're continuing to evolve our support for matching string data. Recently, we began experimenting with a match</code> command that can find matches in strings for you. We plan to continue experimenting and exploring what's possible in this space in the future.</p> Lots of new documentation (sdfnz, twe4ked, chhetripradeep, mfarberbrodsky, cristicismas, Charles-Schleich, coolshaurya, notryanb, pema99, gilesv, yahsinhuangtw, JesterOrNot, nalshihabi, marcelocg, sorrell, pizzafox, iggy14750, DrSensor, mlbright, BradyBromley, andrasio, piotrek-szczygiel)</h2> Happy hacktoberfest! Lots of people wanted to jump in and help document all of our commands, help fix docs that are already there, and generally give things a bit of polish. Much appreciated everyone who came by and contributed.</p> Tons of stability fixes (thegedge, sophiajt, wycats, JonnyWalker81, vsoch, pka)</h2> As always, there were a bunch of stability and correctness both in Nu and in the CI that support Nu's development</p> Looking ahead</h1> We'll continuing filling out the features that will let folks use Nu as their everyday shell. Coming up are better support for the environment, path, and for command aliases.</p> Nushell 0.3.0 2019-09-24T00:00:00+00:00 Nushell, or Nu for short, is a new shell that takes a modern, structured approach to your commandline. It works seamlessly with the data from your filesystem, operating system, and a growing number of file formats to make it easy to build powerful commandline pipelines.</p> We're happy to announce that today we're releasing Nushell 0.3.0. Nu has seen numerous bugfixes, performance improvements, and features added since its initial public release (which was only a few weeks ago!)</p> Where to get it</h1> Nu 0.3.0 is available as pre-built binaries</a> or from crates.io</a>. If you have Rust installed you can install it using cargo +nightly install nu</code> (or if you want all the features cargo +nightly install nu --all-features</code>).</p> What's new</h1> New table design (Porges, sophiajt)</h2> </p> One of the most striking differences is that Nu now uses UTF-8 box drawing to draw its tables.</p> </p> The table is also configurable. If you feel like the table is a bit too heavy, you can also configure it to work in light mode using this command:</p> > config --set [table_mode light] </code></pre> New file formats (pmeredit, est31, andrasio, sophiajt)</h2> Nu now natively supports new file formats, including: bson, tsv, sqlite, and url-encoded strings.</p> New commands (incrop, ramonsnir, ijt, sophiajt, andrasio, JonnyWalker81, chhetripradeep, pka)</h2> Since the 0.2.0 release, Nu has gained a set of additional commands, including:</p> help</code> - built-in help system</li> reverse</code> - reverse a table</li> last</code> - like the first</code> command, but working from the bottom of the table, return n rows</li> embed</code> - create a new table using the current table as a start</li> fetch</code> - get the contents of a URL (this originally lived in open</code>)</li> post</code> - post to a URL and get the results</li> pwd</code> - print the working directory</li> env</code> - give access to a number of important paths and environment settings</li> pivot</code> - pivot a table so that rows become columns and vice-versa</li> echo</code> - a built-in echo command</li> </ul> Error improvements (sophiajt)</h2> </p> We've made steady progress on improving error messages. Recently, we added "did you mean?" errors to help when you mistype the name of columns. Nu 0.3.0 also has gone through a few passes to generally polish the errors to include more information when an error occurs.</p> Shell features (pka, wycats, twe4ked, andrasio, iamcodemaker)</h2> Nu now supports ~</code> to refer to your home directory as part of a path. You can also use cd -</code> to return to a previous directory you were working in.</p> The shell itself also got a few updates. You can now enable 'vi' mode, if you'd prefer vi-like bindings (by running config --set [edit_mode vi]</code>). On non-Windows platforms, you can also use Ctrl-R to enable Sublime-style history searching (we're hoping to enable Windows support for this soon).</p> Plugin improvements (sophiajt, andrasio)</h2> Plugins have continued to mature, and it's now possible to use plugins to extend Nu with support for new file formats.</p> We've also added documentation on how to write Nu plugins in both Rust and Python</a>, with info on Ruby coming soon.</p> Docker support (vsoch, DrSensor)</h2> If you're interested in using Nu and Docker together, be sure to check out the new Docker support</a>.</p> Book progress</h2> The Nu book</a> is now available in three</em> languages: English, Spanish, and Japanese.</p> We've also created a Nu contributor book</a>, which will help developers who want to contribute to Nu learn about its philosophy, design, and how to create plugins.</p> Acknowledgements</h2> est31, pmeredit, twe4ked, DrSensor, vsoch, pka, jankoprowski, Porges, max-sixty, ijt, djc, vthriller, taiki-e, oskarskog, iamcodemaker, JonnyWalker81, yaahc, tim77, svartalf, ramonsnir, orf, lesichkovm, ineol, incrop, eoinkelly, devnought, chhetripradeep, aidanharris, GuillaumeGomez, and BatmanAod for contributing to the Nu codebase. Also a big</em> thanks to our issue reporters!</p> ymgyt - for the Japanese version of the Nu book!</p> mistydemeo, romanlevin, ralvessa, pka, lord, kkalyan, jankprowski, and boisgera for helping with the book.</p> Looking ahead</h2> We're hard at work on the upcoming milestone release</a>. If you'd like to help, feel free to jump in! We have an active discord</a> you can join for both users and contributors.</p> Introducing nushell 2019-08-23T00:00:00+00:00 Today, we're introducing a new shell, written in Rust. It draws inspiration from the classic Unix philosophy of pipelines, the structured data approach of PowerShell, functional programming, systems programming, and more.</p> It's called Nushell, or just Nu for short. We have a book</a> (¡también se habla Español</a>!). We have a repo</a>.</p> This release was made by Sophia Turner (me), Yehuda Katz, and Andrés Robalino, with contributions from Odin Dutton.</p> </p> But why?</h1> Many of us have gotten used to bash (or zsh/fish/etc), and don't understand why you would need another kind of shell. That was me, too, a few months ago before I started working on this. My friend Yehuda had discovered PowerShell and was going on and on about how amazing it was to do more with the shell, but until he actually gave me a demo, I didn't really believe him.</p> Then he talked me into joining him on an idea he had. What if we could take the ideas of a structured shell and make it more functional (as opposed to object-oriented)? What if, like PowerShell, it worked on Windows, Linux, and macOS? What if it had great error messages? I fell in love with the project ideas, made a few new friends, and many nights and weekends later I'd like to show you what we've made.</p> In this post, I'll talk about how a few simple ideas drive how Nu works, what Nu can do with them, and where we hope to go in the future.</p> Simple ideas</h1> To Nu, everything is data. When you type ls</code>, you're given a table of information about the directory you're listing:</p> </p> Rather than having to remember different flags to ls</code>, we can just work with the data it gives back. We can find the files greater than a certain size:</p> </p> Or we could choose to sort it by a column, or only show directories, or more. That by itself is fun but perhaps not compelling enough.</p> </p> Where this simple concept - that everything in Nu is data - starts to shine when we try other commands and realize that we're using the same commands to filter, to sort, etc. Rather than having the need to remember all the parameters to all the commands, we can just use the same verbs to act over our data, regardless of where the data came from. Nu pushes this idea even further.</p> </p> Nu also understands structured text files like JSON, TOML, YAML, and more. Opening these files gives us the same tables we saw with ls</code> and ps</code>. Again, this lets us use the same commands to filter our data, explore it, and use it.</p> Working with the outside world</h1> The above approach could be fun, but if we're not careful, it could become a walled garden. What happens outside of the commands Nu comes with?</p> First, let's take a look at working with a file that Nu doesn't understand.</p> > open people.psv Octavia \| Butler \| Writer Bob \| Ross \| Painter Antonio \| Vivaldi \| Composer </code></pre> To work with this in Nu, we need to do two steps: figure out where the rows are, and then figure out what the columns are. The rows are pretty easy, we just have one record per row:</p> > open people.psv \| lines ---+------------------------------ # \| value ---+------------------------------ 0 \| Octavia \| Butler \| Writer 1 \| Bob \| Ross \| Painter 2 \| Antonio \| Vivaldi \| Composer ---+------------------------------ </code></pre> Next, we can create our columns by splitting each row at the pipe (\|</code>) symbol:</p> > open people.psv \| lines \| split-column "\|" ---+----------+-----------+----------- # \| Column1 \| Column2 \| Column3 ---+----------+-----------+----------- 0 \| Octavia \| Butler \| Writer 1 \| Bob \| Ross \| Painter 2 \| Antonio \| Vivaldi \| Composer ---+----------+-----------+----------- </code></pre> That's already good enough that we can work with the data. We can go a step further and name the columns if we want:</p> > open people.psv \| lines \| split-column " \| " firstname lastname job ---+-----------+----------+---------- # \| firstname \| lastname \| job ---+-----------+----------+---------- 0 \| Octavia \| Butler \| Writer 1 \| Bob \| Ross \| Painter 2 \| Antonio \| Vivaldi \| Composer ---+-----------+----------+---------- </code></pre> But what about working with commands outside of Nu? Let's first call the native version of ls</code> instead of the Nu version:</p> > ^ls assets Cargo.lock docs images Makefile.toml README.md rustfmt2.toml src tests Cargo2.toml Cargo.toml extra LICENSE open readonly.txt rustfmt.toml target </code></pre> We'll use the same commands we used on data to bring it into Nu:</p> ^ls \| split-row " " file ----+--------------- # \| value ----+--------------- 0 \| assets 1 \| Cargo2.toml 2 \| Cargo.lock 3 \| Cargo.toml 4 \| docs 5 \| extra 6 \| images 7 \| LICENSE 8 \| Makefile.toml 9 \| open 10 \| README.md 11 \| readonly.txt 12 \| rustfmt2.toml 13 \| rustfmt.toml 14 \| src 15 \| target 16 \| tests ----+--------------- </code></pre> Or maybe we want to work with the native ls -la</code>:</p> ^ls -la \| lines \| split-column " " ----+------------+---------+----------+----------+---------+---------+---------+---------+--------------- # \| Column1 \| Column2 \| Column3 \| Column4 \| Column5 \| Column6 \| Column7 \| Column8 \| Column9 ----+------------+---------+----------+----------+---------+---------+---------+---------+--------------- 0 \| total \| 296 \| \| \| \| \| \| \| 1 \| drwxr-xr-x \| 13 \| sophia \| sophia \| 4096 \| Aug \| 24 \| 03:24 \| . 2 \| drwxr-xr-x \| 21 \| sophia \| sophia \| 4096 \| Aug \| 22 \| 17:00 \| .. 3 \| drwxr-xr-x \| 2 \| sophia \| sophia \| 4096 \| Aug \| 3 \| 05:39 \| assets 4 \| drwxr-xr-x \| 2 \| sophia \| sophia \| 4096 \| Aug \| 21 \| 19:29 \| .azure 5 \| drwxr-xr-x \| 2 \| sophia \| sophia \| 4096 \| Jun \| 23 \| 05:09 \| .cargo 6 \| -rw-r--r-- \| 1 \| sophia \| sophia \| 2963 \| Aug \| 22 \| 20:17 \| Cargo2.toml 7 \| -rw-r--r-- \| 1 \| sophia \| sophia \| 201255 \| Aug \| 24 \| 03:24 \| Cargo.lock 8 \| -rw-r--r-- \| 1 \| sophia \| sophia \| 3127 \| Aug \| 24 \| 03:24 \| Cargo.toml 9 \| drwxr-xr-x \| 2 \| sophia \| sophia \| 4096 \| Jun \| 17 \| 15:32 \| docs 10 \| -rw-r--r-- \| 1 \| sophia \| sophia \| 148 \| Jun \| 17 \| 15:32 \| .editorconfig 11 \| drwxr-xr-x \| 4 \| sophia \| sophia \| 4096 \| Aug \| 22 \| 19:29 \| extra 12 \| drwxr-xr-x \| 8 \| sophia \| sophia \| 4096 \| Aug \| 24 \| 03:24 \| .git 13 \| -rw-r--r-- \| 1 \| sophia \| sophia \| 58 \| Aug \| 10 \| 11:08 \| .gitignore 14 \| drwxr-xr-x \| 2 \| sophia \| sophia \| 4096 \| Aug \| 24 \| 03:24 \| images 15 \| -rw-r--r-- \| 1 \| sophia \| sophia \| 1085 \| Jun \| 17 \| 15:32 \| LICENSE 16 \| -rw-r--r-- \| 1 \| sophia \| sophia \| 614 \| Jun \| 17 \| 15:32 \| Makefile.toml 17 \| -rw-r--r-- \| 1 \| sophia \| sophia \| 0 \| Aug \| 23 \| 04:58 \| open 18 \| -rw-r--r-- \| 1 \| sophia \| sophia \| 11375 \| Aug \| 24 \| 03:24 \| README.md 19 \| -r--r--r-- \| 1 \| sophia \| sophia \| 0 \| Jul \| 4 \| 03:51 \| readonly.txt 20 \| -rw-r--r-- \| 1 \| sophia \| sophia \| 37 \| Aug \| 23 \| 04:54 \| rustfmt2.toml 21 \| -rw-r--r-- \| 1 \| sophia \| sophia \| 16 \| Aug \| 1 \| 19:45 \| rustfmt.toml 22 \| drwxr-xr-x \| 10 \| sophia \| sophia \| 4096 \| Aug \| 24 \| 03:24 \| src 23 \| drwxr-xr-x \| 4 \| sophia \| sophia \| 4096 \| Aug \| 22 \| 19:22 \| target 24 \| drwxr-xr-x \| 4 \| sophia \| sophia \| 4096 \| Aug \| 22 \| 04:15 \| tests 25 \| drwxrwxr-x \| 2 \| sophia \| sophia \| 4096 \| Jul \| 19 \| 15:18 \| .vscode ----+------------+---------+----------+----------+---------+---------+---------+---------+--------------- </code></pre> After a bit of experimenting, we might come up with a command like this:</p> > ^ls -la \| lines \| skip 1 \| split-column " " perms files group user size month day time name ----+------------+-------+----------+----------+--------+-------+-----+-------+--------------- # \| perms \| files \| group \| user \| size \| month \| day \| time \| name ----+------------+-------+----------+----------+--------+-------+-----+-------+--------------- 0 \| drwxr-xr-x \| 13 \| sophia \| sophia \| 4096 \| Aug \| 24 \| 03:24 \| . 1 \| drwxr-xr-x \| 21 \| sophia \| sophia \| 4096 \| Aug \| 22 \| 17:00 \| .. 2 \| drwxr-xr-x \| 2 \| sophia \| sophia \| 4096 \| Aug \| 3 \| 05:39 \| assets 3 \| drwxr-xr-x \| 2 \| sophia \| sophia \| 4096 \| Aug \| 21 \| 19:29 \| .azure 4 \| drwxr-xr-x \| 2 \| sophia \| sophia \| 4096 \| Jun \| 23 \| 05:09 \| .cargo 5 \| -rw-r--r-- \| 1 \| sophia \| sophia \| 2963 \| Aug \| 22 \| 20:17 \| Cargo2.toml 6 \| -rw-r--r-- \| 1 \| sophia \| sophia \| 201255 \| Aug \| 24 \| 03:24 \| Cargo.lock 7 \| -rw-r--r-- \| 1 \| sophia \| sophia \| 3127 \| Aug \| 24 \| 03:24 \| Cargo.toml 8 \| drwxr-xr-x \| 2 \| sophia \| sophia \| 4096 \| Jun \| 17 \| 15:32 \| docs 9 \| -rw-r--r-- \| 1 \| sophia \| sophia \| 148 \| Jun \| 17 \| 15:32 \| .editorconfig 10 \| drwxr-xr-x \| 4 \| sophia \| sophia \| 4096 \| Aug \| 22 \| 19:29 \| extra 11 \| drwxr-xr-x \| 8 \| sophia \| sophia \| 4096 \| Aug \| 24 \| 03:24 \| .git 12 \| -rw-r--r-- \| 1 \| sophia \| sophia \| 58 \| Aug \| 10 \| 11:08 \| .gitignore 13 \| drwxr-xr-x \| 2 \| sophia \| sophia \| 4096 \| Aug \| 24 \| 03:24 \| images 14 \| -rw-r--r-- \| 1 \| sophia \| sophia \| 1085 \| Jun \| 17 \| 15:32 \| LICENSE 15 \| -rw-r--r-- \| 1 \| sophia \| sophia \| 614 \| Jun \| 17 \| 15:32 \| Makefile.toml 16 \| -rw-r--r-- \| 1 \| sophia \| sophia \| 0 \| Aug \| 23 \| 04:58 \| open 17 \| -rw-r--r-- \| 1 \| sophia \| sophia \| 11375 \| Aug \| 24 \| 03:24 \| README.md 18 \| -r--r--r-- \| 1 \| sophia \| sophia \| 0 \| Jul \| 4 \| 03:51 \| readonly.txt 19 \| -rw-r--r-- \| 1 \| sophia \| sophia \| 37 \| Aug \| 23 \| 04:54 \| rustfmt2.toml 20 \| -rw-r--r-- \| 1 \| sophia \| sophia \| 16 \| Aug \| 1 \| 19:45 \| rustfmt.toml 21 \| drwxr-xr-x \| 10 \| sophia \| sophia \| 4096 \| Aug \| 24 \| 03:24 \| src 22 \| drwxr-xr-x \| 4 \| sophia \| sophia \| 4096 \| Aug \| 22 \| 19:22 \| target 23 \| drwxr-xr-x \| 4 \| sophia \| sophia \| 4096 \| Aug \| 22 \| 04:15 \| tests 24 \| drwxrwxr-x \| 2 \| sophia \| sophia \| 4096 \| Jul \| 19 \| 15:18 \| .vscode ----+------------+-------+----------+----------+--------+-------+-----+-------+--------------- </code></pre> Because Nu let's you manipulate your data until it's how you want it, there's a feeling of playing with your data. You get used to using the verbs, and then you can use them on anything. When you're ready, you can write it back to disk.</p> Oh, before I forget - I wanted to quickly show how to get data from Nu back out into the outside world. Here's an example of calling echo</code> on each filename in a directory:</p> > ls \| get name \| echo $it </code></pre> You can see that we can mix-and-match commands that are inside of Nu with those that are outside, and data will still flow between them as expected. But Nu is more than just a pipeline.</p> More than a pipeline</h1> As we built Nu, we realized we could experiment with other parts of how a shell works. The first of these experiments lead us to an observation: if everything is data in Nu, we should be able to view this data.</p> </p> We've seen the tables. Nu also supports opening and looking at text and binary data. If we open a source file, we can scroll around in a syntax-highlighted file. If we open an xml, we can look at its data. We can even open a binary file and look at what's inside (hint: there's even a fun easter egg if you open certain kinds binary files, especially if you've installed Nu with the optional rawkey</code> feature).</p> Being able to view data is helpful, and this kind of polish extends to other aspects, like error messages:</p> </p> Nu takes heavy inspiration from the error messages in Rust</a>. As much as possible, draw your eyes to the problem.</p> Combined with the pipeline, some pretty interesting errors are possible:</p> </p> You might wonder how does that even work. Nu has a metadata system (still early!) that you can read about in the Metadata chapter</a> of the Nu book</a>. Let's just take a quick peek at it:</p> > open Cargo.toml ------------+--------------+------------------+----------+---------- bin \| dependencies \| dev-dependencies \| lib \| package ------------+--------------+------------------+----------+---------- [11 items] \| [object] \| [object] \| [object] \| [object] ------------+--------------+------------------+----------+---------- > open Cargo.toml \| tags ----------+------------------------------------------ span \| origin ----------+------------------------------------------ [object] \| /home/sophia/Source/nushell/Cargo.toml ----------+------------------------------------------ </code></pre> Data that flows through the pipeline gets a set of additional metadata tagged to it. We can use this later to figure out how to display the contents, show a better error message, and more.</p> Shells, plural</h1> Let's say you're in a directory, but you'd really like to flip back and forth between it and one or two others. You could open up multiple tabs, multiple terminals, if you're on a Unix system you could use "screen", and probably even more than that. What if the shells were just built in?</p> In Nu, we can enter</code> a directory, which adds it to a ring of shells we can bounce between:</p> > enter ../rhai/ /home/sophia/Source/rhai(master)> shells ---+---+------------+------------------------------- # \| \| name \| path ---+---+------------+------------------------------- 0 \| \| filesystem \| /home/sophia/Source/nushell 1 \| X \| filesystem \| /home/sophia/Source/rhai ---+---+------------+------------------------------- </code></pre> Using n</code> and p</code> we can jump back and forth between the shells. exit</code> gets us out of a shell.</p> You might noticed that name</code> column in the shells</code> table. Why's that there? Oh no... oh yes.</p> > enter Cargo.toml /> shells ---+---+--------------------------------------------+------------------------------- # \| \| name \| path ---+---+--------------------------------------------+------------------------------- 0 \| \| filesystem \| /home/sophia/Source/nushell 1 \| \| filesystem \| /home/sophia/Source/rhai 2 \| X \| {/home/sophia/Source/nushell/Cargo.toml} \| / ---+---+--------------------------------------------+------------------------------- </code></pre> That's right, we're in the file. Can we cd</code>? Oh yes, we can:</p> /> ls ------------+--------------+------------------+----------+---------- bin \| dependencies \| dev-dependencies \| lib \| package ------------+--------------+------------------+----------+---------- [11 items] \| [object] \| [object] \| [object] \| [object] ------------+--------------+------------------+----------+---------- /> cd bin /bin> ls ----+----------------------+--------------------------- # \| name \| path ----+----------------------+--------------------------- 0 \| nu_plugin_inc \| src/plugins/inc.rs 1 \| nu_plugin_sum \| src/plugins/sum.rs 2 \| nu_plugin_add \| src/plugins/add.rs 3 \| nu_plugin_edit \| src/plugins/edit.rs 4 \| nu_plugin_str \| src/plugins/str.rs 5 \| nu_plugin_skip \| src/plugins/skip.rs 6 \| nu_plugin_sys \| src/plugins/sys.rs 7 \| nu_plugin_tree \| src/plugins/tree.rs 8 \| nu_plugin_binaryview \| src/plugins/binaryview.rs 9 \| nu_plugin_textview \| src/plugins/textview.rs 10 \| nu \| src/main.rs ----+----------------------+--------------------------- </code></pre> Plugins</h1> Nu can't come with everything you might want to do with it, so we're releasing Nu with the ability to extend it with plugins. There's more information in the plugins chapters</a>. Nu will look for these plugins in your path, and load them up on startup.</p> All because of Rust</h1> Nu would not have been possible without Rust. Internally, it uses async/await, async streams, and liberal use of "serde" to manage serializing and deserializing into the common data format and to communicate with plugins.</p> We also heavily leveraged crates.io</a>. The ability to load numerous file formats, display messages, draw tables, and more all came from the hundreds (thousands?) of generous developers who wrote the crates we use in Nu. A huge</strong> thank you to everyone who contributed to Nu without ever knowing it.</p> What's next?</h1> Nu is just getting started. In this release, we have a foundation to build on. Next, we'll work towards stability, the abilty to use Nu as your main shell, the ability to write functions and scripts in Nu, and much more.</p> If you want to give it a spin, the installation instructions</a> will help you get started. If you want to chat come by our discord</a> and github</a></p> Creating crossplatform Rust terminal apps 2019-04-27T00:00:00+00:00 </p> Look Mom, Pikachu running in Windows CMD!</em></p> I've been wanting to play around with the cool spinning Pikachu demo</a> everyone was talking about. Sadly, it used termion to do its magic, which meant that unfortunately it wouldn't work for me.</p> Termion has been a boon for Rust, with lots of folks using it to create terminal applications. Unfortunately, as a Windows user, I know there's a good chance that if the crate depends on termion that's the end of the line for me, as termion apps just don't work in Windows. Surely, I thought, there must be a better way, but I never managed to find one.</p> Enter crossterm</h1> Yesterday, I discovered a crate I have wanted for at least a year: that holy grail of something like termion that also worked crossplatform. The crate is called crossterm</a>, and it is exactly that. Rather than being an exact copy of termion, I'd call Crossterm a termion-like crate. The API surface feels very similar between the two crates, but it's not identical. That said, often, the translation is 1:1 (or close to it).</p> In this post, I'll detail what I had to do to port to crossterm. You can take a look at my repo</a> and give it a spin.</p> Moving to crossterm</h2> Like termion, crossterm is a single crate to depend on that pulls in what you need for interacting with the terminal:</p> -termion = "1" +crossterm = "0.9.3" </code></pre> Next, is setting up the terminal:</p> - let mut stdout = AlternateScreen::from(stdout().into_raw_mode().unwrap()); // Raw output is clean output - let mut stdin = async_stdin().bytes(); // Async in so input isn't blocking + + let crossterm = Crossterm::new(); + #[allow(unused)] + let screen = RawScreen::into_raw_mode(); + let input = crossterm.input(); + let mut stdin = input.read_async(); + let cursor = cursor(); + + cursor.hide()?; </code></pre> There are a few more steps here, and I took a slightly different approach than termion. In the original, the author uses an AlternateScreen, a way to easily switch back to the previous terminal with the application exits. The crossterm package supports this, though there were is an issue</a> that's being worked on. Using an AlternateScreen is optional, so I just left it off.</p> The next step was to set up raw mode. Raw mode lets us get user input directly without the normal terminal buffering. The order of operations in crossterm above is important.</p> Lastly, I hide the cursor so it doesn't mess up the animation.</p> The event loop is similar, though a little more verbose in my version:</p> - let b = stdin.next(); - if let Some(Ok(b'q')) = b { - break; + if let Some(b) = stdin.next() { + match b { + InputEvent::Keyboard(event) => match event { + KeyEvent::Char('q') => break, + _ => {} + }, + _ => {} + } </code></pre> The event systems between termion and crossterm, while similar, are different enough that I preferred to do two-step match to make sure I understood it. Events for both key and mouse come in through the event stream, which is a nice benefit.</p> Drawing colors to the screen is also very similar between termion and crossterm:</p> - format!("{}{}{}{}", color::Fg(color::Rgb(bg.0, bg.1, bg.2)), - color::Bg(color::Rgb(25,25,25)), - src, - color::Fg(color::Reset)) + print!( + "{}{}{}", + Colored::Fg(Color::Rgb { + r: (pixel.1).0, + g: (pixel.1).1, + b: (pixel.1).2 + }),^M + Colored::Bg(Color::Rgb { + r: 25, + g: 25, + b: 25 + }), + pixel.0 + ) </code></pre> The change from color::Fg</code> to Colored::Fg</code> is straightforward enough I did most of these changes with search/replace.</p> Gotchas</h1> The translation is not without its own set of hiccups. The main one of which you get a hint at here:</p> - pub frame_buffer: Vec<String>, + pub frame_buffer: Vec<(char, (u8, u8, u8))>, </code></pre> Originally, the whole framebuffer was turned into a vector of strings. On most Unix platforms, these would include the ANSI commands to change the color or move the cursor. It's a powerful tool, and being text-based, you can prebuffer the whole frame via string manipulation.</p> Unfortunately, assuming everything is string-based doesn't work in Windows. When crossterm outputs using println!, it's calling into the Windows Console API between each piece of the println. This allows the same code to work, assuming it doesn't buffer the string.</p> To work around this, I stopped buffering the string and instead stored the pixel value in the buffer. When it was time, I drew the whole frame using print! statements.</p> When I first did this, it was noticeably slower than the termion version(roughly 5-10 fps). This is because calling into the Console API that often (once per character) is going to pull down performance. Luckily, I could work around this by just checking if we were already using the color I wanted to render. If we were, I didn't set the color again.</p> Performance between the termion version and the crossterm version now look the same to me, which I call good enough :)</p> Conclusion</h1> We have a way to move more of our terminal apps to being crossplatform, so that they work out-of-the-box on Windows as well as Unix-based systems. It was fun to write the port, and I did the port in a couple hours without ever having used crossterm before.</p> Also, big thanks to Mitchell Hynes</a> for writing the original!</p> The Fallow Year, my Rust2019 post 2018-12-08T00:00:00+00:00 Definition:</strong> fallow: (of agricultural land) Ploughed but left unseeded for more than one planting season.</em></p> Rust is an amazing project. It's unlike anything I've ever seen. Not only are we seeing an ever-growing number of big name users of Rust, we also continue to see leaps in productivity and functionality. Yesterday, the Rust 2018 edition shipped</a>. The culmination of three years of work, it shows off what the community is capable of: new features, backwards compatibility, new ergonomics without sacrificing performance, and the list goes on and on.</p> Which is why it might comes as a little bit of a surprise that I suggest we let the field rest for the year. More specifically, that we should let one field rest while we plant another.</p> Rather than focusing on new designs, I suggest we turn our attention outside of RFCs to other areas of Rust.</p> Examples</h1> The compiler</strong>: The compiler team has been understaffed while also being heavily taxed by the effort of keeping up with new designs. I think they've done a stellar job with the resources they have, but they need help -- and lots of it. RFCs have gone unimplemented simply because not enough people are contributing. Compile times need work. The compiler team is generally too busy to implement core functionality that it hoped to get done this year, like compiler-drive IDE completions.</p> Let's clear through the backlog of the growing list of accepted RFCs. Let's take the time to do the deep refactoring that we haven't had time to do because we were stretched too thin. Let's give ourselves space to experiment with new ways of doing things. Let's get the compiler ready to serve both build and IDE scenarios.</p> RLS and related tools</strong>: We've had big success with projects like rustfmt, and it would be great to mirror that success in the Rust Language Server, or RLS, which powers Rust's IDE support. Unfortunately, the current RLS has to rely on many hacks to work with what is available from the compiler. By freeing up the compiler team, we also give space to the dev tools team to collaborate and finish the RLS, removing the need for hacks that cause incomplete functionality and unpredictabilty.</p> The community</strong>: That's right, the community needs a pause too. Everyone loves new features, but we need a bit of time to let all the new ones sink in before we start designing more. What design patterns will arise in practice? What are the best ways to work with async code at scale? On and on. Getting all this experience will let us see more clearly where to design next.</p> Documentation, videos, and other materials:</strong> Along with the growing knowledge the community is getting by more using the current Rust more deeply, we can also translate this knowledge into more documentation. The need is strong for mid-level Rust developer documentation. What are the Rust design patterns? How does one get the most benefit from Rust quickly? How does one translate C++ knowedge to Rust? And so on.</p> Libraries:</strong> We can use the knowledge we have to create new and better libraries. Existing libraries could continue to mature and adopt Rust 2018 idioms. Areas like GUI development need some clear direction and focus.</p> Outreach:</strong> We've worked on some outreach to companies who might be interested in Rust, but there's more we can do here. New whitepapers, new documentation about how to use Rust in specific sub-fields, like finance, data mining, machine learning, etc.</p> The list</h1> As it turns out, these aren't my ideas. They're yours.</p> This year's Rust survey</a> results highlighted these areas of improvement (sorted by number of comments):</p> the need for better library support</li> a more improved IDE experience</li> the need for broader adoption of Rust generally</li> a richer ecosystem of tools and support</li> an improved learning curve</li> the need for important language features and crates to be stable and supported</li> support for async programming</li> support for GUI development</li> better documentation</li> improved compile times</li> </ul> You'll notice that only one of these, "support for async programming", implies adding new functionality to Rust itself. Rather than asking for new features, users wanted better documentation, tools, library support, IDE support, learning materials, and overall stability.</p> The idea</h1> Let's finish what we started. As much as is possible and practical this year, let's set aside new designs and ship what we've already designed. Let's tackle challenges we haven't had time to give our full attention.</p> In short, let's grow Rust where it needs it most.</p> A Snapshot of Rust's Popularity in July 2018 2018-07-28T00:00:00+00:00 Talking about a language's popularity is traditionally a tricky topic. How do you measure popularity? How do you compare one language to another when they're focused on different styles and different audiences? So, rather than having one or two charts, I'm going to look at a number of "slices" into Rust's growth to see it front different angles.</p> There are also many</em> ways we could measure popularity. For example, looking at the growth of StackOverflow, internal forums, Twitter traffic, Google Trends, or the various language popularity reports. Each of these measures has its own strengths and trade-offs.</p> For our purposes in this post, we're going to focus on a set of data points that should help show growth of usage</em> rather than growth of buzz</em> (that is, the amount people talking are about Rust). If people are using a language to do real work, that's the best indicator of a language's success.</p> The first of place we'll look for Rust usage is in open source, and with that, let's look at GitHub.</p> Growth on GitHub</h1> GitHub offers a few ways to measure how well a particular language is growing. GitHub itself uses the number of pull requests in a given language, which it uses in the GitHub Octoverse</a>.</p> If we were to run this query for 2018 so far, the top fifteen languages (by number of PRs) would be:</p> JavaScript: 1736476</li> Python: 804790</li> Java: 703649</li> Ruby: 560430</li> PHP: 359040</li> C++: 319324</li> TypeScript: 311229</li> Go: 258131</li> C#: 246513</li> CSS: 236795</li> Shell: 168301</li> C: 160889</li> Swift: 67664</li> Scala: 67188</li> Rust: 52936</li> </ol> Rust comes in at 15th place as part of the group of Swift/Scala/Rust. This is pretty good company to be in. All of the languages on this list are known for doing heavy lifting in real applications, rather than being hobby or boutique languages.</p> Being 15th is also significant because, if the Octoverse were held today, Rust would make their top languages list for the first time.</p> Moving on, one gotcha of focusing on PRs is that it lets a relatively small number of highly successful projects skew the results. The popularity of the projects drives high numbers of PRs, and as a result the language looks far better. To compensate for this, let's also look at "active projects". For our purposes, let's define "active projects" as the number of projects that a) are larger than a "hello world", b) have at least 1 star, and c) have been worked on in the last 30 days:</p> JavaScript: 56437</li> Python: 37554</li> Java: 27912</li> C++: 16787</li> PHP: 16451</li> Shell: 12245</li> C#: 11503</li> C: 10662</li> Go: 8816</li> CSS: 8781</li> TypeScript: 8447</li> Ruby: 7518</li> Swift: 3828</li> Objective-C: 3054</li> Rust: 2604</li> Kotlin: 2343</li> Scala: 2072</li> </ol> Another respectable showing for Rust. Again, we see it as part of the languages for doing serious work.</p> The GitHub numbers tell one story, but do we see the same story if we look in other places? With this kind of popularity, we'd expect to see visible use of Rust in industry.</p> Rust's Commercial Users</h1> Rust has seen a surprising amount of growth in commercial usage over the past year alone. Here's a partial list of names people may know:</p> Amazon</strong> - Building tools in Rust</a>.</p> Atlassian</strong> (makers of Jira) - Using Rust in the backend</a>.</p> Dropbox</strong> - Using Rust in both the frontend and backend</a>.</p> Facebook</strong> - Tools for source control</a>.</p> Google</strong> - As part of the Fuchsia project</a>.</p> Microsoft</strong> - Using Rust in part of their new Azure IoT work</a>.</p> npm</strong> - Using Rust in some of the npm core services</a>.</p> Red Hat</strong> - Creating a new storage system</a></p> Reddit</strong> - Using Rust in its comment processing</a></p> Twitter</strong> - As part of the build team support for Twitter</a>.</p> You can see even more on the Friends of Rust</a> that include other familiar names like Baidu, Wire, Mozilla, Samsung, Cloudflare, Chef, Canonical, and more.</p> Across the industry, software companies are beginning to adopt Rust at some level in their organization. As expected, these may be tentative at first, focusing on Rust for build tools, for backend processing, or targeting a specific platform like IoT. Early adopters like Dropbox and Canonical have now grown beyond initial investments to using Rust in an increasing amount of their codebase.</p> All told, it's still early days for Rust. Large companies take time to try new technology before they adopt it more broadly. We're seeing Rust in that part of the adoption curve, and it's doing well here. Languages -- like Go and TypeScript -- that saw explosive growth in recent years also saw similar patterns at this part of their lifecycle.</p> Because of Rust's closeness with C++, we should also look at industries that are traditionally strongly populated by C++ developers. One such industry is game development.</p> Zooming into Games</h1> Despite the game industry's use of established development technologies, Rust is beginning to make in-roads with game developers. Here's a list of usage of Rust in some familiar names among game developers:</p> Chucklefish Games</strong> - Using Rust in multiple upcoming games</a>.</p> Electronic Arts</strong> - Using Rust in SEED</a>.</p> Frostbite Engine</strong> - Using Rust in backend processing</a>.</p> Ready at Dawn Studios</strong> - All new development</a> will be in Rust.</p> Unity</strong> - Using Rust in data engineering</a>.</p> We're also seeing other indie developers</a> beginning to use it for their titles.</p> Just as we saw in the "big name" usage, we see a similar pattern with game developers. Initial usage is targeted, perhaps at backend or data-focused projects. After this, there's a path to developing core technology, and even whole games, in Rust.</p> Jobs</h1> When we ran the 2017 Rust survey</a>, one of the big themes that came up for people not using Rust was around their companies not using it. The jobs just weren't there.</p> In 2018, with a growing numbers of companies and studios using Rust, has this situation improved?</p> In short, yes, though there's still a lot of room to grow. A quick look at LinkedIn's job list shows a number of Rust opportunities from the likes of Amazon as well as a number of companies involved in blockchain, security, science/healthcare, and more. Still, when compared to the volume of job offerings for languages like Go or TypeScript, there's still a gap in terms of availability.</p> Let's move on from Rust's use in companies to look at the growth of the community that supports Rust.</p> Community and Governance</h1> In times past, a language could survive on being used solely for commercial development. These days, however, a language's community forms its foundation. It offers a way to ramp up new users, support new projects, and set the tone for the project going forward.</p> Rust meetups now total over 130 around the world</em></p> The in-person piece of the Rust community, the Rust meetups, have seen 30% growth in the last nine months and now total over 130 around the world. We've likewise seen steady growth in the number of conferences that users can attend for more training and networking.</p> We've also seen some impressive growth in indicators of the online community as well since the last time I reported on them</a>. For example, the Rust reddit recently passed 40k users (up from the 29k of nine months ago), the GitHub repo is nearly at 30k stars (up from 24k), and the official Rust twitter has over 31k followers (up from 24k).</p> This enthusiasm has also translated into the teams that govern Rust itself. Combined, the governing teams have doubled in size in the last 9 months to over 100 positions across the various teams</a>. The scope has also grown in tandem, with new teams that cover infrastructure, developer experiences, and dependency management.</p> Conclusion</h1> These are just some data points about Rust's use in industry and in the software development community. It's difficult to get a complete picture, and even more difficult to predict what will come next. That said, it does seem to shine a promising ray of light on where Rust is today.</p> Where will Rust go tomorrow? That is anyone's guess. I, for one, will certainly be watching and cheering it along.</p> Starting the next thing 2018-07-24T00:00:00+00:00 It's that time again! July 31st will be my last day at Mozilla.</p> Even though I was only at Mozilla for just under two and a half years, I had the good fortune to work on some pretty awesome projects.</p> I joined the Rust team and worked with Niko Matsakis and Aaron Turon on where Rust could grow, building on my experience coming from TypeScript. We learned a ton</em> from each other, and I'm really grateful for all those conversations and the close friendship that resulted. I also got to roll up my sleeves and write some code. The first project was to redesign the Rust error messages</a> with Niko. The next was to create the Rust Language Server</a> with Nick Cameron. Both have grown a life of their own, and it's been awesome to see!</p> I was also lucky to help Jack Moffitt out with the Servo project. Under his guidance, it produced loads of good code and good ideas. These made their way into Firefox</a> and into the Firefox Reality</a> work. Learning from Jack and connecting with the Firefox org (and some of the awesome Firefox managers) was a treat.</p> Finally, my last seven months have been helping create and run the Developer Technology team at Mozilla, a melding of the Rust and WebAssembly efforts under one umbrella. This allowed even closer collaboration between the two projects and has already resulted in lots of cool new technology, tools, and coding practices</a>. I'm excited to watch what they'll be able to come up with next.</p> And speaking of "next", what's next for me? I'm moving on to create a new company. I don't have a ton of details to share, yet, but I hope to do so in the coming months as the ideas and technology start to come together. "Will it use Rust?" I can already hear many of you asking. Most assuredly. If there's a better solution for efficient, write-once-and-don't-worry-about-it software development, I've yet to find it.</p> Thank you, Mozilla! Both for the experience and for the friends I've made along the way.</p> PS: I plan to continue podcasting - it's fun :)</p> How fast can we compile Rust hello world? 2018-05-03T00:00:00+00:00 Seeing Nick Nethercote's blog post about speeding up the compiler</a>, I started wondering just how fast could a Rust compiler be? How fast could we compile a simple example? How fast can we compile a Rust hello world?</p> Starting out</h2> When you do a cargo new hello_rust --bin</code>, you get a simple Rust hello world:</p> fn main() { println!("Hello, world!"); } </code></pre> Awesome. Let's see how long this takes to build:</p> > time cargo build Compiling hello_rust v0.1.0 (file:///home/sophia/Source/hello_rust) Finished dev [unoptimized + debuginfo] target(s) in 0.52 secs real 0m0.548s user 0m0.468s sys 0m0.080s </code></pre> Half a second on my XPS 13 9360, with a i7-7500U @ 2.70GHz with 16 gigs of RAM. That seems, a bit long for a hello world. Maybe cargo is doing extra work?</p> > time rustc src/main.rs real 0m0.378s user 0m0.289s sys 0m0.083s </code></pre> That's definitely better. We're just over 1/3rd of a second.</p> What should we expect? If only there were another language we could compile a hello world for that's comparable...</p> > cat hello_c.c #include <stdio.h> int main() { puts("Hello, from C!"); return 0; } > time clang hello_c.c real 0m0.082s user 0m0.021s sys 0m0.041s </code></pre> Woooaah! Wait a second. You're telling me that compiling C hello world is over 4.5x faster. That's... that's something. Maybe the compiler is doing a bit too much work for this small example? I mean, we don't need much from the standard library, just the println macro.</p> You know, a reasonable person would probably stop here and walk away. Chalk it up to "Rust is more complicated" or "maybe if I knew the right commandline options"</p> No, that's not us. Not today. Today, we're going to write our own Rust translator. We're going to see just how fast we can compile Rust to a working hello world.</p> First stop: let's build a Rust->C translator.</h2> Here's my thinking: if we write our own translator, we can translate the Rust to C ourselves, and then compile the C. That way, we should end up with a compile time close to the C.</p> > cargo new rabbithole --bin </code></pre> Writing a full compiler is complex work, but we're going for speed here. And speed means cutting corners. To that end, we're implementing just enough</em> of a compiler to translate our Rust to C, then we're calling that good enough.</p> First step, we need a parser. Luckily, there already is a crate that can parse Rust code for us.</p> [dependencies] syn = {version = "0.13", features = ["full", "extra-traits"] } </code></pre> extern crate syn; use std::env; use std::fs::File; use std::io::Read; fn main() { let mut args = env::args(); args.next(); match args.next() { Some(fname) => { let mut src = String::new(); let mut file = File::open(&fname).expect("Unable to open file"); file.read_to_string(&mut src).expect("Unable to read file"); let syntax = syn::parse_file(&src).expect("Unable to parse file"); println!("{:#?}", syntax); } _ => { println!("Please supply the file to compile"); } } } </code></pre> > cargo run -- hello.rs File { shebang: None, attrs: [], items: [ Fn( ItemFn { attrs: [], vis: Inherited, constness: None, unsafety: None, abi: None, ident: Ident { term: Term { sym: main } }, decl: FnDecl { fn_token: Fn, generics: Generics { lt_token: None, params: [], gt_token: None, where_clause: None }, paren_token: Paren, inputs: [], variadic: None, output: Default }, block: Block { brace_token: Brace, stmts: [ Item( Macro( ItemMacro { attrs: [], ident: None, mac: Macro { path: Path { leading_colon: None, segments: [ PathSegment { ident: Ident { term: Term { sym: println } }, arguments: None } ] }, bang_token: Bang, delimiter: Paren( Paren ), tts: TokenStream [ Literal { lit: "Hello, world!" } ] }, semi_token: Some( Semi ) } ) ) ] } } ) ] } </code></pre> This might look a bit intimidating, so let's break it down. The top-most thing is a File.</p> File</li> </ul> Each File continues zero or more Items. An Item are things like declarations. Function declarations, for example, are one kind of Item.</p> File Item</li> </ul> </li> </ul> Since this file has one function, it has one item, an ItemFn. This holds all the information about our main function. You can see in here that we have the function's declaration (sometimes called the function prototype, if you come from C). This tells us how many parameters it takes, what its return type is, etc. Luckily for us, this is all empty because main has nothing interesting here. This lets us focus on the body of the function, here called 'block'</p> File Item (ItemFn) Block</li> </ul> </li> </ul> </li> </ul> Blocks contain statements and expressions. This is really the meat of what actually does</em> things, and which is where we'll actually spend our time writing the compiler. Let's take a closer look:</p> block: Block { brace_token: Brace, stmts: [ Item( Macro( ItemMacro { attrs: [], ident: None, mac: Macro { path: Path { leading_colon: None, segments: [ PathSegment { ident: Ident { term: Term { sym: println } }, arguments: None } ] }, bang_token: Bang, delimiter: Paren( Paren ), tts: TokenStream [ Literal { lit: "Hello, world!" } ] }, semi_token: Some( Semi ) } ) ) ] } </code></pre> On closer inspection we see this block has only one statement. A something-something macro something or other. Oh right, println is a macro! Let's focus on just the macro invocation itself:</p> mac: Macro { path: Path { leading_colon: None, segments: [ PathSegment { ident: Ident { term: Term { sym: println } }, arguments: None } ] }, bang_token: Bang, delimiter: Paren( Paren ), tts: TokenStream [ Literal { lit: "Hello, world!" } ] }, </code></pre> Now this is a little more like it. It's still a little cumbersome, but we can see the parts we need. There's a sym</code> thing that says what the macro is. Later on, we can see the string that we're printing is a literal string. We can ignore the rest.</p> That's it. Make sure it's invoking println, and then we'll look for the string that's being printed.</p> To make things a little easier on ourselves, let's make a quick-and-dirty intermediate representation. This will only have one command so far, but it'll be an easy way to de-couple the function that does the parsing from the one that does the code generation.</p> enum Command { PrintLn(String) } </code></pre> With that, we're ready to make our parser that can take in the syn::File</code> and output our Command. There are a few ways we can do this, but to keep things pretty simple, I'm going to make a separate parse</code> function for each layer we talked about earlier: File, Item, and Block. You can think of this as us unpacking the useful information from what syn</code> gives us into much simpler data structures.</p> extern crate syn; use std::env; use std::fs; use std::io::Read; #[derive(Debug)] struct File { functions: Vec<Function>, } #[derive(Debug)] struct Function { body: Vec<Command>, } #[derive(Debug)] enum Command { PrintLn(String), } fn parse_body(body: &syn::Block) -> Result<Vec<Command>, String> { let mut stmts = vec![]; for stmt in &body.stmts { match stmt { syn::Stmt::Item(syn::Item::Macro(ref im)) => { let macro_name = im.mac.path.segments[0].ident.as_ref(); if macro_name == "println" { match im.mac.tts.clone().into_iter().next() { Some(ref arg) => { stmts.push(Command::PrintLn(arg.to_string())); } None => return Err("Expected argument in function".into()), } } else { return Err(format!("Unknown macro: {}", macro_name)); } } _ => return Err("Unexpected statement in function".into()), } } Ok(stmts) } fn parse_fn(item: &syn::ItemFn) -> Result<Function, String> { Ok(Function { body: parse_body(&item.block)?, }) } fn parse_file(file: &syn::File) -> Result<File, String> { let mut functions = vec![]; for item in &file.items { match item { syn::Item::Fn(ref item_fn) => { functions.push(parse_fn(item_fn)?); } _ => return Err("Unexpected item in file".into()), } } Ok(File { functions }) } fn main() { let mut args = env::args(); args.next(); match args.next() { Some(fname) => { let mut src = String::new(); let mut file = fs::File::open(&fname).expect("Unable to open file"); file.read_to_string(&mut src).expect("Unable to read file"); let syntax = syn::parse_file(&src).expect("Unable to parse file"); let result = parse_file(&syntax); println!("{:#?}", result); } _ => { println!("Please supply the file to compile"); } } } </code></pre> Which results in:</p> Ok( File { functions: [ Function { body: [ PrintLn( "\"Hello, world!\"" ) ] } ] } ) </code></pre> That's much nicer! It took a little work reading the syn</code> documentation, but once we got the hang of it, we can now work with real Rust source code and turn it into something we can use.</p> Oh, before we go, let's clean one thing up. You might have noticed that extra set of quotes in the macro argument. No problem, we'll just turn:</p> stmts.push(Command::PrintLn(arg.to_string())); </code></pre> Into:</p> stmts.push(Command::PrintLn(arg.to_string().replace("\"", ""))); </code></pre> Which gives us:</p> Ok( File { functions: [ Function { body: [ PrintLn( "Hello, world!" ) ] } ] } ) </code></pre> Beautiful. Now that we have it in this form, we have a much simpler form that we can hand to the other half of the translator. In this half, we'll be outputting the C code and compiling it.</p> Since we only care about one function in one file, let's just make a quick-and-dirty function to generate the C:</p> fn compile_to_c(file: &File) -> String { let mut c_output = String::new(); c_output += "#include <stdio.h>\n"; c_output += "int main() {\n"; for stmt in &file.functions[0].body { match stmt { Command::PrintLn(ref msg) => { c_output += &format!("puts(\"{}\");\n", msg); } } } c_output += "}\n"; c_output } </code></pre> That's it. Let's call it.</p> println!("{}", compile_to_c(&result.unwrap())); </code></pre> Running it again, we get:</p> #include <stdio.h> int main() { puts("Hello, world!"); } </code></pre> Woah! That looks like C to me. Dare we try to compile it?</p> > cargo run -- ../hello_rust/src/main.rs > testme.c > clang testme.c > ./a.out Hello, world! </code></pre> It... works! We just made a compiler!</p> > time ./target/release/rabbithole ../hello_rust/src/main.rs > testme.c real 0m0.014s user 0m0.010s sys 0m0.004s > time clang testme.c -o testme real 0m0.100s user 0m0.043s sys 0m0.041s </code></pre> Nice, by cutting lots of corners and making our own translator, we got the compile times down from 0.378s to 0.114s, less than 1/3rd of where we were.</p> Should we stop there? Surely we can go further down the rabbit hole. Oh right, C is not the lowest we can go</p> Deeper down the rabbit hole</h2> Let's cook up some assembly. First, let's test how fast we can assemble a hello world in assembly.</p> > cat hello_asm.s .text .global main main: mov $hw_str, %rdi call puts mov $0, %rax ret .data hw_str: .asciz "hello asm" > time as hello_asm.s -o hello_asm.o real 0m0.019s user 0m0.008s sys 0m0.011s > time ld -o hello -dynamic-linker /lib64/ld-linux-x86-64.so.2 /usr/lib/x86_64-linux-gnu/crt1.o /usr/lib/x86_64-linux-gnu/crti.o -lc hello_asm.o /usr/lib/x86_64-linux-gnu/crtn.o real 0m0.044s user 0m0.036s sys 0m0.008s </code></pre> Wow, the assembly output cuts the compile almost in half! If you haven't seen x64 assembly before, that might look a little strange. Instead of working with C code with functions, we've dropped down much closer to the machine itself. The above represents the lowest we can go in Linux. Let's unpack what's happening:</p> .text .global main </code></pre> Our assembly file starts with two things. One, a section that we'll be talking about the code itself (.text) and then the one symbol we'll be making visible (main). This will be the main function that the linker will use later.</p> Next, main begins:</p> main: </code></pre> What follows are series of assembly commands. The hardware keeps numbers and addresses in registers, similar to how you might store them in variables.</p> mov $hw_str, %rdi call puts </code></pre> With that in mind, this doesn't seem quite so bad. The first line is going to set up our register that we'll use as the argument to the puts function (here, in Linux, %rdi). After the argument is ready, we call puts.</p> mov $0, %rax ret </code></pre> After that, we return 0, just like we did in C. Here, in 64-bit assembly, we're returning the return value in %rax.</p> That's it. Oh wait, one more thing. So this was the code section, we still haven't told it what the string is to print (which we called $hw_str earlier).</p> .data hw_str: .asciz "hello asm" </code></pre> That's not bad. We have a data section that contains our strings. We have one string, called hw_str</code>, and we give it a null-terminated string to print.</p> You may have been wondering about our linker line, too:</p> > time ld -o hello -dynamic-linker /lib64/ld-linux-x86-64.so.2 /usr/lib/x86_64-linux-gnu/crt1.o /usr/lib/x86_64-linux-gnu/crti.o -lc hello_asm.o /usr/lib/x86_64-linux-gnu/crtn.o </code></pre> This bit of magic is actually the kind of tasks that your C compiler does for you behind the scenes. In Linux, for a program to run, it needs to be able to access things like the Linux program loader, the C standard library, and the Linux subsystems. The above takes the barebones object file we made with the assembler and gives it all it needs to be a complete Linux executable.</p> Before we write our assembly-outputting function, let's take a quick pause and talk about Windows. We've been giving a lot of examples for Linux, but you can do the same kinds of things in Windows. Make sure you have a 64-bit nasm</a> installed.</p> C:\Users\Sophia\source\fast_hello>type nasm_hello.asm global main extern puts section .text main: push rbp mov rbp, rsp mov rcx, message call puts mov rax, 0 pop rbp ret message: db 'Hello, Rust!', 0 C:\Users\Sophia\source\fast_hello>nasm -f win64 nasm_hello.asm -g -o nasm_hello.obj C:\Users\Sophia\source\fast_hello>link -out:b.exe -defaultlib:libcmt "-libpath:C:\\Program Files (x86)\\Microsoft Visual Studio 14.0\\VC\\lib\\amd64" "-libpath:C:\\Program Files (x86)\\Windows Kits\\10\\Lib\\10.0.15063.0\\ucrt\\x64" "-libpath:C:\\Program Files (x86)\\Windows Kits\\10\\Lib\\10.0.15063.0\\um\\x64" -nologo nasm_hello.obj </code></pre> You can see that the assembly looks different, though it's similar. We do a little more setup, but the core of the work is the same. We set up where the string is, we send it to the puts</code> function, and we return 0 via the rax</code> register. Likewise, after we have an object file from the assembler, we have to do a similar dance with the linker in order to have a fully working Windows executable.</p> With that, we can see all the techniques we're using in Linux can work in Windows, too.</p> Okay, let's finish this up and see where we are. Just like we wrote a C output function, we're going to write an assembly output function. This function will create two strings: one for the code and one for the data, and then stitch them together.</p> fn compile_to_asm(file: &File) -> String { let mut asm_code_output = String::new(); let mut asm_data_output = String::new(); asm_data_output += ".data\n"; asm_code_output += ".text\n"; asm_code_output += ".global main\n"; asm_code_output += "main:\n"; let mut temp_num = 0; for stmt in &file.functions[0].body { match stmt { Command::PrintLn(ref msg) => { asm_code_output += &format!("mov $str_{}, %edi\n", temp_num); asm_code_output += "call puts\n"; asm_data_output += &format!("str_{}:\n", temp_num); asm_data_output += &format!(" .asciz \"{}\"", msg); temp_num += 1; } } } asm_code_output + &asm_data_output } </code></pre> > time ./target/release/rabbithole ../hello_rust/src/main.rs > hello_asm.s real 0m0.015s user 0m0.010s sys 0m0.005s > time as hello_asm.s -o hello_asm.o real 0m0.022s user 0m0.009s sys 0m0.012s > time ld -o hello -dynamic-linker /lib64/ld-linux-x86-64.so.2 /usr/lib/x86_64-linux-gnu/crt1.o /usr/lib/x86_64-linux-gnu/crti.o -lc hello_asm.o /usr/lib/x86_64-linux-gnu/crtn.o real 0m0.043s user 0m0.029s sys 0m0.011s </code></pre> From 0.114s to 0.08s. We dropped another 30% off the compile time!</p> To recap, we started at 0.548s with Cargo, then 0.378s with rustc, 0.114 going via C, and 0.08 going via assembly. Roughly a 6.8x speedup. Can we squeeze anything else out?</p> You might be wondering about keeping more things in memory and getting a bit more caching. It's a good hunch, let's try it!</p> > time (./target/release/rabbithole ../hello_rust/src/main.rs > hello_asm.s && as hello_asm.s -o hello_asm.o && ld -o hello -dynamic-linker /lib64/ld-linux-x86-64.so.2 /usr/lib/x86_64-linux-gnu/crt1.o /usr/lib/x86_64-linux-gnu/crti.o -lc hello_asm.o /usr/lib/x86_64-linux-gnu/crtn.o) real 0m0.048s user 0m0.028s sys 0m0.021s </code></pre> That's another 40% off the compile time. We've now gone from 0.548s to 0.048s, a whole 11x faster. There's even more we can do. For example, we can squeeze out more by writing our own parser, outputting the .o ourselves, and more. But we'll leave that as an exercise for the reader.</p> Conclusion</h2> I hope you had as much fun reading this post as I had in writing it. There are amazing benefits to all the features that cargo and rustc provide, and the Rust ecosystem builds on those benefits. At the same time, it's fun to play around with compiler technology and see that when we get down to it, it's just another Rust program that we can tune to our heart's content.</p> You can see the rabbithole source</a> if you want to play with it yourself.</p> Talking about how we talk about Rust in 2018 2018-01-10T00:00:00+00:00 This is an entry in the #rust2018 blogging efforts to talk about Rust in 2018</em></p> When you come to Rust, you're bound to hear a lot of phrases. "Wrestling with the compiler". "Rust evangelism task force". "Non-lexical lifetimes". "Systems programming". Some seem strange. Others are a bit scary. Some are just a common set you quickly need to learn to before you can even read the Rust news.</p> In this post, I want to talk about how we talk about Rust in 2018. I think there are some ways we can change what we're saying and how we're saying it. These changes will help Rust be more approachable for beginners and an even nicer place in general.</p> "Wrestling with the compiler"</h1> I think we've all been there. Especially when we're starting with Rust, or trying to do something new with it we haven't done before, we feel like the compiler is fighting us every step of the way.</p> Sure, it's frustrating. On a bad day, we might even say "forget it". I wonder, though, is there a better way to talk about it? If you're a Rust user, do you trust the compiler? Are the error messages helping to clarify your thinking? Is it, as some people like to say, a pair programming partner for you?</p> I suggest in 2018, we kick the idea of wrestling with the Rust compiler to the curb and focus on how it helps us rather than the idea of it beating us down. Niko mentioned error messages</a> here to help with that, and I think we should push hard there. While we can change how we talk about the compiler, we can also change how the compiler talks to us as well.</p> By the end of 2018, no more wrestling. It's a tool to help us, and we should work hard both in how we talk and how it works to ensure this is true.</p> "Rust evangelism task force"</h1> We've seen the jokes about Rust having its own task force that swoops in and says things like "you should rewrite it in Rust". Sometimes the swoop happens just as a new security exploit is released. Sometimes it comes out of nowhere.</p> Or maybe you've seen this: someone on a thread, maybe on reddit, posts that maybe</em> Rust isn't doing so well in a benchmark or in a particular scenario. Minutes later there are half a dozen comments dissecting what happened.</p> It's intimidating. And really, it's not helpful to Rust.</p> As a community, it's great that we have so much enthusiasm. I just want to be sure we're channeling that enthusiasm in a way that keeps Rust growing. There are ways we can be courteous by not piling on or jumping in with uninvited suggestions. That might seem like little things, but at scale I think they'll make a big difference.</p> Some of you may be familiar with burntsushi's ripgrep tool. The author of a similar tool, ack, had a great quote</a> about the other day that applies to more than just search tools:</p> "If someone uses ripgrep instead of ack, it doesn’t hurt me. It’s the difference between an abundance vs. scarcity view of the world. I choose abundance."</p> </blockquote> We can adopt the same way of thinking. If someone wants to use Go or C instead of Rust, that's great! Perhaps that's a better fit for what they're working on. Maybe they'll try Rust again in the future. We should make it easy to come and go.</p> "Systems programming"</h1> There are a lot of terms that we use when talking about Rust that I think are simply confusing. Have you ever had this conversation?</p> Person 1:</strong> Have you heard of the Rust programming language?</p> Person 2:</strong> I'm not sure, what is it?</p> Person 1:</strong> It's a systems programming language.</p> Person 2:</strong> What's a systems programming language?</p> Rather than answering the question, we just made it seem even more complicated. Confusingly, even among people who have used Rust for a while there isn't even total consensus on what is meant by a systems programming language. Doubly confusingly, Rust isn't even just</em> a systems programming language anymore. With the ability to create web applications and tie into other languages, it starts to feel more than just systems-level development.</p> You can use Rust to build pretty much anything from a low-level operating system component to a JSX-like web app. In the future, we'll be in space</a>. Systems is both too confusing and too limiting for what Rust will be used to do.</p> Maybe there's a better term that we can use, or a small phrase that captures the right feel?</p> "Non-lexical lifetimes" and more</h1> Lots of the Rust community is pretty facile when it comes to computer science. Some of us have dabbled in Haskell. Others are old-hands at C++. We've seen programming from different angles and have a lot of vocabulary to use to describe Rust.</p> This started out as pretty effective short-hand. "This is like typeclasses in Haskell" we might say. "Lifetimes are lexically-scoped". "You need to use a lifetime variable, they're just like generics..." Etc. We can use it to communicate, but the end result can be a bit of a mouthful.</p> I'm not going to suggest we remove these terms from our vocabulary entirely, but I want us to think about how we can improve what we write publically. People we don't know will come by IRC, read our blog posts, and see our forum threads. They'll be coming from a variety of backgrounds, many of which will be different from our own.</p> If where we're talking is public with a wide audience, let's keep that in mind as we communicate. Keep the door open for more people to join.</p> Rust, at its heart, isn't intimidating. It's a technology that a wide range of programmers can use to target a wide range of applications. We don't need to make it sound or feel more complicated, and those extra few minutes to reword a sentence here or there can help.</p> Improving how we improve Rust in 2018 2018-01-07T00:00:00+00:00 Here's my #rust2018 post to add to the batch.</p> The past few years, I've been lucky to be part of the Rust community survey process. In 2016, this data helped show the need to improve usability. We had good foundations, but they were too hard to get at. In 2017, we saw a renewed need to reach out to companies, to improve documentation, and a continued importance of pushing to making Rust more usable.</p> These surveys were a key part of seeing big issues, but it got me thinking. Are there better ways to get and use data? Can we see Rust more clearly -- for what we do well and where we need to improve -- in the future?</p> When you do performance tuning, you benchmark and run perf profilers to get at the root of the problem. What if we did the equivalent for Rust as a whole?</p> We already measure through the surveys. The problem is, we can only really measure once or twice a year. This results in waiting long periods of time to see if we successfully addressed an issue. The delay may mean we don't catch new issues in time or we may fix a problem only to find out we didn't address the underlying cause.</p> Getting better data, sooner</h1> What if we had a way to have better data sooner? There are a few ways we could do this.</p> Smaller surveys</h2> The first thing we could do is put out smaller surveys and use them as a quick check-in. They might target specific features in a "did this change solve your issue?" way. We do a bit of this in the RFC process already. People who know about a particular RFC can jump in and comment on whether or not it the approach works for them.</p> While we want to continue using RFCs for this feedback, we could extend what we do here. For example, a blog post could talk about a group of RFCs working together to address one core problem. We could have a quick survey as part of the post to gather feedback on whether the combination of changes does what we hoped.</p> This is still pretty manual, though, and is focused developers actively reading RFCs and blogs and volunteering their time, which may be limited. Are there ways we can get more people involved across a wider range of changes?</p> Usability studies</h2> There's another avenue we could explore: usability studies. Work on software development in the past has shown there's naturally a distance between what people think</em> they spend the most time on and what they actually do</em> spend the most time on. Self-reporting often misses these insights.</p> When you actually watch someone working, issues with usability become more obvious.</p> With the rise of coding streams, we already have some usability data already. Perhaps there is a way to leverage the same tools so that we can perform studies remotely. The end result of these could more directly answer how workflows improve, and give us a clearer insight into both the Rust learning process and how more experienced Rust users use Rust in their work.</p> Opt-in statistics</h2> What if Rust developers could opt-in to some statistics gathered from the tools themselves? This would let Rust developers help the Rust project as part of their everyday work.</p> So it's clear, let me state I'm a strong privacy advocate. I have no desire to read people's private source code, to try to "guess" what they're doing, or anything remotely like that. Instead, here are some ideas we could be looking at that are helpful while not being invasive:</p> What error messages are people hitting? How often?</li> Roughly how much time is spent working in Rust?</li> How long are build times compared to the lines of code? How much is incremental compilation helping?</li> Roughly how much are particular language features being used?</li> </ul> I believe there's a way to we can gather data like this in a way that is both opt-in and preserves the privacy of the developer.</p> Transparency</h2> An important aspect of "better data, sooner" is transparency.</p> If we're careful, we can and should make the data we gather public. It gives contributors a window into the Rust world in more depth than we've previously had.</p> Imagine a live "board" where you could go and see Rust being used in real time, seeing what errors are occuring the most or if the latest fixes to nightly improved build times in the wild. This data could help not only people putting code in the compiler, but also folks writing RFCs, documentation, and more.</p> Measure twice, cut once</h1> When I was growing up, my dad would do various woodworking projects around the house. When I'd help, he would remind me to "measure twice, cut once". You didn't want to waste the effort it took to cut the wood if you couldn't use the result.</p> We can take a similar attitude for Rust in 2018. There are a lot of things we could</em> do. Lots of initiatives. Lots of projects. Can we come up with ways to measure twice before we invest the time? I think we can, and Rust will be better for it.</p> Here's to a happy, successful Rust in 2018!</p> Fun facts about Rust's growing popularity 2017-10-30T00:00:00+00:00 There are now 100 friends of Rust</a> (Be sure to hover to learn about how each company uses Rust)</p> </li> There are now 3 Rust podcasts: New Rustacean</a>, Rusty Spike</a>, and Request for Explanation</a></p> </li> First year had one conference, the second year had three, and this year had four</a>.</p> </li> crates.io is not even 3 years old, but already the number of crates has passed CRAN(R) and Hackage(Haskell)</a>.</p> </li> The Rust reddit is now over 29,000 Rustaceans strong</a>.</p> </li> the #rust irc channel has grown to over 1300 people</p> </li> The Rust gitter channel is over 1800 people</a>.</p> </li> our first Rust survey had 3,086 responses. Our second had 5,368 responses, over 2,000 more than the first year.</p> </li> Rust is a top 10 most discussed</a> project on GitHub.</p> </li> Rust is a top 10 most reviewed</a> project on GitHub.</p> </li> Rust is used on hundreds of millions of desktops</a> via Dropbox.</p> </li> The "impl period" now has at least 37 different teams</a> helping implement new designs.</p> </li> The popular @code, ranked 5th in this years Stack Overflow survey, powers its code search with ripgrep</a>, a search tool written in Rust.</p> </li> Atlassian uses Rust in a service for analyzing petabytes of source code.</p> </li> The new O'Reilly Rust book was the most popular on Safari</a>.</p> </li> There are over 100 meetups</a> around the world discussing Rust.</p> </li> npm rewrote performance-critical bottlenecks in the registry service architecture with Rust.</p> </li> Popular indie game developer Chucklefish recently announced their new game will be written in Rust</a>.</p> </li> The number of full-time Rust developers continues to grow. Over half are working on large Rust codebases</a>.</p> </li> Rust was the most loved technology two years in a row</a> in the Stack Overflow survey.</p> </li> @rustlang has over 24k twitter followers</a> 🎉🎉</p> </li> 367 applicants applied for the 12 spots in the Rust Reach initiative</a></p> </li> There are at least 5 Rust books in production, with releases over the next 18 monts.</p> </li> Red Hat added support for Rust in their recent updates</a>.</p> </li> IntelliJ officially supports Rust</a>.</p> </li> Rust's reddit channel hit 100,000 uniques and 1,000,000 views in July</p> </li> Amazon is hiring for developers with Rust experience</a>.</p> </li> Google engineers are experimenting with Rust for low-level system modules</a>.</p> </li> The official Rust YouTube channel</a> has over 90 videos on Rust.</p> </li> Android Rust development grew 10x in 2017 (to 9.9% of devs)</a>.</p> </li> iOS Rust development grew 10x in 2017 (to 5.3% of devs)</a>.</p> </li> Rust has seen slow, steady growth in commercial usage. Now over 20% of Rust devs use it at work</a>.</p> </li> More than 75% of respondents</a> said they feel comfortable in the Rust community, which includes even non-users of Rust.</p> </li> Rust attracts about 18%</a>, or roughly 1/5th, of its userbase in new users every month.</p> </li> The number of Rust devs with more than a year of experience grew to 42% in the last year</a>. That's a lot of experience!</p> </li> More than 2% of all Rust devs work on Rust codebases totalling more than 100k lines of code</a>.</p> </li> A growing number of Rust devs use Windows, which now makes up over 30% of all Rust users</a>.</p> </li> This year we saw a huge increase, at 16% of responses</a>, in the amount and style of documentation requested to cover more learning styles</p> </li> This year we also saw a shift in non-user reasons to be companies don't use it</a>. More folks now would learn it if it can pay the bills.</p> </li> Speaking of module counts, at 16/day Rust is the 7th fastest growing package repo</a>.</p> </li> The Rust internal forum</a> is a hotbed of activity, with popular posts regularly getting over 10k views.</p> </li> A growing number of GNOME developers are using Rust, with ports underway for some important libraries</a></p> </li> Rust and GNOME teams have had one hackfest with another next month</p> </li> Though early days, we're seeing more Rust support in the gstreamer media playback framework</a>.</p> </li> Canonical uses Rust in everything from server monitoring to middleware.</p> </li> Lots of System 76 tools and systems code</a> are written in Rust.</p> </li> Principal Intel engineers are encouraging people to learn Rust</a>.</p> </li> Facebook wrote a mercurial server in Rust</a>.</p> </li> Facebook is also a sponsor for Rust Belt Rust</a>.</p> </li> Over 2000 people</a> have contributed to Rust.</p> </li> The github repo has also gotten over 24k stars, making it one of the most popular programming language repos.</p> </li> Not to be missed, even the Rust book repo had over 180 contributors</a>.</p> </li> Rust first shipped in Firefox 48</a>, almost a year and a half ago.</p> </li> Rust is now used in five different modules in Firefox, with the next release having a new style system written in Rust.</p> </li> Rust Bridge</a> continues to be a good outreach to underrepresented developers, now having its 7th(?) event at this week's Rust Belt Rust</p> </li> Rust is supported by 11 regular subteams and 3 shepherd teams</a> that help mentor and lead discussion</p> </li> Staff engineer at Tumblr mentions that Tumblr is using Rust</a>.</p> </li> Rust is just behind Scala in active github projects (this month: 1263</a> vs 1376</a>)</p> </li> If you haven't seen them yet, the Mozilla Hacks videos are some of Rust's most popular (with one over 50k views</a>)</p> </li> Also in the survey, 16.8% of Rust devs are doing embedded work.</p> </li> Jorge Aparacio</a> has been a long-time supporter for embedded and is writing a book for embedded Rust development</a>.</p> </li> Rust got mentioned in a Batgirl comic</a>.</p> </li> The new Chucklefish game is not the first in Rust. An even earlier game was Shar</a>.</p> </li> Between Rust's first and second year, 56 RFCs were merged.</p> </li> In the last ~3 months, 36 RFCs were merged.</p> </li> Rust has had over 45 releases. Many since 1.0 have been well-received in programming news circles.</p> </li> Another published Rust game is a Snake's Tale</a></p> </li> Speaking of popular posts, one Rust project, alacritty, received over 1000 points on Hacker News</a>.</p> </li> crates.io has had over 240,000,000 downloads</p> </li> standard Rust tools are voted/promoted by the community. RLS, rustfmt, clippy, and rustup each have over 1000 stars on GitHub.</p> </li> for a bit of history, here's a post that helped start it all</a>. To this day, still a regularly referenced post.</p> </li> Ever wonder about the en-US at the end of https://www.rust-lang.org/en-US/ ? The Rust homepage is in 13 different languages!</p> </li> Language interop is an important piece of Rust's growth. We have C++ interop with bindgen</a>.</p> </li> There's also node.js interop with Neon</a>.</p> </li> There's good Ruby/Rust interop with Helix</a> and others</p> </li> Python and Rust also have on-going interop work</a>.</p> </li> In addition to interop, Rust has been an influence on Swift</a>.</p> </li> Rust's focus on memory safety without performance loss has helped other languages as well, including D</a>.</p> </li> Rust has also been inspired by other languages, including a revamp of errors</a> inspired by Elm.</p> </li> Rust has also grown from a wide range of influences, including discussions with top language designers</a>.</p> </li> Rust's 1st job was on creating browser tech. In making Servo, Rust focused on both safety and speed, w/o one excluding the other.</p> </li> Rust focuses on productivity also, leaning on ergonomics</a> for getting more done with fewer language rough edges.</p> </li> IEEE lists Rust as the 22nd most popular programming language</a>.</p> </li> This year, according to Redmonk, Rust jumped from 47th to 26th most popular programming language</a>.</p> </li> Likewise, Google trends shows Rust with (albeit a little noisy) comfortable growth over the last 5 years</a>.</p> </li> Rust is global, with heavy growth in the United States and Europe. These are the meetup locations worldwide</a>.</p> </li> Google trends also shows strong activity recently in parts of Asia</a>.</p> </li> Just noticed the Rust reddit has had 16 posts in the last 24 hours. Not bad!</p> </li> Rust just isn't a lang, but a community as well, with a focus on helpfulness. Growing this</a> is also important.</p> </li> In addition to community, Rust-focused contractors like Integer 32</a> and Asquera</a> have sprung up.</p> </li> Speaking of, there are a number of trainers available for Rust. The Rust community team</a> can help connect you.</p> </li> If you can make it out to one of the conferences, there are great hands-on labs</a> there as well.</p> </li> We've seen through multiple surveys that people stick around once they get used to Rust. It's a joy to watch.</p> </li> I forgot to mention, there's also an unofficial Rust slack</a> channel.</p> </li> As well as an unofficial discord server</a> for Rust discussions.</p> </li> Rust is used commercially in a wide range of areas</a>, incl. embedded systems, a/v transcoding, servers, and more.</p> </li> The "impl period"</a> is a reaction to Rust's growth. Discussions were at such a high volume that focus was needed.</p> </li> The Rust team at Mozilla is spread across three continents, with most hosting meetups/talks.</p> </li> Speaking of meetups, if you ever have a chance to go, check out the SF Rust meetup. Lots of great talks/ideas (also free pizza!)</p> </li> A core piece of Rust's growth is a sense of courtesy. This comes from the code of conduct</a>.</p> </li> Rust is growing into new areas. Recently, there's been a focus on WASM support, with new tools like wargo</a></p> </li> Anecdotally, I can say that I see new faces on Rust twitter regularly. More people are getting excited and sharing their adventures.</p> </li> A big feature for commercial adoption just landed in cargo: alternative registries</a>.</p> </li> The Rust community has its own poet that writes limericks on the news, including yours truly</a>. 😅</p> </li> Another fun survey fact: despite more users and releases every 6 weeks, only ~7% of users had upgrade breakage</a>.</p> </li> Lin Clark</a>, who makes awesome code cartoons, wrote a popular one on the upcoming Firefox style system rewrite</a>.</p> </li> Also, speaking of, there was a followup article on WebRender, another upcoming update to Firefox rendering</a>.</p> </li> While some may see Rust as largely Mozilla-driven, the Rust core team is actually made of folks from four different companies.</p> </li> On a personal note, I've been pleasantly surprised that each Rusty Spike episode has gotten over 500 listens</a>, even though it's new.</p> </li> This is the place-holder for a lot of stuff we can't talk about being done by companies you might know. Wish we could talk about it!</p> </li> Outreach to companies is exclusively grass roots by engineers in those companies promoting Rust and asking people to present.</p> </li> Rust is growing in importance in infosec. Projects like Tor have already begun moving code to Rust</a>.</p> </li> Awesome Rust</a> is well worth a shout-out. If you ever wonder about what's good in the Rust world, check it out.</p> </li> </ul> RLS now available on nightly 2017-04-18T00:00:00+00:00 We've got some good news for Rust IDE support.</p> We recently hit a milestone for the Rust Language Server, a tool that combines the power of the compiler with a fast autocompletion engine. Up to this point, if you wanted to use it you had to build it from scratch. The process was tedious, time-consuming, and error-prone.</p> We're happy to report you can now install RLS from rustup</a>. Landing as a component in rustup means a few important things.</p> First, this will make it easier to get up and running</strong> with the RLS. You no longer have to build the RLS to use it. Getting it up and running now just takes the time to run a few rustup commands:</p> rustup default nightly rustup update nightly rustup component add rls rustup component add rust-analysis rustup component add rust-src </code></pre> You can find more helpful tips in the readme</a>.</p> Second, this also makes it easier for IDE/editor plugins to automatically install the RLS</strong> and use its capabilities. This opens up the possibility for an even smoother user experience that is driven directly by the IDE plugin. We're looking forward to exploring this further.</p> Lastly, it is now easier to stay up-to-date</strong> with the RLS as it improves. Once you install the component, it will update alongside your compiler. All you'll need to do is run the rustup update nightly</code> command, and the RLS and its dependencies will update with it.</p> We've also been hard at work fixing bugs and improving the experience of using the RLS so that it's more stable and accurate than before. For larger projects, you may also notice that the RLS is a bit faster as well.</p> We're excited to get this out to more people and hear your feedback</a>. There's still plenty of work to do, and if you'd like to jump in and help</a>, we'd love to have you.</p> Using Rust in Windows 2017-03-28T00:00:00+00:00 As a Rust user, I was discouraged by not having all the good tips for using Rust in Windows in one place. This is an attempt to just that.</p> Getting started</h1> Step 1: Install Visual Studio Community edition</h2> My recommendation for working with Rust in Windows is to first get Visual Studio 2017 Community</a> installed. Edit:</em> Make sure you check the boxes to install C++ support</strong> when you instal Visual Studio, this provides the linker. The Rust compiler will use the linker that VS provides to translate our Rust code all the way to being an application. Having VS installed pays off in more ways than one, as we'll see later.</p> Step 2: Install rustup</h2> The Rust toolchain (including the compiler and the cargo</code> build tool) are managed by a tool called rustup</a>. This is the easiest way get going with Rust. It also has a few advantages, like allowing you to more easily switch the style of compiler (stable/beta/nightly) as well as building for other targets, like embedded systems.</p> Once Visual Studio is installed, rustup will detect it and let us install the compiler that works with the VS toolchain. Rustup on the initial install will let you pick the compiler, and you can select working with nightly or release. What you choose is up to you. Nightly gives you access to new features but hasn't been tested as thoroughly as the stable release compiler.</p> Whether you choose Nightly or Stable, you'll have commands available to you at the terminal. You won't need a separate shell to get to them. The commands 'rustc' and 'cargo' are the Rust compiler and Rust package manager respectively.</p> Step 3: Make sure everything is working</h2> In a source directory, you can test if Rust has been installed correctly by creating a simple project, compiling it, and running it.</p> cargo new hello_world --bin cd hello_world cargo run (or "cargo build" followed by "cargo run") </code></pre> You can find out lots of helpful general getting started tips</a> in the Rust documentation.</p> Writing Rust Code in Windows</h1> Currently, IDE support for Rust is still in its infancy. If you don't need fully intellisense-style IDE experience, pretty much all editors now have basic Rust support. Pick your favorite one, install their Rust plugin or syntax highlighter, and you're good to go. Many of these plugins come with support for a project called 'racer', which provides an autocompletion. Visual Studio users may be interested in Visual Rust</a>.</p> For the more daring, there's a project called the Rust Language Server</a>, which is based on the Language Server Protocol</a> developed for VS Code and now part of a growing number of editors and IDEs.</p> You can find steps for how to set up the RLS and use it with VS Code on its readme</a>. That said, the RLS is still pre-beta quality as of the writing of this post, so please exercise caution if you go this route.</p> Debugging with Visual Studio</h1> Earlier we installed Visual Studio. It seems a waste to have this nice VS install sitting around and not getting more benefit from it it. Let's use it for debugging.</p> Before we do some debugging, there are two steps we'll want to do beforehand:</p> Step 1: Install rust-src</h2> When we're debugging, VS will sometimes complain because it can't find the source code to the standard libs. We can remove this announce by installing the rust-src, and helping VS find the source it's looking for (this is also a helpful step for racer</code> support)</p> rustup component add rust-src </code></pre> This will let you find the source to the standard libs when you're debugging. After the command is run, it will be put in your .multirust directory. For me that directory is:</p> C:\Users\Sophia Turner\.multirust\toolchains\nightly-x86_64-pc-windows-msvc\lib\rustlib\src\rust\src </code></pre> When Visual Studio asks you for a file from the standard library source, this should give you a good place to start looking for it.</p> Step 2: Add natvis for Rust</h2> Rust has its own common structs, like String and Vec, that Visual Studio won't understand by default currently. To be able to view these data structures, we use native visualisation or .natvis files. Rustup will helpfully make them available for us, we just have to copy them into our personal directory. For me, the .natvis files are in:</p> C:\Users\Sophia Turner\.multirust\toolchains\nightly-x86_64-pc-windows-msvc\lib\rustlib\src\rust\src\etc\natvis </code></pre> NOTE: you may need to install the nightly</code> toolchain to have this directory. You don't need to use the nightly compiler as your main compiler, though, to take advantage of them.</p> I copy these files into:</p> C:\Users\Sophia Turner\Documents\Visual Studio 2017\Visualizers </code></pre> If you don't have a Visualizers</code> directory, go ahead and create it and then copy the files into it. Once you start up VS, it should pick them up and give you a little better view of your Rust data.</p> With that, debugging a Rust app works very similar to debugging other apps. There are a few steps you need to do:</p> Compile your projects with debug info. The easiest way is to build using cargo build</code> and NOT cargo build --release</code>.</li> Run your application</li> Attach the VS debugger using the 'Attach...' feature. You should be able to find your application in the list. Once you attach, you can hit pause, set breakpoints, and step as normal.</li> </ul> Debugging with VSCode and MSVC</h1> You can also setup vscode as your development and debugging platform. There are some good tips on how to do this in this blog post</a>.</p> Conclusion</h1> There is still plenty of room for improvement, but already there's enough here that we can write, build, and debug Rust applications without too much fuss. The Rust teams are working to continue to improve the experience of developing in Windows on Rust, but it's encouraging to see the progress and where it could go in the future.</p> Rust Language Server Alpha 2 Release 2017-03-02T00:00:00+00:00 Today, we're happy to announce the second alpha</a> for the Rust Language Server, a project build to bring high-quality Rust IDE support to any IDE or editor. This release brings new features, better stability, and an easier installation than the first alpha.</p> Support for new project types</h2> When a project first starts, it tends to begin life as a library or as an application. As projects grow, sometimes they take on aspects of both, being both a library and an application that can be executed.</p> With alpha 2, we add support for this hybrid project type. Why, you might ask?</p> </p> Editing Cargo's source code</em></p> With this change we can now work on Cargo in RLS. We're continuing to improve the experience and take on even larger codebases, but it's promising to see the RLS already able to work with real-world projects, like Cargo.</p> Support for custom derive</h2> Rust projects that need serialisation often reach for the popular serde</a> crate to do the work. With alpha 2, we now support the new serde annotations that are based on the new custom derive capabilities, which is sometimes called "macros 1.1".</p> Support for test code</h2> </p> Editing a test in Tokio</em></p> With alpha 2, you'll also be able to get intellisense for the #[test]</code> functions code in your project, allowing you to seamlessly switch between application and test code without having to change anything in your IDE or project. We'd like to extend this coverage more in the future.</p> Better stability</h2> We've fixed dozens of issues in the RLS and related crates. As a result, the experience is much more solid. We continue to make stability a focus as we push towards the beta release, but we're already pleased with the progress since the first alpha.</p> Easier installation</h2> When you look at the setup instructions</a> for the RLS now, you'll see we've got it down to four steps.</p> </p> Standard library documentation on hover</em></p> We've also simplified getting access to the standard library documentation. In one step, you can get now all the types and documentation for the standard library to your RLS install, allowing you to get the same IDE experience with the standard library that you do with your own code.</p> Part of the Rust language nursery</h2> The RLS, as it's matured, has proven that it's ready to take a step closer to being an official Rust project. Recently, the Rust tools team voted to bring the RLS in the Rust language 'nursery', a set of blessed projects that are working towards becoming an officially supported part of the Rust toolchain. We're excited to take this step and are committed to continuing to improve it towards the eventual 1.0 release.</p> Looking forward</h1> We're hopeful this will let more people set up the RLS and get the best experience. In the future, we hope to transition the RLS over to the rustup tool, which should allow you to add RLS support with one easy command. This release is a stepping stone to that vision.</p> Announcing Rust Language Server Alpha Release 2017-01-18T00:00:00+00:00 Today, we're announcing the first alpha release of the Rust Language Server (aka RLS)</a>. With this alpha release, this is the first time we're encouraging early adopters to try on real projects and send us feedback</a>. The RLS has now reached a level of maturity where it should be able to run against most Cargo-based Rust projects.</p> But what exactly is the Rust Language Server?</p> In a nutshell</h1> The Rust Language Server is a way of providing editors and IDEs with a range of functionality. Rather than leaving each editor plugin to have to parse and understand the types in your program and provide you with capabilities like refactoring, the RLS centralizes all this logic and provides it to the editor via a standard language server protocol</a>.</p> In this alpha release, this allows editors and IDEs to share the following capabilities:</p> auto-completion</strong> - allowing you to complete symbols and press '.' for member lists</li> goto definition</strong> - jump to the definition of a given symbol</li> goto symbol</strong> - jump to the definition of a symbol you know the name of</li> find all references</strong> - show all the locations a given symbol is referenced</li> rename/refactor</strong> - rename all instances of a given symbol to a new name</li> types on hover</strong> - get the type of a symbol</li> show errors</strong> - as the user types, get live analysis showing errors as they happen</li> </ul> The alpha release of the RLS has been run successfully on Linux, Mac, and Windows.</p> How to get started</h1> Step 0</strong>: Make sure you have a recent nightly rustc/cargo</a>, git</strong>, python</strong>, node</strong>, and cmake</strong> installed</p> Step 1</strong>: Check out the RLS:</p> git clone https://github.com/sophiajt/rls </code></pre> Step 2</strong>: Set up your editor (here we show VS Code plugin</a>)</p> git clone https://github.com/sophiajt/rls_vscode.git cd rls_vscode npm install </code></pre> Step 3</strong>: Set the RLS_ROOT environment variable to point to where you checked out the RLS:</p> export RLS_ROOT=/Source/rls </code></pre> At this point, you can open the project in VS Code, and when you run the project, it will open a new instance of VS Code with the RLS plugin enabled. The plugin will build RLS for you.</p> Note:</strong> the initial analysis of a project can take quite a long time, as metadata is built for all the project dependencies.</p> Note 2:</strong> analysis currently ignores test code, so you won't get IDE support for it. We have a fix for this in the works.</p> How does it work</h1> The current version of the RLS is built from a combination of two tools: racer</a> and the Rust compiler</a>.</p> Racer allows us to get quick-and-dirty completions, allow you to get completion results in sub-second times. The trade-off this makes is that the results are not as accurate.</p> For tasks that require higher accuracy, like safe refactoring or code navigation, the RLS uses the Rust compiler directly. Because this type of analysis, while more accurate, takes longer to complete, the RLS will tell the editor when the analysis has successfully completed.</p> Help, it doesn't work!</h1> The RLS is closely tied to particular versions of rustc. If you're a rustup user, the DLLs it needs are managed by rustup. In this case, you'll get an error like this if you run the rls by hand:</p> jturner-23759:rls jturner$ ~/.cargo/bin/rls dyld: Library not loaded: @rpath/librustc_driver-6eb85298.dylib Referenced from: /Users/jturner/.cargo/bin/rls Reason: image not found Abort trap: 6 </code></pre> If you see an error like this, you can run the rls by instead running the command cargo run</code> in the rls directory. In the future, as we improve the rls integration with the main Rust tools, this will no longer be an issue.</p> What's next</h1> The RLS is still early in its life and needs to go through a stabilization period before it's ready for 1.0. This alpha release marks the first release in this series.</p> The next release will be a second alpha</a> that includes another set of bugfixes and polish. This will be followed by a beta</a>. The beta will be the first that will be distributed as a binary and will be much closer to final 1.0 experience in terms of ease-of-install and ease-of-use.</p> I want to help</h1> Great! There are a few ways you can jump in.</p> Test the RLS</strong> - use the RLS on your projects and tell us how it feels. This will help us shake it out and make sure it's ready for production.</p> Contribute to RLS:</strong> - we'd like to add more refactorings, more Rust-centric code navigation (eg find all impls for a type), and possibly more advanced features like lifetime visualization and macro debugging.</p> Contribute an editor plugin</strong> - there are already at least six plugins</a> that follow the Language Server Protocol that the RLS uses, each in its own state of completion. If you have experience with one of those editors, or if you have a favorite editor you don't see on the list, you can help us by improving support. While the RLS will continue to grow and mature, it's really the plugin where the "rubber meets the road". It's the plugin that helps the developer get the most out of what the RLS provides.</p> Hack on the compiler</strong> - ultimately, we want to drive everything in the RLS straight from the compiler. This means being able to have sub-second response times for interactive completions as well. The Rust compiler team has already started a series of refactoring</a> to make the transition to an interactive mode possible. If you have a background in compilers, IDEs, or static analysis tools and would like to help with this work, you can contact the compiler team at #rustc on irc.mozilla.org or through the Rust compiler project on GitHub</a>.</p> Programming Language and Compilers Reading List 2016-10-08T00:00:00+00:00 This week, the talented Julien Fitzpatrick</a> (btw, you should check out their RustConf talk</a> if you haven't already) asked what a good list would be for people who are interested in programming languages and compilers. I took it as a good excuse to write a blog post with some of my recommendations. I'm going to attempt to put together a list that doesn't require a lot of background information, but as these things go, sometimes you'll have to re-read portions of books to get their benefit. Also, this isn't going to be an exhaustive list. I have never read through an exhaustive reading list before. Instead, this is definitely biased towards writing real code, getting things working, and learning by doing. Without further ado, let's get started!</p> Getting started</h2> To kick things off, the first thing to do is to remove any fears we have about thinking that compilers are big complicated things that are only possible by the most heroic of efforts. Yes, some compilers can be quite complex, but that's true of many kinds of industrial software. At its core, though, the compiler just does a few relatively simple things. So let's demystify those things.</p> First off, let's make a small compiler. @thejameskyle</a> has written a cool tutorial</a> that helps take you through the steps of creating a simple compiler. Rather than chewing your way through dusty tome after dusty tome before you write your first line of code, my suggestion would be to always have a code editor open at each step of the way. Play with the ideas as you learn them.</p> Alright, next, after we've played around with our little compiler and made a few changes and see how our changes affect the output, it's time to start building up our knowledge. Fundamentally, how a programming language works can be described in abstractions that build on other abstractions. If we peel these abstractions away, we're left with a pretty simple foundation.</p> One of the compiler books I liked that help build up these abstractions from a simple base is Jeremy Siek</a>'s compiler course</a>. The notes are laid out by chapter, and each chapter introduces a new concept that you'll add to the compiler. I enjoyed the hands-on approach, since you can think about how each feature affects each part of the compiler. It definitely gave me a "oh my god, I can't believe this works!" moments. You may also want to adjust to fit your particular style. For example, if something like the assembly piece feels awkward, you could instead output code to a language you're more familiar with, and work through the other steps of the compiler.</p> On the programming language side, probably one of the better survey books I've seen is Programming Language Pragmatics</a>. I'd probably pair it with something like Masterminds of Programming</a> or Coders at Work</a>. These two books are interviews with programming language creators. Creating programming languages is definitely equal parts art and engineering, and bouncing between the creators talking about the art and seeing how the engineering works in practice might help see parts of the picture a bit clearer.</p> Working some problems</h2> I used to hate doing homework. I mean, who didn't? But getting a little older, I realize just how useful doing some of the problems in the book can be. That's the time where you actually are solving little puzzles. And, if you're lucky, it's also a great chance to have one of those "a ha!" moments that unlock the next set of secrets.</p> If you're like I was at the time, you've heard of "functional programming" about a million times but weren't sure why anyone should care all that much. It wasn't really until I was working through Introduction to Functional Programming by Bird/Wadler</a> that things started to click. The problems in this book are great. They're juicy enough that you might have to work at them a little, but you come away with a much better sense of the concepts in each chapter.</p> Another book I enjoyed the problems in is the Dragon book</a>. The problem with the Dragon book, though, is that it definitely feels like a stuffy college textbook. That said, it's worth a skim. My favorite part about it are the problems in the earlier chapters. Working through those gave me a bit more confidence I understood the concepts.</p> Getting more modern</h2> Perhaps another problem with the well-known dragon book is that the dragon is a bit long in the tooth these days. Modern techniques have moved on a bit from where they were a few decades ago.</p> One thing to check out to see a bit more of how modern compilers work is the LLVM chapter</a> of Architecture of Open Source Applications (which may also be a fun read for someone interested in how things work generally).</p> I don't know of a good book on this topic, but there's a good video that goes into some of the ways that modern compilers work differently</a> than those in the dragon book.</p> For the brave</h1> When you've worked your way to this point and are feeling pretty confident, it's time to scale to the dizzying heights. To get there, you'll inevitably come across Types and Programming Languages</a> (which you might hear pronounced "tapple"). I'm not going to mince words, this book is brutal. It took months and months of working on it to crack it. I managed to finally get my head around it by methodically using a hands-on approach. I went chapter by chapter</a> and implemented each new language concept the author introduced. It took a little while, but it managed to turn the greek symbols into something a bit more concrete.</p> At this point you have quite the toolbox. You've written a few small compilers, you've seen how language features work and how they interact with each other, you can read academic papers (if need be), and maybe you've even made a programming language of your own. Now you can branch out to almost anywhere.</p> Helping with the Rust Errors 2016-08-03T00:00:00+00:00 I recently did some work with Niko Matsakis on a new compiler error format. You can try them out by setting RUST_NEW_ERROR_FORMAT=true</code>. While I put together a blog post talking about the design for the main blog, I also wanted to open it up for people to come help us move to the new errors. I've split up this post based on background for testers, doc writers, first-time compiler hackers, and experienced compiler hackers.</p> I'm good at testing!</h1> Perfect! We've love your help to come up with tests that show where an error message is confusing or just plain wrong. If you'd like a little help getting started, there are a bunch of unit tests for the compiler that all fail intentionally. You can find these in the rust-lang repo under the src/test/compile_fail</code> directory. You'll notice we've already started a list of errors</a>, but that list is incomplete. We'd like to make sure every error code that needs to be updated makes it onto that list so that others can help update them.</p> There are a bunch</em> of good examples in the compile_fail</code> directory that explore various error codes. If you look around there you can also search for error messages and see if there is already a test. Running this test will help you see if the error has been updated to the new format.</p> If you can find a test case that needs to be updated, file an issue like one of those listed in the "list of errors" link and cc @sophiajt. I can help get the error updated to the new format and tag the error so that others can find it.</p> I love writing docs!</h1> Good, we need that too. There are a couple places where doc writers can help out, especially if if you already know a little Rust.</p> The first place is helping us write clearer error messages. Even now, and especailly as we move to the new format, some errors are confusing and sometimes wrong. By helping write clearer error wording, and by experimenting with new ways to describe the error that is occuring, you can help us further improve the errors for the next developer.</p> If you look at the list of errors</a> we have so far you can see what we're going for. The likely need here is that errors need new labels. The general style of the new errors:</p> The error message text. This describes the general problem.</li> The primary error label (the '^^^' underline). This describes the 'what' of the error. It tries to summarize what went wrong in an fairly brief, yet approachable way.</li> The secondary error label (the '+++' underline). This describes the 'why' of the error. By reading these the user can see the order of operations that lead to the error.</li> </ul> You can also explore the src/test/compile-fail</code> of the compiler source code for a lot of unit tests that fail intentionally. By looking through these, you can help find errors that aren't update or that use confusing notes and labels.</p> Worth mentioning too, if you're a writer, also keep a lookout for Stage 2 (which hopefully will be ready in the coming weeks). We'll need help in the future writing helpful extended messages that describe errors. A great place to jump into help out here and learn about other places doc writers can help is #rust-docs on irc.mozilla.org.</p> I know Rust but I've never hacked the compiler!</h1> Excellent, a first-time compiler hacker. Let's walk through updating an error message to the new format. You can look through the list of errors</a> to see which ones need updating.</p> Once you've downloaded the Rust compiler source</a>, run the command:</p> > python src/bootstrap/bootstrap.py --step libstd --stage 1</code></p> This will build "Stage 1" of the compiler. After the first compile, this saves us a bunch</em> of time as we test our ideas as it is the minimal compiler rather than a full install.</p> Once built, we can run this by hand using (here, I'm on OS X):</p> RUST_NEW_ERROR_FORMAT=true ./build/x86_64-apple-darwin/stage1/bin/rustc <rust file></code></p> Let's take a look at an error that needs to be updated. You can find a lot of error tests in the src/test/compile-fail</code> directory in the compiler source. For this example, I'll use src/test/compile-fail/borrowck/borrowck-match-binding-is-assignment.rs</code>. You can also see which error this is looking for the EXXXX number, which is the error number. This one is E0384.</p> error[E0384]: re-assignment of immutable variable `x` --> src/test/compile-fail/borrowck/borrowck-match-binding-is-assignment.rs:50:13 \| 50 \| x += 1; \| ^^^^^^ \| note: prior assignment occurs here --> src/test/compile-fail/borrowck/borrowck-match-binding-is-assignment.rs:49:10 \| 49 \| [x,_,_] => { \| ^ </code></pre> We can see that's still using some of the old logic because the underlines don't have a label. You can also tell because there's an attached note is simple enough to be that label. Let's change the note to be a label.</p> To fix this, we first need to find it in the compiler source. Search tools to the rescue. I tend to use the error number (and sometimes part of the error message text)</p> jturner-23759:rust jturner$ grep -r E0384 src src/librustc_borrowck/borrowck/mod.rs: self.tcx.sess, span, E0384, src/librustc_borrowck/diagnostics.rs:E0384: r##" </code></pre> We now have the line that creates the error and another that lists it for us. We only need to change the first of these.</p> Looking at the code, sure enough, this is using the older style:</p> struct_span_err!( self.tcx.sess, span, E0384, "re-assignment of immutable variable `{}`", self.loan_path_to_string(lp)) .span_note(assign.span, "prior assignment occurs here") .emit(); </code></pre> This is a pretty easy fix. We just move from .span_note</code> to .span_label</code> and create a simple label, like this:</p> struct_span_err!( self.tcx.sess, span, E0384, "re-assignment of immutable variable `{}`", self.loan_path_to_string(lp)) .span_label(assign.span, &format!("prior assignment occurs here")) .emit(); </code></pre> Once we build and run the new compiler against the error message, we get something where the label is now part of the main snippet.</p> error[E0384]: re-assignment of immutable variable `x` --> src/test/compile-fail/borrowck/borrowck-match-binding-is-assignment.rs:50:13 \| 49 \| [x,_,_] => { \| - prior assignment occurs here 50 \| x += 1; \| ^^^^^^ </code></pre> That looks a little better, but we're not done. You'll notice that we don't have a label on our primary span (the ^^^^ part). In the new format, we like to have a label here because it's one of the first places your eye will look when you see an error message.</p> The second is that the label "prior assignment occurs here". That feels a little awkward. Since the label is now part of our code snippet, we can use a better wording. Let's update the error to look like this:</p> error[E0384]: re-assignment of immutable variable `x` --> src/test/compile-fail/borrowck/borrowck-match-binding-is-assignment.rs:50:13 \| 49 \| [x,_,_] => { \| - first assignment to `x` 50 \| x += 1; \| ^^^^^^ re-assignment of immutable variable </code></pre> With a little bit of wording changes and the new primary label, that's not bad. Of course, as you get comfortable working with these errors you may find even better wording that's more clear. Next, let's update the code to output this style:</p> struct_span_err!( self.tcx.sess, span, E0384, "re-assignment of immutable variable `{}`", self.loan_path_to_string(lp)) .span_label(assign.span, &format!("first assignment to `{}`", self.loan_path_to_string(lp))) .span_label(span, &format!("re-assignment of immutable variable")) .emit(); </code></pre> You'll notice that I reused the same span</code> that the main error message uses in my new .span_label</code>. When I do this, I'm telling the DiagnosticBuilder that this span_label is talking about the primary span and is the root of what is causing the error. Other labels are secondary spans because they're more informational.</p> And with that, we're done! Well, almost. We can't forget to update the unit tests.</p> To quickly check the unit tests that will fail because of our changes, we can run:</p> python src/bootstrap/bootstrap.py --step check-cfail --stage 1</code></p> Compile-fail unit tests are set up with their own annotations that say what labels to expect and where. As we update errors, we'll need to also update the corresponding test the new expected labels.</p> failures: [compile-fail] compile-fail/asm-out-assign-imm.rs [compile-fail] compile-fail/assign-imm-local-twice.rs [compile-fail] compile-fail/borrowck/borrowck-insert-during-each.rs [compile-fail] compile-fail/liveness-assign-imm-local-in-loop.rs [compile-fail] compile-fail/liveness-assign-imm-local-in-op-eq.rs [compile-fail] compile-fail/liveness-assign-imm-local-with-init.rs test result: FAILED. 2381 passed; 6 failed; 15 ignored; 0 measured </code></pre> Sure enough, we need to update some of the unit tests. Let's run these individually. To do this, we can use:</p> python src/bootstrap/bootstrap.py --step check-cfail <test name> --stage 1</code></p> like so:</p> python src/bootstrap/bootstrap.py --step check-cfail asm-out-assign-imm --stage 1</code></p> A fair amount of text comes out, but what we're interested in is:</p> error: /Users/jturner/Source/rust/src/test/compile-fail/asm-out-assign-imm.rs:21: unexpected "note": 'first assignment to `x`' error: /Users/jturner/Source/rust/src/test/compile-fail/asm-out-assign-imm.rs:24: unexpected "note": 're-assignment of immutable variable' error: /Users/jturner/Source/rust/src/test/compile-fail/asm-out-assign-imm.rs:21: expected note not found: prior assignment occurs here </code></pre> These lines tell us what's breaking our test case. In this example we see there are notes being printed that it doesn't expect and one expected note not found. This makes sense, since we changed the wording in the error. Let's go ahead and fix this.</p> We can change:</p> x = 1; //~ NOTE prior assignment occurs here </code></pre> To:</p> x = 1; //~ NOTE first assignment </code></pre> The //~</code> tells the test runner what the expected message is. Here, the message is a NOTE, which corresponds to the label we changed.</p> One more to go. We also added a new label on the primary span. We can insert a new note where this goes:</p> unsafe { asm!("mov $1, $0" : "=r"(x) : "r"(5)); //~^ ERROR re-assignment of immutable variable `x` //~\| NOTE re-assignment of immutable //~\| NOTE in this expansion of asm! } </code></pre> Notice: we use the //~</code> when the note is by itself, or //~^</code> and //~\|</code> when we're looking for multiple notes.</em></p> With that, the test now passes:</p> running 1 test test [compile-fail] compile-fail/asm-out-assign-imm.rs ... ok test result: ok. 1 passed; 0 failed; 0 ignored; 0 measured </code></pre> From there we work our way through the errors. Having these error tests failing gives us a chance to be sure that our fix is good. Here, when we fix compile-fail/liveness-assign-imm-local-in-loop.rs</code> we see that in a loop the two spans we're using might be the same span if the error happens in a loop. Let's fix that:</p> let mut err = struct_span_err!( self.tcx.sess, span, E0384, "re-assignment of immutable variable `{}`", self.loan_path_to_string(lp)); err.span_label(span, &format!("re-assignment of immutable variable")); if span != assign.span { err.span_label(assign.span, &format!("first assignment to `{}`", self.loan_path_to_string(lp))); } err.emit(); </code></pre> While we're at it, let's run a quick tidy to make sure our code conforms to the compiler coding standard:</p> python src/bootstrap/bootstrap.py --stage 1 --step check-tidy</code></p> Perfect. Ship it!</p> After we've finished with updating the error and the corresponding unit tests, we can create a PR. When you file the PR, put the message "r? @sophiajt" at the bottom of your commit message. This will add me to do the code review.</p> Extra credit</h1> A keen eye will notice that the file we've been working on did not show up in our failed tests. "Why is that?" you might ask. Unit tests that look at the compiler messages work in a few different ways. Some look at how a test looks (eg, tests in the src/test/ui directory). Others make sure the content is there, like the unit tests we just fixed. Because there is a lot of content to look for there are two kinds of "look for content" error test cases. The first kind will only look at the main message. You can see that this is the case because these test cases have no indicators to look for NOTE or HELP, only ERROR and WARN.</p> Watch what happens if we add a single NOTE to our original test case:</p> error: /Users/jturner/Source/rust/src/test/compile-fail/borrowck/borrowck-match-binding-is-assignment.rs:25: unexpected "note": 'first assignment to `x`' error: /Users/jturner/Source/rust/src/test/compile-fail/borrowck/borrowck-match-binding-is-assignment.rs:32: unexpected "note": 'first assignment to `x`' error: /Users/jturner/Source/rust/src/test/compile-fail/borrowck/borrowck-match-binding-is-assignment.rs:33: unexpected "note": 're-assignment of immutable variable' error: /Users/jturner/Source/rust/src/test/compile-fail/borrowck/borrowck-match-binding-is-assignment.rs:38: unexpected "note": 'first assignment to `x`' error: /Users/jturner/Source/rust/src/test/compile-fail/borrowck/borrowck-match-binding-is-assignment.rs:39: unexpected "note": 're-assignment of immutable variable' error: /Users/jturner/Source/rust/src/test/compile-fail/borrowck/borrowck-match-binding-is-assignment.rs:44: unexpected "note": 'first assignment to `x`' error: /Users/jturner/Source/rust/src/test/compile-fail/borrowck/borrowck-match-binding-is-assignment.rs:45: unexpected "note": 're-assignment of immutable variable' error: /Users/jturner/Source/rust/src/test/compile-fail/borrowck/borrowck-match-binding-is-assignment.rs:50: unexpected "note": 'first assignment to `x`' error: /Users/jturner/Source/rust/src/test/compile-fail/borrowck/borrowck-match-binding-is-assignment.rs:51: unexpected "note": 're-assignment of immutable variable' </code></pre> That's more like it! Now, we can continue fixing this unit test like we did the others.</p> Tips</h2> As you convert errors you may need to change span_err</code> (or span_err!</code>) to struct_span_err</code> (or struct_span_err!</code>), add new .span_labels, or tweak the wording for the error message to feel right.</p> I'm an experienced Rust compiler hacker!</h1> Perfect! There are a few hairy things we need to figure out, still, and they may require the keen eye of a compiler hacker. In the list of errors</a> there is a second list of issues for experienced compiler hackers. Additionally, there are areas that need a bit more work to make them feel like the new messages.</p> Region errors</h2> Regions errors currently provide a decent amount of information, but unfortunately to understand it you currently need a pretty decent understanding of Rust already. Ideally, these messages would get a similar treatment to the borrowck errors mentioned above.</p> Something like:</p> error[E0495]: cannot infer an appropriate lifetime for lifetime parameter 'r in function call due to conflicting requirements --> src/test/compile-fail/regions-infer-call-3.rs:18:24 \| 18 \| let z = with(\|y\| { select(x, y) }); \| ^^^^^^^^^^^^ \| note: first, the lifetime cannot outlive the anonymous lifetime #1 defined on the block at 18:21... --> src/test/compile-fail/regions-infer-call-3.rs:18:22 \| 18 \| let z = with(\|y\| { select(x, y) }); \| ^^^^^^^^^^^^^^^^ note: ...so that reference does not outlive borrowed content --> src/test/compile-fail/regions-infer-call-3.rs:18:34 \| 18 \| let z = with(\|y\| { select(x, y) }); \| ^ note: but, the lifetime must be valid for the call at 18:12... --> src/test/compile-fail/regions-infer-call-3.rs:18:13 \| 18 \| let z = with(\|y\| { select(x, y) }); \| ^^^^^^^^^^^^^^^^^^^^^^^^^^ note: ...so type `&isize` of expression is valid during the expression --> src/test/compile-fail/regions-infer-call-3.rs:18:13 \| 18 \| let z = with(\|y\| { select(x, y) }); \| ^^^^^^^^^^^^^^^^^^^^^^^^^^ </code></pre> You can see that we say things like "for the call at 18:12" or "on the block at 18:21". The span information is available but we're not really using it well. Instead of saying this, we may want to calculate the actual span that relates to "the call at 18:12" or the end point of "the block at 18:21".</p> Macro errors</h2> I've started a thread on improving macro errors</a> on the Rust internals list. The hope is to look at a better overall design for macro errors that fits better with the new error style.</p> Thanks!</h1> Thanks for wanting to jump in and help! You can find me on irc.mozilla.org as sophiajt and on github.com as sophiajt. If you have any questions, please ask. You can also get lots of good help in #rust-internals and #rustc on irc.mozilla.org.</p> Rhai 0.2 release 2016-03-26T00:00:00+00:00 I've just updated Rhai</a> to 0.2. This release focuses on improving the interaction between Rhai and Rust. The result is a cleaner, easier-to-use API. This new API does mean some API breakages, so you'll need to update your code to work with 0.2.</p> With this release, Rhai also now has arrays as a built-in type.</p> You can upgrade/install Rhai from cargo</a>.</p> API improvements</h1> Rhai 0.2 has a lot of small improvements that give it a much more natural feel in Rust. The main three areas are function registration, getters/setter helpers, and simpler eval.</p> Function registration</h2> Function registration now works through method calls on the engine, and we can infer the expected function type. The end result is generally much cleaner.</p> [Before]</strong></p> &(add as fn(x: i32, y: i32)->i32).register(&mut engine, "add"); </code></pre> [Now]</strong></p> engine.register_fn("add", add); </code></pre> Getter/setter helpers</h2> To help cut down on repetition, Rhai 0.2 has a few helpers for working with getters and setters.</p> [Before]</strong></p> &(TestStruct::get_x as fn(&mut TestStruct)->i32).register(&mut engine, "get$x"); &(TestStruct::set_x as fn(&mut TestStruct, i32)->()).register(&mut engine, "set$x"); </code></pre> [Now]</strong></p> engine.register_get_set("x", TestStruct::get_x, TestStruct::set_x); </code></pre> Simpler eval</h2> Eval will now take the expected return type as a type parameter, letting you cut out a few steps. We've also moved to using &str rather than String for eval inputs.</p> [Before]</strong></p> if let Ok(result) = engine.eval("40 + 2".to_string()).unwrap().downcast::<i32>() {} </code></pre> [Now]</strong></p> if let Ok(result) = engine.eval::<i32>("40 + 2") {} </code></pre> New feature: arrays</h1> In addition to the API improvements, Rhai now has a built-in array type.</p> var y = [1, 2, 3]; y[1] = 5; print(y[1]); </code></pre> Thanks!</h1> Thanks for the feedback on Rhai's first release. Also, big thanks to @nikomatsakis</a>, who gave me a few key pointers on how improve the API.</p> Looking forward to hearing from you.</p> Embedded scripting in Rust 2016-03-14T00:00:00+00:00 For the last few weeks, I've been working on an embedded scripting language for Rust, based loosely on ChaiScript</a> called Rhai</a>. What's an embedded scripting language? While the definition might depend on who you ask, for this post embedded scripting has a few distinct features:</p> A language that derives all functionality from bindings to the hosting environment</li> As a result, it allows complete control over the API exposed to the script, letting you lock down the script to your application's needs</li> </ul> In short, it's like the little brother of a full scripting language. It's smaller, thinner, but still useful for a variety of apps.</p> Because I'm using Rust as the host language, I also put a few other constraints on the project to make it work with a wider range of use cases.</p> No use of 'unsafe' - this lets us script where security is a concern</li> No additional dependencies - keeps our design small</li> Easy-to-use - even being safe and lightweight, it still needs to be easy to get started</li> </ul> Introducing: Rhai</h1> Let's take a look at what Rhai looks like, then we'll break down how it works.</p> Hello world</h2> As I mentioned earlier, one of the goals with Rhai was to have minimal setup to get going. This example is a full "hello world" of scripting that evaluates some script and gives us the result:</p> extern crate rhai; use rhai::Engine; fn main() { let mut engine = Engine::new(); if let Ok(result) = engine.eval("40 + 2".to_string()) .unwrap().downcast::<i32>() { println!("Answer: {}", result); // prints 42 } } </code></pre> Looking a little closer. We pull in the Rhai crate and get the scripting engine into view:</p> extern crate rhai; use rhai::Engine; </code></pre> Next, we create an instance of the engine:</p> let mut engine = Engine::new(); </code></pre> Finally, we call into the engine. Perhaps the most interesting part here is the roundtrip. The 'eval' method takes a String</code>, and returns a Result<Box<Any>, EvalAltResult></code>. Because we're in a scripting context, everything we're working with is dynamically typed, so as values get passed around, we make liberal use of Box<Any></code>. Here, the expression "40 + 2" is going to evaluated to give us a Box'ed number. Of course, we can also have errors occur during script execution, so we need to return a Result.</p> To go from a Result to the actual numeric value, we have to take a few steps to peel away the layers. I cheat and use 'unwrap' to get through the Result, then the downcast takes us from a Box<Any></code> to a Box<i32></code>.</p> engine.eval("40 + 2".to_string()).unwrap().downcast::<i32>() </code></pre> That's it. We've done a full round trip of creating script and then making sense of its output. The language, at this point, has only basic functionality because we haven't registered anything the script can use. So let's fix that.</p> Registering your functions</h2> To make your functions visible to the script, you'll need to register them with the script engine. Let's do the same example, but this time we'll create an add function in our native code and call it from script:</p> extern crate rhai; use rhai::{Engine, FnRegister}; fn add(x: i32, y: i32) -> i32 { x + y } fn main() { let mut engine = Engine::new(); &(add as fn(x: i32, y: i32)->i32).register(&mut engine, "add"); if let Ok(result) = engine.eval("add(40, 2)".to_string()) .unwrap().downcast::<i32>() { println!("Answer: {}", result); // prints 42 } } </code></pre> The big change in this example is this line:</p> &(add as fn(x: i32, y: i32)->i32).register(&mut engine, "add"); </code></pre> This uses the function registration capability taken from the FnRegister trait that we've added to our imports. It's a trait implemenented on functions which gives them a register method. We call the method and pass into it both our engine and the name we want the function to be called in script. With that, we're good to go and can call the function.</p> Let's do two more examples so we can see how some of the other details play out.</p> Function overloading</h2> Rhai also has supported for function overloading. This gives you a way to work with generic functions (like the example below) or to register multiple functions to the same name. During call resolution, the engine will call these functions and will fall through to the case that matches the types at runtime.</p> use std::fmt::Display; extern crate rhai; use rhai::{Engine, FnRegister}; fn showit<T: Display>(x: &mut T) -> () { println!("{}", x) } fn main() { let mut engine = Engine::new(); &(showit as fn(x: &mut i32)->()).register(&mut engine, "print"); &(showit as fn(x: &mut bool)->()).register(&mut engine, "print"); &(showit as fn(x: &mut String)->()).register(&mut engine, "print"); } </code></pre> Working with custom types</h2> Finally, in addition to working with your functions, Rhai also supports working with custom types. In this example, we introduce a few more features. First, you can register functions that take a &mut</code> as their first argument. This allows them to be used as methods in the script. Next, we register a special getter function. This lets us read (or write, if we register a setter also) the member directly in script.</p> You'll notice too that we make sure our object is clone-able. As we'll talk about in the next section, this ends up being a requirement when working with values in the script engine. We make sure to register our custom type with the engine to make cloning possible from inside the engine.</p> extern crate rhai; use rhai::{Engine, FnRegister}; #[derive(Clone)] struct TestStruct { x: i32 } impl TestStruct { fn update(&mut self) { self.x += 1000; } fn get_x(&mut self) -> i32 { self.x } fn new() -> TestStruct { TestStruct { x: 1 } } } fn main() { let mut engine = Engine::new(); engine.register_type::<TestStruct>(); &(TestStruct::new as fn()->TestStruct).register(&mut engine, "new_ts"); &(TestStruct::update as fn(&mut TestStruct)->()) .register(&mut engine, "update"); &(TestStruct::get_x as fn(&mut TestStruct)->i32) .register(&mut engine, "get$x"); if let Ok(result) = engine.eval("var myts = new_ts(); myts.update(); myts.x".to_string()) .unwrap().downcast::<i32>() { println!("result: {}", result); // prints 1001 } } </code></pre> How it works</h1> Let's take a look at how the scripting engine works. There may be even better ways to do some of these steps, so I'm eager to hear your feedback.</p> Without further ado, let's jump in.</p> Function resolution</h2> The heart of the system is the function resolver. In Rhai, all function calls, even all operators like plus and minus, end up going through the resolver. Yet, it's a relatively simple piece of code.</p> First, find all functions which match the name. Let's say 'foo'</li> For each 'foo' function, try to call the function with the argument(s) given</li> If it succeeds, return the resulting value</li> If it fails, continue trying the rest of the functions registered to 'foo'</li> If none succeed, error</li> </ul> Going through all the functions which match a name gives us the ability to do overloaded functions. It also lets us do operators, which themselves are commonly overloaded.</p> Function registration</h2> The next piece to working with functions in the engine is registering them. There are two kinds of functions that are registered: inner functions, or those created inside the script itself, and external functions, or those created in the native code.</p> Inner functions are relatively simple. They work by creating a new scope, adding the function arguments to this new scope, and then executing the function body.</p> External functions, if you will, are where the magic happens. Here, we handle the boundary between the scripting world and the native world.</p> Let's look at how to register a function with one pass-by-value argument fn(T)->U</code>:</p> impl<T: Any+Clone, U: Any+Clone> FnRegister for fn(T)->U { fn register(self, engine: &mut Engine, name: &str) { let wrapped : Box<Fn(&mut Box<Any>)->Result<Box<Any>, EvalAltResult>> = Box::new( move \|arg: &mut Box<Any>\| { let inside = (arg).downcast_mut() as Option<&mut T>; match inside { Some(b) => Ok(Box::new(self(b.clone())) as Box<Any>), None => Err(EvalAltResult::ErrorFunctionArgMismatch) } } ); let ent = engine.fns.entry(name.to_string()).or_insert(Vec::new()); (ent).push(FnType::ExternalFn1(wrapped)); } } </code></pre> There's a lot going on, so let's break it down:</p> impl<T: Any+Clone, U: Any+Clone> FnRegister for fn(T)->U </code></pre> As I mentioned earlier, all the types used in the scripting engine need to be clone-able. We enforce that here as well because we'll use the clone feature in the call. We are implementing the FnRegister trait on all functions that match this constraint.</p> Next, we are going to create a wrapper function which will call the function given:</p> let wrapped : Box<Fn(&mut Box<Any>)->Result<Box<Any>, EvalAltResult>> </code></pre> All functions of arity one have this internal wrapper function, regardless of if they are pass by value or reference. This lets us work through a vector of functions easily during function resolution, without having to have too much information out the specific type of the function.</p> Next, we have the wrapper function itself:</p> Box::new( move \|arg: &mut Box<Any>\| { let inside = (arg).downcast_mut() as Option<&mut T>; match inside { Some(b) => Ok(Box::new(self(b.clone())) as Box<Any>), None => Err(EvalAltResult::ErrorFunctionArgMismatch) } } ); </code></pre> The wrapper is a closure that moves its environment. In this case, it's moving the function we're wrapping. Once it has ownership, we can safely return the wrapper and complete registration.</p> Inside the closure, we handle working with the argument. Because we know the type of the function during registration, we use this knowledge to downcast the argument and give it the proper type. If this succeeds, we know we have the correct function and can call it. If it fails, we know this isn't a match and later can try the next function in the overloads.</p> To call the function, since we're pass by value, we need to take the extra cloning step to get to a value rather than a reference. This lets us respect the ownership model, albeit at the cost of the clone.</p> That's the heart of how the resolution system works. Resolution works through each possible match, and registration gives us a wrapper function that can try each function against the types known at compile time.</p> The rest of the story</h2> Many of the remaining features built on these two main components. As mentioned earlier, when you register special get and set functions for fields, the engine will use them when you work with the fields in script.</p> The engine will register a set of basic arithmetic and comparison operators to get started with. These also use the same function resolution mechanism.</p> For variables, the engine works by passing a scope stack around. For lookup, in order to follow the ownership rules (and maintain pass-by-value semantics), we clone variable values when they're used in an expression. During assignment, we use a mutable reference to their value in the scope stack so that it can be updated with the right-hand side of the assignment.</p> What's next?</h1> Rhai is still very young and has a bit of growing to do before it's ready for primetime. Parsing and error reporting are barebones and don't have the kind of error recovery you'd expect from more industrial-strength scripting. There are also a number of convenience features that would be relatively easy to add, like vectors, iterators, for..in.., and so on.</p> If you'd like to help out, jump over to the GitHub site</a>.</p> Going down the rabbit hole with Rust traits 2016-02-09T00:00:00+00:00 One of the first things you might notice about Rust, if you come to it from other OOP-style languages, is that Rust separates methods from the data they work on. You create your struct</code>, then you impl</code> a few methods on it later.</p> struct Rect { height: i32, width: i32 } impl Rect { fn area(&self) -> i32 { self.height self.width } } fn main() { let r = Rect { height: 10, width: 5 }; println!("area: {}", r.area()); } </code></pre> Which, to me, seemed slightly awkward at first. But, as I dug in a little futher, I started asking questions like "Can I do that to existing types? Even basic types like the String type?" Indeed you can. In this post we'll see some of what's possible.</p> Warning: we're going down a rabbit hole under the influence of the Bad Idea Bears</a>. I hope you have your snorkel. :)</em></p> Warming up</h1> We've talked a little about the basics already. In Rust, impl</code> gives us a way to "add" methods onto types. But, when we try to do the same thing to String</code>, we hit a snag:</p> impl String { fn my_len(&self) -> usize { self.len() } } fn main() { let name = "foo".to_string(); println!("my length: {}", name.my_len()); } </code></pre> When we try to compile this, the compiler gives us back an error:</p> mylen.rs:1:1: 3:2 error: cannot define inherent `impl` for a type outside of the crate where the type is defined; define and implement a trait or new type instead [E0116] mylen.rs:1 impl String { mylen.rs:2 fn my_len(&self) -> usize { self.len() } mylen.rs:3 } </code></pre> So we can't impl</code> for a type outside of... oh wait. We can do it, so long as we give it a trait to work with.</p> Try #2:</p> trait MyLen { fn my_len(&self) -> usize; } impl MyLen for String { fn my_len(&self) -> usize { self.len() } } fn main() { let name = "foo".to_string(); println!("my length: {}", name.my_len()); } </code></pre> Woah, that worked! With that, we peek into the rabbit hole.</p> Traits and extending strings</h1> To kick off our explorations, let's give ourselves an example to use. Let's say we want to read in the contents of some file, and we get the filename for that file from the commandline.</p> use std::io::Error; fn read_all(fname: String) -> Result<String, Error> { use std::fs::File; use std::io::prelude::; let mut contents = String::new(); let mut f = try!(File::open(fname)); try!(f.read_to_string(&mut contents)); Ok(contents) } fn main() { use std::env::args; if let Some(fname) = args().nth(1) { if let Ok(contents) = read_all(fname) { println!("{}", contents); } } } </code></pre> Wouldn't it be nice to not mix and match helper functions and methods? Wouldn't it read a little nicer to havefname.read_all()</code>?</p> Using the same pattern we used before, this is pretty easy to do.</p> use std::io::Error; trait FileReader { fn read_all(&self) -> Result<String, Error>; } impl FileReader for str { fn read_all(&self) -> Result<String, Error> { use std::fs::File; use std::io::prelude::; let mut contents = String::new(); let mut f = try!(File::open(self)); try!(f.read_to_string(&mut contents)); Ok(contents) } } fn main() { use std::env::args; if let Some(fname) = args().nth(1) { if let Ok(contents) = fname.read_all() { println!("{}", contents); } } } </code></pre> That's not bad! Go get my nth argument, check if we're good, then read the contents.</p> If we could stop ourselves there, we'd be mostly okay. Our example is still pretty readable, though we might get a few dirty looks from the senior devs in the lunch line.</p> Traits and extending Option</h1> The Bad Idea Bears call us deeper! Can we play a little code "golf"</a> and squeeze out using Option</code> in our main? Sure! All we need to do is create another impl</code> for our FileReader</code> trait, this time for Option</code> (specifically those that contain Strings):</p> use std::io::Error; trait FileReader { fn read_all(&self) -> Result<String, Error>; } impl FileReader for str { fn read_all(&self) -> Result<String, Error> { use std::fs::File; use std::io::prelude::; let mut contents = String::new(); let mut f = try!(File::open(self)); try!(f.read_to_string(&mut contents)); Ok(contents) } } impl FileReader for Option<String> { fn read_all(&self) -> Result<String, Error> { use std::io::ErrorKind; match self { Option::None => Err(Error::new(ErrorKind::NotFound, "No such file or directory")), Option::Some(ref s) => s.read_all() } } } fn main() { use std::env::args; if let Ok(contents) = args().nth(1).read_all() { println!("{}", contents); } } </code></pre> Yay! Our main is smaller, and we've still managed to maintain at least a little</em> of our decency. I mean, we could have used unwrap</code>, and thrown care to the wind! Instead, we're still doing error-checking on our contents. Albeit, our error-checking is kinda smooshed into one place.</p> What's that, Bad Idea Bear? We should go further? Okay!</p> Traits and extending iterators</h1> We're doing a read_all()</code> on the Option, but what if we go right to the arguments themselves?</p> If we poke around in the docs, we find out that args() returns its own type, Args</a>. As we read on we see that Args is just an Iterator</a>.</p> That gives us a couple choices. We can implement a new trait on Args:</p> trait FileReaderIter { fn read_nth(&mut self, pos: usize) -> Result<String, Error>; } impl FileReaderIter for Args { fn read_nth(&mut self, pos: usize) -> Result<String, Error> { self.nth(pos).read_all() } } </code></pre> Or, we can work with Iterator:</p> trait FileReaderIter : Iterator<Item=String> + Sized { fn read_nth(&mut self, pos: usize) -> Result<String, Error> { self.nth(pos).read_all() } } impl FileReaderIter for Args {} </code></pre> Let's actually stop here for a minute and look at this one. We can't extend Iterator itself. This is a Good Thing(tm), because if we could, we would end up having to extend countless implementors of Iterator. Instead, we need to create a new trait that extends from Iterator.</p> Next, we need to say what kind</em> of iterator we can work on. We know ahead of time that this works on specifically iterators over Strings. We use the <Item=String></code> to lock that knowledge in place for the compiler. This is called an associated type</em>.</p> With the new trait and associated type, we're almost done. Lastly, we're doing something a little different this time around. We have a default implementation inside of the trait instead of just having a prototype. A default implementation saves you the trouble of having to repeat yourself with each type that implements the trait, if the function is general and trivial.</p> The gotcha is that if you use a default implementation that uses self</code>, the compiler needs to know how big self</code> will be because it will pass it by-value. Hence, anything that implements FileReaderIter</code> must have a known size so the compiler knows how many bytes to copy in when self</code> is passed to the method. To ensure this, we add another trait to inherit from, Sized</code>. With that, we can say that Args trivially implements this trait.</p> Phew, okay, that's it. What have we gotten for our effort?</p> Both approaches give us this simple main:</p> fn main() { use std::env::args; if let Ok(contents) = args().read_nth(1) { println!("{}", contents); } } </code></pre> At this point we're far enough down in the rabbit hole that there's only a pinpoint of light from the entrance.</p> Let's work our way back out, shall we?</p> (ps: if you want to follow Bad Idea Bear further, this little nugget is hidden in the Rust docs: both Result and Option implement iterator traits and can be used as iterators</a>)</em></p> Climbing back out of the rabbit hole</h1> Cleverness and readability often trade off with each other. As we tried to get more clever and squeeze our main down, the end result became more code we had to maintain. We also increased the amount of 'magic' we used, whether it was treating multiple failure styles/cases as one case or the amount of additional capabilities we were giving types like String and Option.</p> Rust's trait system is powerful and clever. It lets you express the algorithms that are core to your project in a way that can even extend the basic Rust API. Indeed, that's one reason why the Rust language and core libraries are intentionally kept pretty small and tidy: so you can make these kind of additions (using eg. modules from creates.io</a>) and the standard library doesn't get in your way.</p> But, as they say, with great power comes great opportunities to follow Bad Idea Bears down rabbit holes.</p> Rust quickie - matching Strings 2016-02-08T00:00:00+00:00 In case you find yourself trying to match a String</code> (perhaps as part of an Option</code> or Result</code>), here's a little trick.</p> As a concrete example, let's say you're working with commandline args and want to do different things if it's there and equal to a special value (like "-"), if it's any other filename, or if it's not there.</p> match args().nth(1) { Some("-") => println!("Input is stdin"), Some(ref x) => println!("Open file: {}", x), None => println!("Open default file /foo/bar") } </code></pre> When you try to compile this, you get:</p> error: mismatched types: expected `collections::string::String`, found `&'static str` (expected struct `collections::string::String`, found &-ptr) [E0308] </code></pre> Bummer! At least on the surface, it looks like we don't have an easy way to use a string constant like "-" in our pattern match when what we're matching against a String.</p> Luckily, Rust pattern matching has a way to help us. We can approximate the above by using match guards</a>. Match guards allow us to put an additional if</code> expression on the pattern.</p> match args().nth(1) { Some(ref x) if x == "-" => println!("Input is stdin"), Some(ref x) => println!("Open file: {}", x), None => println!("Open default file /foo/bar") } </code></pre> Match guards give you an extra bit of "juice" to express your logic.</p> Feature Bias, or Rethinking the Blub Paradox 2016-01-23T00:00:00+00:00 I got a lot of great feedback on my previous post "Rust and the Blub Paradox"</a>, which inspired me to write a follow-up.</p> In its original formulation, the Blub Paradox</a> is something of a tool for people to feel smug about their favorite language. I heard a lot of feedback that it feels condescending. I hadn't realized that in my mind the Blub Paradox had morphed from its original form into something different.</p> In this post, I'd like to break down the Blub Paradox into to what I see as its fundamental observation: that we, as programmers, can have a form of unconscious bias. I'll call this Feature Bias</em> so that we don't confuse it with Paul's original idea.</p> There are no universally better languages</h1> There is no way to universally sort programming languages on a continuum that is monotonically increasing in awesomeness. It's impossible. Not only are languages too complex to have any kind of universal ranking, they are tools and as tools the value is left to the tool user. It's like asking "is pencil better than a paint brush?" It all depends on what you're doing and what you're more comfortable with.</p> The types of languages that Paul criticizes in Beating the Averages aren't universally worse than his favorite languages. Taking one example, everyone loves to put Java down, but have you ever watched Notch (the creator of Minecraft) code in it</a>? He seems perfectly content to create with his tool of choice, and he's certainly capable of doing so.</p> Feature Bias happens locally per feature</h1> Rather than trying to sort languages universally, a useful version of the Blub Paradox is a local phenomenon. It happens on a feature by feature level.</p> If a programmer accustomed to working with a feature sees a language that lacks the feature, they can also see the use cases where programmers in that other language might benefit from its use.</p> If a programmer who doesn't use a feature looks at a language with this feature, until they've spent time to understand it, they may see the feature as unnecessary.</p> Feature Bias happens to most programmers, even the best</h1> As a language PM at Microsoft sitting on a few programming language committees, I saw Feature Bias happen to people from junior developers to famous language designers with decades of industry experience. The first time a developer sees a new feature, regardless of skill level, there's a chance they may experience the bias.</p> True story: the C# feature LINQ had to 'bake' for at least a year with the design committee before enough members saw its value for it to become part of the language. All the while the champions behind it were trying to educate and get other committee members to give it a chance.</p> Features don't have universal value</h1> Once the developer puts in the time to understand the feature, they still</em> may not like it. That's not the paradox/bias anymore. They've put the time in to understand the trade-offs, and they don't feel the trade-offs are worth it. That's a totally acceptable response.</p> The root of the bias</h1> To summarize, just like other unconscious biases, this bias is pointing out something that's common for people to experience and they may not realize they're experiencing it.</p> A lot of programmers I know, including myself on a good day, use an understanding of the bias as a way to train ourselves to take time with each feature and work through the knee-jerk "oh this is crap!" reaction. I definitely still fall victim to it. Just as with other biases, you have to keep working at it.</p> In seeing the responses from people, it's unfortunate that the original Blub Paradox has been used as a club to beat people with. It makes it that much harder to see where it can be helpful. In my experience, it can shine a light on an area that helps us become better programmers and teachers.</p> Rust and the Blub Paradox 2016-01-22T00:00:00+00:00 A few weeks ago, I read an analysis of Rust, D, and Go</a> by Andrei Alexandrescu. Andrei, a respected member of the C++ community</a> and a core developer of the D programming language</a>, took a stab at Rust at the end of his writeup with what seems like a pretty astute observation:</p> "Reading any amount of Rust code evokes the joke "friends don't let friends skip leg day" and the comic imagery of men with hulky torsos resting on skinny legs. Rust puts safe, precise memory management front and center of everything. Unfortunately, that's seldom the problem domain, which means a large fraction of the thinking and coding are dedicated to essentially a clerical job"</p> Having met Andrei and seen a few of his talks, I know that he likes to poke fun. Still, let's take the bait. Is his observation funny because it's a funny image or is it funny because there's an underlying truth to what he's saying?</p> EDIT:</strong> thanks to all the responses to this post, I've written a follow-up on where the Blub Paradox is wrong and where it's helpful</a>.</p> EDIT #2:</strong> Wow, some people also seem to be taking this article as an attack on Andrei. Not at all! Like I said, I've chatted with him in person and he's very much the type that likes to say funny things, whether about himself or the wide array of programming languages. I just took one of his quotes about Rust as a starting point and had fun with it and looked at what might be weird about Rust coming from other languages.</p> Blub paradox</h1> Whenever I think about how useful certain language features are, I almost always come back to Paul Graham's essay "Beating the Averages"</a>. In it, he observes an interesting phenomenon among programmers he calls "the Blub Paradox". To recap for those who haven't heard of it, the paradox goes something like this: let's say there's a programmer using a language called Blub. Blub falls somewhere in the middle of the programming language pack in terms of capability. It's not the lowest, most minimal language nor is it the most powerful.</p> When the Blub programmer looks "down" the spectrum of languages, they can easily see that those languages are less expressive than their preferred Blub language. Unfortunately, when they look "up" the spectrum of languages, they can't tell they're looking up. As Paul puts it:</p> "What he sees are merely weird languages. He probably considers them about equivalent in power to Blub, but with all this other hairy stuff thrown in as well. Blub is good enough for him, because he thinks in Blub."</p> When I read that years ago, I remember thinking "wow, that's pretty astute." Little did I know that years later it would become firmly ingrained as part of how I thought about teaching programming as a programming language PM.</p> As a programming language PM at Microsoft, I worked on TypeScript</a>, a typed version of JavaScript. Without fail, if I was speaking to an audience of predominantly JavaScript developers, I would get many frowns as I tried to enumerate the advantages of adding a bit of type-checking to JavaScript. Even if it was optional. Even if I showed half a dozen powerful advantages. As Paul notes, it just looked "weird". To a JavaScript developer, TypeScript would often just seem like a language that was equally expressive with "hairy" stuff thrown in.</p> As I talked to more language PMs, and I met more and more people at conferences, I learned that Paul's observation was not only accurate, it was also surprisingly universal. Most programmers will struggle when looking at a new language they don't use. Often, they'll have allergic reactions to features that are foreign. It takes time to work with the features enough to understand why they aren't just unnecessary ornamenation gumming up the works.</p> In short, the Blub Paradox is something that as programmers we actively need to assume we're falling into and then work to get ourselves out of.</p> Let's do just that. Let's look at a few stand-out Rust features and assume these features are weird and unnecessary. From there, we'll see if we can work to un-Blub ourselves.</p> Weird feature #1: Rust-style polymorphism</h1> Let's make a Rust program that uses a bit of polymorphism to print out two different structs. I'll show you the code first, then break it down.</p> use std::fmt; struct Foo { x: i32 } impl fmt::Display for Foo { fn fmt(&self, f: &mut fmt::Formatter) -> fmt::Result { write!(f, "(x: {})", self.x) } } struct Bar { x: i32, y: i32 } impl fmt::Display for Bar { fn fmt(&self, f: &mut fmt::Formatter) -> fmt::Result { write!(f, "(x: {}, y: {})", self.x, self.y) } } fn print_me<T: fmt::Display>(obj : T) { println!("Value: {}", obj) } fn main() { let foo = Foo {x: 7}; let bar = Bar {x: 5, y: 10}; print_me(foo); print_me(bar); } </code></pre> What an eye-full! There's polymorphism there, but it doesn't look anything like OO. It's generic, and not only generic, but it has constraints. And what's this impl</code> stuff?</p> Let's break it into parts. I'm creating two structs to hold our values. I go a step further and make sure they implement something called fmt::Display</code>. In C++, you might overload the << operator for ostream. The end result is similar in Rust. After I have this in place, I can call print functions and pass these structs directly.</p> With that, we're half way through the example.</p> Next, we have our print_me</code> function. This function is generic, and will accept anything so long as what is passed in can do fmt::Display</code>. Luckily, we've already said how both structs can do just that.</p> The rest is straight-forward. We create a few structs and pass them to print_me</code>.</p> Phew... that seemed like a bit of work. And really, that's how polymorphism starts in Rust. It's all with generics.</p> Now, let's switch to C++ for a minute. Someone using C++, especially a beginner, might not go the generic route at first and might instead opt to use OO-style polymorphism:</p> #include <iostream> class Foo { public: int x; virtual void print(); }; class Bar: public Foo { public: int y; virtual void print(); }; void Foo::print() { std::cout << "x: " << this->x << '\n'; } void Bar::print() { std::cout << "x: " << this->x << " y: " << this->y << '\n'; } void print(Foo foo) { foo.print(); } void print2(Foo &foo) { foo.print(); } void print3(Foo foo) { foo->print(); } int main() { Bar bar; bar.x = 5; bar.y = 10; print(bar); print2(bar); print3(&bar); } </code></pre> Pretty easy, right? Okay, here's a quick quiz for you. What does the C++ code print?</p> If you guessed wrong, don't worry. You're in good company.</p> If you guessed right, congrats! Now, think a minute about how much do you have to understand about C++ to know the answer. From what I see, you need to understand the stack, how objects are copied, when they are copied, pointers, references, v-tables, and virtual dispatch. Just to do a few simple lines of OO.</p> When I first started learning C++, I had a pretty steep hill to climb. Luckily, my cousin</a> is an expert C++ programmer and took me under-wing to show me the ropes. Still, I made tons of beginner mistakes like the one above. Why? In part, the hill is steep because of the cognitive load of learning C++.</p> Some of that cognitive load is inherent in programming a computer at a system's level. You need to understand the stack. You need to understand how pointers work. What C++ adds is another layer of complexity around knowing when a value isn't fully copied and when virtual dispatch is and isn't used -- all without any warning to the user if they do something that is "probably a bad idea(tm)".</p> This isn't to knock on C++. Much of Rust is based on keeping the C++ philosophy of low-level efficient abstractions intact. In fact, you can even write code that's very similar to the Rust example</a>.</p> What Rust does</em> do in this case is to separate inheritance from polymorphism and push you towards genericity from the start. In doing so, it gets you thinking generically from day #1. It might seem weird to separate inheritance from polymorphism if you're used to them being used together.</p> That separation gives us one of our first Blub moments: what's the advantage of separating inheritance and polymorphism? In fact, does Rust even have inheritance?</p> Believe it or not, at least as of Rust 1.5, Rust does not allow inheritance of structs. Instead, we can grow the functionality of structs from outside them using a feature called 'traits'. Traits allow you a way to add methods, require methods, and retrofit datastructures for existing systems. They're also allowed to inherit, so one trait can extend another.</p> If you poke around in the Rust, you'll also notice something else. We don't have the same issues we had with C++. There's no worrying about what gets lost when the function is called or how virtual dispatch works. In Rust, we're uniform regardless of the type. Because of this, a whole class of beginner errors vanish.</p> Weird feature #2: what, no exceptions?</h1> While we're on the subject of things Rust doesn't have, its next weird feature is that it lacks exceptions. Isn't that like going backwards? How do we talk about errors that only occur sometimes? How do we bubble out errors so we can handle them in one place?</p> Next you'll be saying I have to learn monads.</p> Actually... no, just kidding, you don't need to learn monads. Rust's exception-less programming is a lot more straight-forward. Here's an example of what this looks like in practice. First, examples of what the functions look like:</p> impl SystemTime { /// Returns the system time corresponding to "now". pub fn now() -> SystemTime; /// Returns an `Err` if `earlier` is later pub fn duration_from_earlier(&self, earlier: SystemTime) -> Result<Duration, SystemTimeError>; } </code></pre> Notice the now</code> function returns a SystemTime and doesn't have any error cases, while duration_from_earlier</code> does have a Result that can be either a Duration or the error SystemTimeError. At a glance, you know a function's success and possible failure cases.</p> But all those failure cases are bound to stack up. Who wants to see them all over the code? Sure, it's good to do error checking, but the point of exceptions is to be able to handle them not only locally but also to let them bubble up and handle them in one place.</p> Rust lets you do that, too.</p> fn load_header(file: &mut File) -> Result<Header, io::Error> { Ok(Header { header_block: try!(file.read_u32()) }) } fn load_metadata(file: &mut File) -> Result<Metadata, io::Error> { Ok(Metadata { metadata_block: try!(file.read_u32()) }) } fn load_audio(file: &mut File) -> Result<Audio, io::Error> { let header = try!(load_header(file)); let metadata = try!(load_metadata(file)); Ok(Audio { header: header, metadata: metadata }) } </code></pre> While it may not look like it at first, the above example uses the "bubbling" style of exceptions. The trick is in a Rust macro called try!</code>. What it does is pretty simple. It will call a function for you. If that call returns with a success value, it will hand that to you. If it instead returns an error, try!</code> will return from the containing function with that error.</p> The end result means that if if load_header</code> ever has a problem when calling file.read_u32()</code>, that Error will be returned instead. If it is, the same thing happens in load_audio</code>, meaning that it, too, will return the same Error. And so on, until the calling function finally handles the error case.</p> I wrote a post about try!</code></a>, if you'd like to learn more.</p> Weird feature #3: the borrow-checker</h1> You know, it's funny. Many people mention the borrow-checker as the first thing about Rust. Often, they even rank it as the</em> thing that sets Rust apart. To Andrei, this is the "hulky torso" part of Rust. To me, though, the borrow-checker is just another pass in the compiler. Just like type-checking, it helps catch a level of bugs before they happen at runtime. That's it. It might seem hefty at first, but I'd argue that's just because it's different and learning to work with it grows a new muscle as a programmer, not unlike working with a new type system.</p> What kinds of bugs does the borrow-checker catch, you ask?</p> Use-after-free</h2> Ah yes, the classic problem of freeing memory and then trying to use it later. Quite often, this is what makes programs crash with the dreaded "null pointer exception".</p> There are a good handful of C++ best practices to help avoid this (using RAII, preferring references or smart pointers over raw pointers, documenting the ownership in your API docs, etc), which is likely what Andrei referred to as "large fraction of the thinking and coding are dedicated to essentially a clerical job". A team of highly trained C++ programmers would see avoiding use-after-free as largely clerical because that's about what it amounts to -- so long as you obey all the best practices, never cheat, and you always only grow the team with more experts.</p> Invalid iterator</h2> Ever modify a container you're iterating over in C++ and then have a random crash later on? I have. If you managed to add or remove an item to the container that was enough to force memory reallocation, your iterator has become invalid.</p> I don't often make this error, but I still do from time to time.</p> Data races</h2> In Rust, data is either shared or mutable. If it's mutable, it can't be shared, so you don't risk mutating from two threads in a race. If it's shared, you can't mutate, so you can read to your heart's content from multiple threads.</p> If you're coming from C++, or any other language with a set of good parallel libraries, this is going to feel overly strict to say the least. Luckily, this isn't the whole story, but instead the general basis that gives you a simple set of rules to build more abstractions on. The rest of the story is still young and forming. There are a growing number of libraries focused on concurrency</a> that you can follow if you're interested to learn more.</p> Tracking ownership</h2> This may seem a bit redundant, but it's actually something that C++ struggles with. I alluded earlier to "documenting the ownership in your API docs". The problem is that this information is in a comment rather than the code itself.</p> Here's a scenario: you're in C++ and you need to call a library that you didn't write. Let's say it's a C-based library and takes a raw pointer. Will it delete what you passed in? Will it take ownership of it and put it into one of its data structures? Maybe you're calling into a scripting engine like Ruby. Then who owns it?</p> Rather than reading through docs, Rust lets you verify that your assumptions match reality when it tracks how you are using the API with the borrow-checker.</p> And more</h2> Borrow checking helps other classes of errors. For example, it helps improve local reasoning when you're writing a function because you can safely assume that any mutable values coming in are disjoint and won't interact with each other when you change them.</p> This incidentally also opens the door for additional optimizations because the compiler can ensure that values can never be both aliased and mutable, something that is difficult to do in C-based languages with alias analysis.</p> There's more good info about the borrow checker and classes of errors it prevents in "You can't spell trust without Rust"</a> if you're interested.</p> Weird feature #4: rules are meant to be broken</h1> I'd argue that one of Rust's strongest suits is that it's pragmatic. Most of Rust's stringent checks can be circumvented using features like unsafe</code>, mem::transmute</code>, and more. That's right, don't like the borrow-checker for your use case? No problem, turn it off.</p> Rust is made to be able to express anything you can do in a C-like systems language. The advantage of Rust is that it's much easier to start with safe defaults that can be pushed out of the way when needed. It's much more difficult to build safety on top of unsafe language features.</p> While Rust lets you pick and choose, it biases towards not letting you shoot your foot off.</p> Did Rust miss leg day?</h1> Speaking of feet, did Rust miss leg day after all? Is it lopsided? Has it been focused on the wrong things?</p> Rust is getting stronger, and luckily it has enough safeguards in place to know how to go into the squat without pulling its back. This part can't be understated. The philosophy behind Rust is one of having good fundamentals. As long as the good fundamentals are there, the language can grow on that foundation.</p> But you don't have to take my word on it. Like any fix for the Blub Paradox, it comes down to programmers picking up Rust and seeing if the kinds of capabilities Rust gives them allows them to express themselves efficiently and in a way that avoids classes of bugs that distract from the problem at hand.</p> It's natural to think that some features are ornamental when you're first coming to them. Time and exposure are the real test.</p> Special thanks to Yehuda Katz and Jason Turner for reviewing this post and sending me their feedback</em></p> Travel log (Maui) 2015-12-15T00:00:00+00:00 When I left Seattle, I jokingly told my friend "the first country I'm travelling to is Hawaii".</p> "But it is isn't a different country," she replied.</p> It is. Oh it is.</p> </p> Looking at Maui from the Kula Lodge</em></p> Having travelled around the United States, I can't say there are many places even similar to Hawaii. Or, more precisely, Maui, the island I spent most of my time on this past week.</p> To start with many of the views look absolutely, totally fake. Yes, the above picture was my view at lunch one day. Only after staying in Maui for a week did I start to get used to the fact that driving around the interior of Maui, at any moment you could turn the corner and see another breath-taking view.</p> I don't know if I could get used to it really</em>. Half my shots of the beach or looking out over the mountain look like they're from a postcard.</p> </p> Kamaole Beach III</em></p> Sure, I guess someone might over time get used to the amazing mixture of volcanic rock, perfect beaches, and mountain views rounded out with gorgeous weather. Add to that easy access to a wide variety of fresh produce and fish, and you have yourself a quite literal slice of paradise. It became quickly obvious why Hawaii has its reputation. It is, quite simply, nothing short of amazing.</p> To give it an even more foreign feel, I was greeted by a number of foods and plants I'd never heard of before. Fruit that tastes like cinnamon (complete with a gritty texture), starfruit, and even the plentyful macadamia nuts used in new ways (like in the crust on Mahi Mahi, for example).</p> Speaking of macadamia nuts...</p> </p> Like I said, a variety of foods. The plants, too, came in all shapes and sizes. One of the most striking is the Rainbow Eucalyptus.</p> Rainbow Eucalyptus at Garden of Eden</em></p> The above picture is from the Garden of Eden</a> on the Road to Hana. Both the Garden and the Road to Hana are well worth exploring. The Road is a windy affair that takes you around through some of the various micro-climates as you traverse the island. My friend and I explored it one day, stopping at the Garden of Eden, a nature reserve that contained many cool varieties of plants, including the Rainbow Eucalyptus above. In addition, many varieties of bamboo, bananas, flowers of various sorts, and starfruit flourish.</p> But you don't really have to go very far in any direction to get the island experience.</p> </p> Maui, as one may expect being a holiday spot for many, is a great reminder to take things a little slower. Stop and look at the flowers, enjoy the views, soak in the sun, and rest up for whatever adventure might come along tomorrow. There's also a general friendly, caring spirit that people generally strive for called the Aloha Spirit, sharing its name with the famous greeting. As a sense of goodwill, it carries through the island's inhabitants and even into governance</a>. Just as important as soaking in the sun is to soak in some of the Aloha while you're there and take it with you.</p> </p> I'm looking forward to returning to Hawaii and seeing what else it has to offer. Like many places, it feels like I just scratched the surface. What I found was a warm, welcoming place, both in temperature and in vibe. Highly recommended checking it out. It's well-worth the flight. Just don't rush too much, or you'll miss some of the best parts.</p> Mahalo for reading!</p> HandmadeCon 2015 2015-12-05T00:00:00+00:00 HandmadeCon 2015, where even the welcome banner is handmade</em></p> Today, I attended HandmadeCon 2015</a>. HandmadeCon is the first convention for the Handmade Hero</a> project by Casey Muratori, a game industry gun-for-hire who most recently worked on The Witness</a>, the follow-up game for Sophia Blow, the creator of Braid</a>.</p> Key takeaways</h1> There was a ton</em> of material in the interviews. Enough so that I suspect if two people wrote down their takeaways, they might have wildly different lists. Still, here are the takeaways I had:</p> Quick iteration is king. Working towards faster turnaround time is going to deeply impact overall quality.</li> Lock-in fundamental game choices as early as possible. Examples: game controls for platforming, game speed for open-world. These impact how you design levels, and any change may cause you to have to redo levels, puzzles, assets, etc.</li> To work in the game industry you need to show both self-motivated passion and curiosity. The rest can be learned.</li> </ul> Format</h1> Casey (right) interviewing Sophia Blow</em></p> Casey has hand-picked a list of game developers with deep technical experience for the conference. With each developer, he interviewed them about their strengths and would tease out the key points from the discussion, often drilling in to learn more. I hadn't heard of some of the people he'd picked, but I quickly realized he'd hand-selected each for their specific expertise in gaming, whether controls, code layout, networking, design, or ingenuity.</p> Tommy Refenes</h2> Super Meat Boy</em></h2> Key quote(s)</h3> "Best way to get the best work out of people is to give people a way to give you the best work"</em></li> </ul> Assets pipeline</h3> Original asset pipeline used JSFL to output texture files with indexing from the Adobe Flash tools</li> Now he just works with the SWF directly. He (gasp</em>) read the SWF manuals.</li> </ul> Movement</h3> Spent three months perfecting controls using a square that jumped around the screen</li> Movement was built on a set of hand-coded equations whose parameters were tweaked until everything felt right. He offhandedly called these "garbage numbers/equations"</li> Rather than being real-world, everything was based on how it played Eg) 200ms of grace time where meat boy sticks to the wall, allowing you to press the other way and then hit the jump button, or vice versa, to initiate a wall jump</li> </ul> </li> After the initial controls tuning, they were locked in and did not change for the rest of the development</li> This allowed level design to build on the controls</li> Explored things like analog, but ended up going digital. Reasons: never want a "half jump" or a slow acceleration</li> </ul> Testing</h3> Fans are the best testers</li> PC is crazy complex, and you're bound to bake in some hardware configuration assumptions</li> He's working on crowd-sourced alpha/beta testing platform that he can ship builds out to specific testers</li> </ul> Mike Acton</h2> Insomniac [creators of Ratchet and Clank]</em></h2> Key quote(s)</h3> "Adding stuff to the problem does not make the problem simpler"</em></li> "Everytime I hear 'I can just optimize at the end' I want to slap someone"</em> Probably a chain of failures lead to this.</li> "Capture and visualize"</em> your data. Actually look at the data. eg) build tools to look at the spacial queries that developers are running.</li> </ul> Reaction to Tommy</h3> Mike started off with a few reactions to Tommy's interview.</li> Tommy's input style isn't garbage, it is</em> real and it fits the input of his movements and output of "Tommy smiles"</li> Doing things that are "real world" may actually just be the wrong answer and move you the wrong direction</li> He whole-heartedly agreed about locking down controls so you can lock down how levels are made</li> On a related note: ovement speed, esp in an open-world game, greatly affect how the engine works because of streaming off disk, etc</li> Read the CPU manuals</li> BTW, docs are seriously important. Doc your own internal formats, too. This is your key input.</li> Also, focus on your own problem and your own game. Too many indie devs mimicking other frameworks/tools.</li> </ul> Hiring a game developer</h3> Main corollary for game developer success: curiosity</li> Demonstrate what you've done with your own willpower</li> Learn how to practice, and take practice seriously. Anything else is an excuse. Your career is one of the most important things in your life, up there with family.</li> Practice with the goal of getting better/faster and solving the problem. Throw it away rather than keeping it. If it's practice, it's not a project. You're honing a skill rather than building a thing.</li> </ul> Pat Wyatt</h2> Guild Wars, Diablo, StarCraft and more</em></h2> Key quote(s)</h3> "Make your iteration speed as fast as you can, because this will determine the quality of your game"</em></li> </ul> Early network stacks</h3> Synchronized network models where client and server both had game state and were kept in lock-step</li> Warcraft: RTS are not so twitchy, don't need to see effect simultaneously</li> Eg) 4 network moves per second, and a move is when you've heard from everyone</li> Starcraft similar to Warcraft</li> </ul> Diablo on Battle.Net</h3> Move from singleplayer to real-time multiplayer</li> First idea: Synchronized at first, to help prevent cheating</li> Ideally, network model should be separate from UI data, since one is updated by network, and one by the client, and they may be updated at different times</li> Problem with synchonized client/server: if you don't do it perfectly, you diverge and it's hard to debug</li> Diablo written too quickly to do a synchronized version</li> Solution: "Asynchronous loosely-coupled game"</li> Each level has a 'master', and the master is the consensus driver for world state for that level</li> Replayable log of state that can be given to a new master if the first one drops out</li> Game host declares who is master for a given level based on who first entered that level</li> State change happens in two steps: "I am attempting to pick up the sword" then master plays through all attempts and break ties</li> DON'T DO A GAME LIKE THIS. People will cheat. Griefers will take advantage</li> Ask yourself: "How will people hack this?" "How will people abuse others?"</li> </ul> </li> </ul> Guild Wars</h3> All client/server, all TCP</li> Network functionality split across many servers</li> File server: handles versions and creates deltas based on what client has available, compresses, and sends update to client for assets in the manifest</li> Asset ids automatically assigned by perforce full path</li> File servers were put near users because this helps throughput. Related: tcp window and waiting for ack for each packet</li> Lobby server: authentication and presence. Arena, instance, etc all controlled through lobby server because you don't have a persistent connection. Helps with p2p messages, guild msgs etc, encryption</li> Matchmaking server: will match up servers with capacity with needs of user</li> Using tons of servers allows you scale up and distribute across machines</li> Network code is shared across all servers. Every server is actually a generic modular server. It just hotloads the specific module. eg) lobby server just loads the lobby module</li> Need RPC protocol that supports versioning. These days, things like protocol buffers</a> would work fine.</li> </ul> UDP vs TCP</h3> Mobile - UDP works great, since you're already going to be losing packets</li> MMO - TCP is great because you don't want to lose packets</li> </ul> Network game states</h3> Failure is always legal. Eg) can't load model file, you don't crash. Instead you get a white box</li> Server is authoritative and may pull players back to acceptable state. Also, every client sees their version of the server truth, like the arrow that "bends in space" because target moved</li> </ul> Best practices</h3> Always simulate network jitter and latency, minimum 200ms, so you are always creating robust networking code</li> You will always be hacked Deeply understand your interaction model</li> Rate limiting helps prevent abuse</li> Hiding IP addresses so they can't be targetted for attacks like DDOS</li> </ul> </li> </ul> Hiring a game developer</h3> Demonstrate passion</li> Read "cracking the coding interview"</li> </ul> Sophia Blow</h2> Braid, The Witness</em></h2> Iteration is king. More you optimize the less you can modify.</li> Programming for fungibility to allow for faster iteration.</li> Program for a decoupled system, as much as possible. This includes game mechanics, types of puzzles, etc</li> Doing programming as a game designer, you can more closely point the best way forward in the game design as you know exactly what's possible</li> Programming as a designer also helps you explore more of the design space than you might otherwise have. Eg) in Braid the enemies unaffected by your time travel.</li> Good design can often be a search space with a seed value and things that seem like time wasted, but that's part of the journey.</li> </ul> Ron Gilbert</h2> Secret of Monkey Island, creator of SCUMM</em></h2> For Maniac Mansion, C64 was hooked to a Unix/Vax machine with hotloading to C64. This is how they could swap out and do fast iterations</li> To learn how to do an interpreter, he banked out BASIC ROM to RAM and then manipulated it. Cool hack.</li> For pathfinding, used dijkstra's algorithm with rectangle shapes. This evolved over time.</li> Later games used seamlessly stiched-together soundtracks that changed as the player progressed.</li> Cutscenes, which came from Lucasfilm games, were originally hand-scripted. They had to "cheat" a little so the player couldn't interrupt the cutscene with their own movement. Eg) Making the character the player was following appear far enough ahead of the player so that the player couldn't run in front and interrupt animation</li> </ul> Travel log (Blacksburg) 2015-12-03T00:00:00+00:00 This is the first of posts detailing my current trip, which will take me around southwest Virginia, Seattle, Hawaii, Australia, New Zealand, Hong Kong, and Japan.</p> My first stop was Blacksburg, home of Virginia Tech. I lived in Blacksburg from 1996 - 2009, so it was both familiar and a little alien, as it hasn't stood still since I've been gone. Still, a great time to reconnect with friends that I haven't seen in years.</p> Virginia Tech</h1> </p> Torgersen Hall at Virginia Tech</em></p> It was nice to take a stroll around campus and see all the familiar buildings. Even new building felt familiar with the use of Hokie Stone, a distinctive limestone taken from the local quarries.</p> The students were just coming back from Thanksgiving break, so when I first got here there were very few people around, giving it a ghost town feeling. Not in a bad way, though. I found it peaceful and quiet.</p> War Memorial Chapel</em></p> When you're walking around the Tech campus, you can't help but notice the Burruss Hall building. It's a giant building that is both familiar and imposing. If you've seen any brochures about Virginia Tech, it's likely you've already seen it.</p> </p> Staring up at Burruss Hall</em></p> If you step back from Burruss Hall and look down, you'll notice a ring of stones. This is the memorial for those killed in the shootings on April 16, 2007</a>. Each stone has a name of someone who was killed that day.</p> </p> Rather than feeling overtly sad, the memorial feels more like a place to recognize and pay respect to the lost while at the same time giving the living space to move on and go about their lives. Still, the losses are felt, and the lost remembered.</p> </p> Blacksburg</h1> </p> Look out at Blackburg from the Virginia Tech library</em></p> Virginia Tech is surrounded by the town of Blacksburg. It's a fairly small affair, nestled into the hills of Virginia. Downtown Blacksburg is only a few streets wide, with a few dozen restaurants and shops packed together before the streets stretch into the neighboring residential areas.</p> Because of its relatively small size, the student population inevitably develops a fondness for certain restaurants. Some of these became institutions that would go on to serve each new batch of students. One such institution is the Bollo's cafe and bakery.</p> </p> Bollo's, a Blacksburg institution (with photogenic dog)</em></p> It was good to see that it's still around. As an undergrad student at Virginia Tech, I spent many a night here laughing with friends, going on dates, playing games, and drinking the strongest coffee I'd ever had. Sorry Seattle, you weren't my first latte.</p> Speaking of Seattle, which is currently being swallowed by the efficient apartment complex, Blacksburg still has lots of great houses in prime locations. The house I'm staying in is rich with character, created sometime in the early 1900s. Its heating has been modernized, but it still has the history intact. In the basement is a shower for the field workers to wash off.</p> </p> It was great to see my Blacksburg friends again. It felt like taking a slow bath in the familiar before rising to walk off to the unknown. A perfect way to start the trip. I can't wait to see what adventures lie ahead.</p> Building a simple JIT in Rust 2015-12-01T00:00:00+00:00 The other day I threw together a simple Just-In-Time compiler (or JIT, for short), and I thought it'd be fun to show the steps I did. With this, it should be possible to create a page of executable memory, write some machine code into it, and then treat it like a function call from Rust.</p> Let's get started! You can grab my JIT project</a>, if you want to follow along. I've only tested this in OS X, but it should adaptable for other platforms.</p> Updated 12/2/2015:</strong> Thanks to some reader comments, I've updated the code below to not use malloc and instead use posix_memalign directly as the allocation step, which avoids leaking memory.</p> Creating executable memory</h1> For our JIT, we'll need to first allocate the memory that will hold our JIT'ted code. The most important thing here is to create a block of memory that's executable so that we can later jump into it, run code, and then return back out.</p> To do this, we need a few functions from the standard C library, which we can access through the libc</code> external module.</p> // added to Cargo.toml [dependencies] libc = "0.2.2" // main.rs extern crate libc; </code></pre> Once we've pulled in libc, we need to allocated memory. Not just any memory, we need to allocate aligned</em> memory. Some operating systems, like OS X, require that executable code memory starts at a particular alignment. For example, that the address is exactly a multiple of 0x1000.</p> const PAGE_SIZE: usize = 4096; // ... unsafe { let mut page : mut libc::c_void = mem::uninitialized(); libc::posix_memalign(&mut page, PAGE_SIZE, size); } </code></pre> Once allocated, we now have memory we can safely jump to. Well, almost. Before we can run code in our newly-allocated memory, we need to enable executing code in this area of memory.</p> unsafe { libc::mprotect(page, size, libc::PROT_EXEC \| libc::PROT_READ \| libc::PROT_WRITE); } </code></pre> Now we're ready. We have something we can write into and then jump into. Since running JIT code is basically "no man's land" without any safeguards, it's easy to get yourself in trouble. One step that I add is to also fill the memory block with the RET</code> instruction, which will let us return from our function even if we happen to accidentally run other memory in the block.</p> extern { fn memset(s: mut libc::c_void, c: libc::uint32_t, n: libc::size_t) -> mut libc::c_void; } // ... unsafe { memset(page, 0xc3, size); // prepopulate with 'RET' calls (0xc3) } </code></pre> We've allocated our JIT memory, aligned it, set it as executable, and then filled it with the RET</code> instruction. We're ready to write some code into it.</p> Note:</em> You'll notice we don't explicitly call libc::free()</code> after we allocate. For simplicity, we treat the JIT'ed functions we create as long-lived, but a full-blown JIT would likely use better memory hygiene.</p> Getting ready to write our first program</h1> To write our first JIT program, let's first make a way to more easily work with our memory. In its current state, it's a raw C void* pointer, but this is a bit unweildy as we don't have a lot of support to work with that in Rust. To make something that feels more natural, we'll create a new struct type that will hold our pointer and allow us to create indexing functions for easier access. This will let us access the memory like m[0] = 0x10</code>.</p> Let's create the struct we'll use:</p> use std::mem; struct JitMemory { contents : mut u8 } fn alloc() -> JitMemory { let contents: mut u8; unsafe { //note: allocate 'page' as before //then, transmute contents = mem::transmute(page); } JitMemory { contents: contents } } </code></pre> We've got the JitMemory struct which will hold the memory we've allocated, and we use the mem::transmute</code> call to convert our raw void pointer to a raw u8 pointer, which will be easier to work with in our next step.</p> Now that we have the struct, let's create some indexing functions.</p> use std::ops::{Index, IndexMut}; impl Index<usize> for JitMemory { type Output = u8; fn index(&self, _index: usize) -> &u8 { unsafe {&self.contents.offset(_index as isize) } } } impl IndexMut<usize> for JitMemory { fn index_mut(&mut self, _index: usize) -> &mut u8 { unsafe {&mut self.contents.offset(_index as isize) } } } </code></pre> With these in place, we can now more easily write instructions into memory. Before we do, let's put everything we'd done so far into a constructor for our struct:</p> impl JitMemory { fn new(num_pages: usize) -> JitMemory { let contents : mut u8; unsafe { let size = num_pages PAGE_SIZE; let mut _contents : mut libc::c_void = mem::uninitialized(); // avoid uninitalized warning libc::posix_memalign(&mut _contents, PAGE_SIZE, size); libc::mprotect(_contents, size, libc::PROT_EXEC \| libc::PROT_READ \| libc::PROT_WRITE); memset(_contents, 0xc3, size); // for now, prepopulate with 'RET' contents = mem::transmute(_contents); } JitMemory { contents: contents } } } </code></pre> Writing our first JIT program</h1> With our new constructor and indexing functions in place, we can write our first JIT program. The simplest "hello world" that I use when doing JIT is a function that takes no parameters and returns a simple value.</p> We can do this with two assembly instructions:</p> MOV RAX, 0x3 ; move our return value (0x3) into RAX, ; the register in x64 used for return values RET ; return from the function call </code></pre> Great, now we just need to write this program into our JIT memory. But wait, we don't have an assembler :)</p> Not to fear, there are plenty of assemblers to get us started. There are even online assemblers</a> you can use. Let's plug the first line MOV RAX, 0x3</code> into our online assembler.</p> The top line of the result is the raw hex. This is what we want. These are the actual bits we'll be writing into memory for our function: 48C7C003000000</p> With these bytes, and the RET instruction we already filled our memory with, we now have our full function. We can use our indexing functions to write this out:</p> let mut jit : JitMemory = JitMemory::new(1); // allocate a page of memory jit[0] = 0x48; // mov RAX, 0x3 jit[1] = 0xc7; jit[2] = 0xc0; jit[3] = 0x03; jit[4] = 0x00; jit[5] = 0x00; jit[6] = 0x00; </code></pre> Turning our memory into a function</h1> We have an executable block of memory with the code for our function filled in. The last step is to turn this into a Rust function we can call. We do this by doing another mem::transmute:</p> fn run_jit() -> (fn() -> i64) { let mut jit : JitMemory = JitMemory::new(1); jit[0] = 0x48; // mov RAX, 0x3 jit[1] = 0xc7; jit[2] = 0xc0; jit[3] = 0x03; jit[4] = 0x00; jit[5] = 0x00; jit[6] = 0x00; unsafe { mem::transmute(jit.contents) } } </code></pre> And with that, after a few handfuls of unsafe calls and transmutes, we have our function. The only thing left is to call it:</p> fn main() { let fun = run_jit(); println!("{}", fun()); } </code></pre> Debugging</h1> When working with a JIT, it's inevitable at some point something will go wrong, and we'll need to dive into the debugger. Luckily, Rust works with LLDB out of the box.</p> Here are some helpful LLDB commands to get you started:</p> Start the JIT in LLDB:</p> lldb target/debug/rustyjit</code></p> Set a breakpoint for the line that contains the call into our JIT code:</p> > br set -line 64</code></p> Run to our breakpoint:</p> > run</code></p> Now that we're about to run our JIT code, there are a few things we can do.</p> The first one is to verify the address of our code:</p> > p fun</code></p> Unfortunately, that's not the most helpful thing in the world. What we often want is look at the memory behind the function:</p> > mem read fun</code></p> This gives us back something like this:</p> 0x100804000: 48 c7 c0 03 00 00 00 c3 c3 c3 c3 c3 c3 c3 c3 c3 H??....????????? 0x100804010: c3 c3 c3 c3 c3 c3 c3 c3 c3 c3 c3 c3 c3 c3 c3 c3 ???????????????? </code></pre> That's better. Now we can see if the bytes we expect to be there are in fact there. If we wanted, we could update memory using mem write</code> if something was out of place.</p> We can also use the built in disassembler to read back out what the asm instructions are for our function:</p> > dis -A x64 -s fun</code></p> Finally, we can continue with the program:</p> > cont</code></p> Lots ahead</h1> Now that we have a tiny JIT and a way to debug it, the sky is the limit. Turning source code into machine code is the heart of any compiler, and with these few additional steps that compiler could be made to output code that we can run directly.</p> Porting a C# NES emulator to Rust 2015-11-20T00:00:00+00:00 Many years ago, I created a simple NES emulator in C#</a> using SDL and Mono. For fun, I wanted to see what it would be like to port that emulator to Rust. A couple weeks later, after poking on it on and off between packing for a move, I was able to get it working</a>.</p> </p> What I did</h1> Loading a binary file</h2> Loading the types of files commonly used in emulators requires us to work with binary files. To do so, we need a way of loading the raw data both as just plain bytes and also as numbers. Rust, by default, comes with a fairly slim API</a> for dealing with binary data. No matter, Rust is also very flexible and allows you to build on additional parts to the API to suit your needs.</p> I started there, and built out a few functions that would let me read in the numbers from the file both using big-endian and little-endian.</p> pub trait BitReader { fn read_u32_be(&mut self) -> Result<u32, io::Error>; fn read_u32_le(&mut self) -> Result<u32, io::Error>; } impl BitReader for File { fn read_u32_be(&mut self) -> Result<u32, io::Error> { let mut buffer = [0; 4]; try!(self.read(&mut buffer)); Ok(buffer[3] as u32 + ((buffer[2] as u32) << 8) + ((buffer[1] as u32) << 16) + ((buffer[0] as u32) << 24)) } fn read_u32_le(&mut self) -> Result<u32, io::Error> { let mut buffer = [0; 4]; try!(self.read(&mut buffer)); Ok(buffer[0] as u32 + ((buffer[1] as u32) << 8) + ((buffer[2] as u32) << 16) + ((buffer[3] as u32) << 24)) } } </code></pre> Once I had the trait, I could import it where I wanted to use these extra read capabilities on File.</p> With these extra methods in place, I could plow through the binary file loading with a mix of the File read functionality and the BitReader trait.</p> I did notice I used the odd concoction buffer.iter().cloned().collect()</code> to convert from an array to a Vec of the same contained type:</p> for _ in 0..(num_prg_pages4) { let mut buffer = [0; 4096]; try!(f.read(&mut buffer)); prg_rom.push(buffer.iter().cloned().collect()); } </code></pre> There may be a more Rust-native way of doing it. Personally, I think it'd be nice to be able to read directly into a Vec, so you could elide that step.</p> Side-by-side porting</h2> One of the first things I did after I had file loading in place was to start porting over the CPU emulation. Because I wanted the code to be fairy clean, I opted to not do a literal port. Instead, I opened two separate editor windows: one for the original C# source, and one for the new Rust source. Tedious, but I hoped that a port focused on what would fit better with Rust would be less error-prone.</p> There's another reason not to do copy/pasting of code when porting: too many little things needed to be changed to the Rust style. For example, Rust-friendly names often differ from the C# equivalent (eg the ReadChrRom method would need to be read_chr_rom to match Rust naming conventions). Like-wise, little things that help readability in Rust, like the "no parens around if guards" means that copy/pasted code would need to be touched up. All told, a minor frustration, but it gave ample time to make the code look a bit cleaner in the transfer so likely a net win.</p> Feeling more comfortable with Rust's use of expressions, I could write code in a more dense way. For example, here is some code from the ROR or Rotate Right opcode in C#:</p> if ((valueholder & 0x1) == 0x1) bitholder = 1; else bitholder = 0; valueholder = (byte)(valueholder >> 1); if (carry_flag == 1) valueholder = (byte)(valueholder \| 0x80); carry_flag = bitholder; if ((valueholder & 0xff) == 0x0) zero_flag = 1; else zero_flag = 0; if ((valueholder & 0x80) == 0x80) sign_flag = 1; else sign_flag = 0; </code></pre> and here's what it became in Rust:</p> let bit = (value & 0x1) == 0x1; value = (value >> 1) & 0x7f; value += if self.carry {0x80} else {0}; self.carry = bit; self.zero = value == 0; self.sign = (value & 0x80) == 0x80; </code></pre> Some of these changes could have been done in the original C#, but I didn't notice them at the time. Still, I think the result Rust is still pretty readable, even though it's more compact.</p> While we're talking about bit rotation, one of the annoyances I experienced working on bit manipulation code is that Rust is more vigilant about overflow checking. In hindsight, this is actually a good thing, but coming from languages like C# and JavaScript that are less vigilant, it was a bit of a headache to adjust to. I ended up doing some of my own bit masks so I didn't have to worry about it, but I noticed that they've added a number of "wrapping" operations in Rust 1.4</a> that allow you to work around the overflow checks.</p> Hitting the first ownership error</h2> A few days into the port, I hit my first ownership error. Here's a quick rundown of the NES, so you can see why that might be so:</p> CPU: The CPU :)</li> PPU: The NES's graphics unit</li> MMU: Not technically not called the MMU, but let's call it an MMU for our purposes</li> Cart: The program to run</li> </ul> Now that we have the parts, here's how they talk to each other:</p> CPU asks the MMU for an instruction, and the CPU may also write back to memory</li> PPU reads memory from the Cart, and can also transfer memory from MMU to graphics memory</li> MMU needs to read information from the Cart and can also read and write from the PPU</li> </ul> All the subsystems, in essence, need to be able to talk to each other at some level. Which is exactly how I set up the C# project. Singleton objects with references to each other. When I ported over to TypeScript a few years ago, similar idea.</p> But in Rust, ownership needs to be clear. If you call a method, you can't also pass something you own to that method, like so:</p> foo.x.y(foo.x.z) </code></pre> The net result is that I went back to the drawing board and thought long and hard about what was actually</em> necessary for each component. After a couple of refactorings, I think I landed on a solution that's both cleaner and clearer. Every object has one owner, and I don't do any ref cells or ref counting.</p> Deriving debug</h2> There are a few tricks that I loved in Haskell that I wished other languages would pick up. One of them is the ability to automatically be able to print out objects for you.</p> In Rust, you can do this using the 'derive' command:</p> #[derive(Debug)] struct Foo { x: u32 } //... let foo = Foo { x: 10 } println!("{:?}", foo); </code></pre> By deriving Debug, you have the ability to output a debug format of that structure, which is output using {:?}.</p> This is very handy to help bootstrap a project, when you quickly want to look at the contents of what is working so far. After a while, I found I ended up removing these helpers and putting in my own, customized implementations.</p> SDL</h2> Using SDL was dead simple. I used the SDL2 Rust bindings</a>. After a few tries to follow the directions to install SDL2 carefully, I had it working in both Windows and OS X. After it's installed, the bindings "just work".</p> I'd never used SDL2 before, but after looking through the examples directory</a>, especially the texture sample, I was good to go.</p> What I didn't do</h1> Chase crashing bugs</h2> After wrestling a few ownership errors, I largely spent my time on correctness. I ended up having a few hard-to-find issues where I flipped the boolean logic on a check or discovering I forgot to implement a piece of the machine that was required. After many hours pouring over the code side by side, or looking at the output log, I finally managed to get the outputs similar.</p> Just as in the C# and JavaScript versions of the emulator, I didn't worry about any crashes. A few times I had an arithmetic overflow or error accessing a vector out of bounds, but those were easy to fix.</p> Optimize</h2> After I got the emulator working, I let it rip. The result was something unplayably fast. Not bad for a first try.</p> In hindsight, this isn't too surprising. The CPU gets to work directly with the MMU. Ditto with the MMU and the PPU. The systems can call functions in each other without any abstraction, VM, or GC. By default, the core structs sit on the stack.</p> Using Rust had nudged me into doing what would have made sense for a decent C-based approach, without the risks. Of course, it's not like Rust prevents you from adding in a hefty abstraction layer, but the defaults seem to push you in the right direction.</p> Unit test</h2> For this to feel like a real Rust project, though, it needs some tests. I asked Patrick Walton</a>, the author of probably the first NES emulator for Rust</a>, about testing emulators with Rust unit tests, but he didn't use them for his project either.</p> Still, I suspect there's a way of creating fairly testable, and still performant, parts of the emulator. For example, opcodes could each have their own unit test if there was a way, possibly with traits, to decouple them from working so closely with the MMU.</p> Add audio support</h2> In Patrick's video about his emulator</a>, he mentions that he took it on to show that Rust can do real-time processing. For an emulator, that means that the audio is emulated properly and sounds correct. I haven't added any audio support, but it would be a fun project at some point.</p> Precisely emulate the machine</h2> The never-ending rabbit hole for any emulation project is to see just how precisely you can emulate the hardware. There's plenty of work here.</p> What I've learned</h1> I'm getting a little more comfortable with Rust. Thinking through an ownership refactoring can still take a while, but it's starting to feel like the hurdle you take on so that you can worry less about safety and performance. I still have a ways to go before it becomes second nature.</p> Cargo shines again, and the SDL bindings were easy to put in place and worked as expected. No complaints there.</p> Feels like I'm still scratching the surface. There are likely some slick spots to use traits and macros to cut down on the redundancy and coupling, repectively. But I'm happy with the progress.</p> Steam Machines enter the market 2015-11-11T00:00:00+00:00 This week marks the release of the first official Steam Machines</a>. For the last five months, I've been a proud owner of the Alienware Alpha</a>, one of the three Steam Machine types now available. Knowing what could be possible, I've been curiously awaiting the official launch.</p> The plan</h1> Valve's motivation for creating the Steam Machine seems pretty straight forward: move to the TV and compete more directly with game consoles. Currently, that space is shared by the big three: Sony, Microsoft, and Nintendo. Why not make that the big four instead, with Valve in the mix?</p> To get there, Valve has started with three initial devices.</p> </p> The Steam Controller is a gamepad with two round touchpads and a set of traditional gamepad buttons. It can be configured to work as a normal gamepad for games with gamepad support. Alternatively, the touchpads can be used just as touchpads to play games that work with a mouse instead of gamepad.</p> </p> The Steam Link is something of a gaming bridge between the PC and your TV, allowing you to stream games from your PC over to your TV. At around $50, it's pretty inexpensive.</p> </p> Alienware Steam Machine</em></p> Lastly, the Steam Machine. The Steam Machine itself isn't actually a thing. Rather, it's a PC specification that hardware vendors can build to. Out of the gate, three hardware vendors have already produced a Steam Machine: Alienware (pictured above), Zotac, and Syber. Steam Machines also come with a new operating system called SteamOS</a>, a form of Linux that's been optimized for gaming.</p> The right plan?</h1> Now that we know the plan, is that plan the right plan? Specifically, does the Steam Machine have a home?</p> It's clear to me that a device like the Steam Machine has value in the market. It's smaller than a gaming console, more flexible, and just as powerful. I've spent countless hours gaming, reading news, watching movies, and even programming on mine. I've even stopped using my Playstation 4, opting instead to do all my gaming on the Alpha.</p> But as I look at the Steam Machine release this week, I have some concerns.</p> Concern 1: No flagship Steam Machine</h2> The console market is notorious for razor-thin margins on release hardware. Traditionally, consoles are often sold as loss leaders</em> in their early years. Meaning, they are sold below the manufacturing costs</a>. The thought here is the licensing profits from the games bought by initial adopters will help offset the loss. Once the console becomes embedded in the market, manufacturing improves, and over many, many small iterations, the console itself becomes cheap enough to make a profit.</p> Consoles are always, with rare exception, first-party. This means that the company who built the software stack that the games run on is also the company who owns and sells the hardware. Nintendo, Microsoft, Sony, and even previous game vendors like Atari and Sega, all made first-party consoles.</p> Rather than being first-party, Valve has opted for an approach that's closer to non-console markets like phones: third-party hardware. Early software plays like Microsoft with MS-DOS and Windows, and more recent ones like Google's Android platform are some examples here. The gist is that the standard platform works against a standard hardware spec. It doesn't really matter who the vendor of the hardware is, so long as it matches the spec.</p> That might look good on paper for Valve. Since they're new into the hardware game, why not let the vendors who know what they're doing do it?</p> It's still a risky play, and, I'd argue, one that's potentially riskier than partnering with one vendor to create a Steam-branded first-party machine. Some reasons why:</p> Third-party vendors can't afford to sell loss leader machines.</strong> This means the machines will necessarily be lower-powered than they could be. This narrows the performance per $ gap between this batch of Steam Machines and established consoles like Playstation 4 and Xbox One. Which, in turn, hits customers in the wallet. While the Steam Machines are cool pieces of technology, they are also spendy pieces of technology. For example, the Zotac machine is $900 at the time of writing. At that price, I could buy 3 WiiUs</strong> -or- an Xbox One and</em> a Playstation 4</strong>, with money left over for games.</p> Third-party purchase experience is going to be whatever that vendor makes available.</strong> For example, this morning when looking at the new Alienware machine I got website errors on the Dell.com site. Of course, some of this will get better over time, but it's difficult to see how both a higher price tag and a worse purchasing experience is going to help the Steam Machine brand.</p> As an aside, you might think that being third-party would help you create the Steam Machine of your dreams, but as of the time of this writing, you can't even configure an Alienware machine to use SSD. For a gaming machine. No SSD. What? If you're like me, that's the first thing you'll want to upgrade. I asked the Alienware rep over Twitter about it and got:</p> </p> Again, likely something that will be fixed in time, but not the best initial experience and one that will turn-off some gamers from the Steam Machine idea.</p> Lastly, Valve's Steam brand is already better than the third-party vendor brands</strong>. Steam is the</strong> major name in PC gaming today. Period. Who is Zotac or Syber? I honestly have never heard of them. Why play second fiddle to brands that aren't as strong as your own?</p> Concern 2: SteamOS</h2> The motivations here also seem straightforward: Valve gets to create a platform that a) they control and can tweak to their hearts content, b) supports a standard set of APIs that other vendors can leverage and is possibly less of a moving target than the competing APIs, and c) is free.</p> With c), Valve helps ensure that vendors can cut costs and deliver less-expensive Steam Machines. But the whole play here is very confusing.</p> Remember that a Steam Machine is a whole PC. Unlike a game console like Playstation 4, these guys can be a full Windows 10 machine if you install Windows on them. And Windows already</em> runs Steam. So, in fact, a Steam Machine with SteamOS is actually a less powerful experience than taking SteamOS off and putting Windows in its place.</p> As for the advantages, again it seems like Valve is trying to take a page from the Android playbook.</p> Unfortunately, the current market is decidedly different than the market Android found itself in. With the release of the iPhone, and its rise in the smartphone market, you had a gap created with a lot of phone vendors wanting an iPhone-like experience without an easy way to get it. By creating a comparable work-alike, many vendors could sign up and build out the Android ecosystem. Is SteamOS entering a market of hardware vendors looking to enter the game console market? Having only one major PC brand, Dell, signing on to ship hardware with SteamOS is an indicator this may not be the case.</p> Likewise, are game developers scrambling to get onto SteamOS? Guessing by the size of the game lists, SteamOS has about a fifth the number of titles available for Steam on Windows. That, by itself, is actually pretty impressive. But, consoles really live and die on the titles themselves. Let's do a quick rundown of my favorite kind of game, RPGs:</p> Witcher III: no</strong></li> Final Fantasy XIV - A Realm Reborn: no</strong></li> Dark Souls II: no</strong></li> </ul> In fact, I struggled to find triple-A RPGs from the last few years that run on SteamOS. What's telling is that all three are available on Windows as well on consoles, where they use gamepad support. In theory, this should be a solid group of games to show off your powerful new platform.</p> In fact, Fallout 4, which was also released this week, does not have SteamOS support</strong>. Likely to be one of the biggest games of the year, and it won't run on Valve's newly-shipping machines.</p> I suspect that the tech-savvy buyers will do exactly what I did when I got my machine: take out the hard drive, put in an SSD, and install Windows.</p> Concern 3: Partnerships, or lack thereof</h2> Concerns 1 & 2 above could be summarized as "Steam Machine lacks partnerships". The more high profile hardware companies competing to create the best Steam Machine, the better the end result. The more high profile game companies making games for the Steam Machine, the more SteamOS feels like an established platform like Playstation 4 and Xbox One.</p> Partnerships are a catch-22. It's difficult to get them without first showing that your platform is a big enough draw to support the investment it takes to port a game or build a machine. Without the partnerships, the platform never grows to that size.</p> Road ahead</h1> It's too early to say whether or not Valve will be successful. This release of the Steam Machine is muddied by a confusing story and a lack of partnerships, but there is still time to improve the experience. When Steam first came out, it had few games and many thought it was a weak idea. It, of course, grew to the giant it is today.</p> Like building a game distribution mechanism like Steam, a game console is also a long play. It takes years and lots of cash to build up a brand in the console market. Microsoft famously rumored to have spent north of a billion dollars to establish the first Xbox</a>. From outward appearances, Valve is trying to compete with the console on a fraction of that budget.</p> Will Valve and its partners have the patience to build their own niche? Time will tell.</p> Learning to 'try!' things in Rust 2015-11-04T00:00:00+00:00 I started learning Rust in earnest a few weeks ago. Coming from C++, a fair share of the idioms felt right at home. There was clear memory management and an eye towards lightweight abstractions.</p> But when I started looking around for how to do exceptions, I was surprised to find that Rust had none. In its place were types like this:</p> fn foo_maker() -> Result<Foo, io::Error> { ...} </code></pre> I recognized this style, having come from playing with languages like Haskell, but despite doing some functional programming before I wasn't entirely sure how to approach it. At first blush, the Result value seems pretty straightforward, it either returns something of Foo or an io::Error. We know which one it returns by pattern matching out the value:</p> let result = foo_maker(); match result { Ok(foo) -> foo, Err(err) -> println!("Error: {:?}", err) // Ooops, we hit an error, tell the user } </code></pre> Great, you might think, that's cute. Well, at least at first. Real code tends to have a few reads in a row:</p> let num_chans = read_u16_le(&mut f); let samples_per_sec = read_u32_le(&mut f); let avg_bytes_per_sec = read_u32_le(&mut f); let block_align = read_u16_le(&mut f); let bits_per_sample = read_u16_le(&mut f); </code></pre> Since we're reading from a file, any of these reads could hit an error. Perhaps the hard drive dies, or we prematurely hit the end of the file. Because of that, each function returns a Result that we later have to pattern match to get the value out. This means we have to pattern match every result. Yuck.</p> If you poke around on the internets, you'll find a sneaky way around this using .unwrap()</code>. The .unwrap()</code> call does just what it name implies: it unwraps the Result and gives us the value, assuming the result is Ok. What if it errored with an Err? Well, then the application panics.</p> let num_chans = read_u16_le(&mut f).unwrap(); let samples_per_sec = read_u32_le(&mut f).unwrap(); let avg_bytes_per_sec = read_u32_le(&mut f).unwrap(); let block_align = read_u16_le(&mut f).unwrap(); let bits_per_sample = read_u16_le(&mut f).unwrap(); </code></pre> If you felt a cold shiver run down your spine looking at that, good on you :). Now, let's talk about how we actually fix this code.</p> Do or do not. There is a try!</h1> Looking a bit further in Rust examples, I learned about another way to call functions using the try!</code> macro. Maybe, I thought, try!</code> would save us. So I tried it out:</p> use std::io; use std::fs::File; fn main() { let mut f = try!(File::open("foo.txt")); } </code></pre> ...and BOOM</strong></p> <std macros>:5:8: 6:42 error: mismatched types: expected `()`, found `core::result::Result<_, _>` (expected (), found enum `core::result::Result`) [E0308] <std macros>:5 return $ crate:: result:: Result:: Err ( <std macros>:6 $ crate:: convert:: From:: from ( err ) ) } } ) read_a_file.rs:5:17: 5:44 note: in this expansion of try! (defined in <std macros>) <std macros>:5:8: 6:42 help: run `rustc --explain E0308` to see a detailed explanation error: aborting due to previous error </code></pre> Okay, seriously? I feel like I jumped in the Haskell pool after drinking a C++ template error cocktail. What is</em> that?</p> Luckily after asking around, I was finally able to piece together what was going wrong. The hint lies in these lines:</p> expected `()`, found `core::result::Result<_, _>` </code></pre> Ah, some type mismatch is happening. Our main function doesn't return anything, which is the ()</code> in the 'expected' above, but try!</code> only works inside functions that return a Result</code>. This might be the first time in my life I've ever seen a function in a C-style language that was callable only inside functions of a particular type.</p> Shouldn't be too hard to fix, just put it in a function that returns Result, right?</p> use std::io; use std::fs::File; fn read_file() -> Result<(), io::Error> { let mut f = try!(File::open("foo.txt")); Ok(()) } fn main() { read_file(); } </code></pre> Yup, that did it. Then I promptly closed my editor and walked away. I mean, this thing just felt alien</em>. Unnatural. Gross.</p> What I didn't realize was that a few weeks later I'd grok the genius in this approach.</p> Genius #1 - Unlike try/catch, try! is precise</h2> Let's look at a real-life example from a project I'm working on:</p> fn load_data(fname: &String) -> Result<Data, io::Error> { let mut f = try!(File::open(fname)); let num_pages = try!(f.read_u8()) * 4; let mut pages : Vec<Vec<u8>> = Vec::new(); for _ in 0..num_pages { let mut buffer = [0; 4096]; try!(f.read(&mut buffer)); pages.push(buffer.iter().cloned().collect()); } // ... } </code></pre> It may take your eyes a minute to adjust to the new syntax, but knowing what we know, we can look at the code above and know exactly which functions might give us errors. The try!()</code> call even works on calls that are part of an expression, like try!(f.read_u8()) * 4</code>.</p> Compare this to traditional try/catch blocks and how they tend to get put around whole sections of code when in reality only parts of the code blocks may throw exceptions. The end result is code that's very clear and doesn't muddy exceptional code and value-based code together.</p> Genius #2 - Feels like exceptions</h2> With this little bit of syntax, we get something that is value-based, but still feels like exceptions.</p> Let's say your task is to write code that will load a variety of audio file formats. In each audio format loader, you have various subsections that get loaded (eg, a header, a metadata chunk, a chunk of audio data, etc). It'd be great if you could create specialized functions for each. If there are any errors, you just want to bubble that back out to the original caller loading the file. The resulting code may look something like:</p> fn load_header(file: &mut File) -> Result<Header, io::Error> { Ok(Header { header_block: try!(file.read_u32()) }) } fn load_metadata(file: &mut File) -> Result<Metadata, io::Error> { Ok(Metadata { metadata_block: try!(file.read_u32()) }) } fn load_audio(file: &mut File) -> Result<Audio, io::Error> { let header = try!(load_header(file)); let metadata = try!(load_metadata(file)); Ok(Audio { header: header, metadata: metadata }) } </code></pre> We can refactor our file loading into helper functions, and we continue to use the same familiar try!</code> pattern. How does this work? The trick is that all the Results agree that io::Error</code> is the error case. If one of my helper functions hits a bad read and has to return an error, that error naturally bubbles all the way out without calling any additional code.</p> The end result is that we have precise syntax that is transparent about errors at every layer, stays pretty lightweight, and gives us the familiar "bubble out" feel of languages with exceptions.</p> Trade-offs</h1> Of course, there are trade-offs. We end up repeating try!()</code> in places we might have just wrapped in one big try/catch block. For example, the fix to our earlier use case might look like this:</p> let num_chans = try!(read_u16_le(&mut f)); let samples_per_sec = try!(read_u32_le(&mut f)); let avg_bytes_per_sec = try!(read_u32_le(&mut f)); let block_align = try!(read_u16_le(&mut f)); let bits_per_sample = try!(read_u16_le(&mut f)); </code></pre> Personally, I'm happy making the trade-off for the added precision.</p> We also saw another trade-off earlier. If you use try!()</code> in a function that doesn't return a Result, you get an incomprehensible error message. But once you know what's going on, you can use it with confidence.</p> Telling the truth in code, function signatures and transparency 2015-10-28T00:00:00+00:00 When people talk about reasons to use a type system, there are always a couple usual suspects: error checking, documentation, and if you're an IDE person like me, tooling/editor tricks that use type information, like auto-complete.</p> But there's another aspect to the types of function signatures that's related to documentation but is perhaps stands on its own: transparency.</p> What is transparency?</h1> For this post, transparency helps us answer a few questions:</p> What does this function require?</li> What does this function return?</li> What is this function going to do with what I give it? Can it modify it?</li> Can it delete it?</li> </ul> </li> Can this function fail? If so, how?</li> Does this function do anything besides use its value? ie) does this function "change the world" in some way?</li> </ul> </li> </ul> Why is transparency important?</h1> A way most of us can relate to transparency is in the relationships you have in your life. When you're not transparent with another person -- keeping all of your thoughts to yourself, never saying when you're angry, etc -- how does the other person ever know how to react to you?</p> Transparency in code works similarly. When a function is transparent, you know what you need to do when the function is called, and you know how to react when the function is called. Let's look at a few examples. In each example, we're going to talk about a function that takes in an employee name and writes that name to a file. Examples</p> Python</h2> def writeToFile(name): # write out to the file </code></pre> We'll start with the simplest form of transparency: minimal transparency. Looking at what we have it's tempting to say we don't know anything at all, but that's not quite true. We do know the function name and that it takes a single parameter.</p> Things we don't know:</p> What kind of thing does it take?</li> Can it modify the thing we give it?</li> Can it delete what we give it?</li> Does it return anything?</li> Does it throw any exceptions?</li> Does it have any permanent effect on our system eg) send files over the network, add to a database, launch the missiles?</li> </ul> </li> </ul> Without this knowledge, we resort to aggressively testing, commenting, and using naming and coding conventions to get the idea across.</p> C</h2> int writeToFile(char name) { / write out to the file / } </code></pre> Moving into C, we start to see a bit more information popping out. We know that it takes in a char</code> for the name. Since this also isn't const</code>, we know that it's allowed to change the contents of the string we pass in. We also know it returns an int</code>, likely for error codes.</p> We are starting to see into what the function can do, but there's still quite a bit we don't know:</p> Can this function change our system?</li> Can the name</code> value be deleted?</li> Can the function raise a signal (a C-flavored way of exceptions/events)?</li> And what about those error codes? How do we know the int</code> is for error codes, and not returning the length of data written?</li> If it is for error codes, how do we know which ones it supports?</li> </ul> </li> </ul> We've improved some, but we still lean heavily on conventions, code comments, and testing to ensure that this function behaves correctly. As an aside: one interesting point of C is that it doesn't have exceptions, like many languages, so you're almost encouraged to have a more value-based programming style, though it lacks many common features that help fill out the value-based story (pattern matching, algebraic data types, etc).</p> Rust</h2> fn writeToFile (name: &str) -> Result<()> { // write out to the file } </code></pre> Above is a simple Rust version. If you've never seen Rust before, let's quickly break down it says:</p> The fn</code> keyword creates our function</li> Next, we give the function a name: writeToFile</li> Our function gets one argument called 'name' which has the type &str</code> &str</code> is a type in two parts: str</code> is the simple string type</li> &</code> meaning that this value is being borrowed and is owned by someone else (aka you're not allowed to delete it)</li> </ul> </li> </ul> </li> Finally, we return something of type Result<()></code> Result<()></code> is the Rust way of saying I can return either: a) successfully, without a value (the () above)</li> b) an error</li> </ul> </li> </ul> </li> </ul> The Rust example tells a bit more of the story than the C version. In Rust, everything is immutable by default, so we know from looking at the signature that what we pass in can't be changed. It also won't be deleted, because what is passed is 'borrowed' rather than us giving ownership over to this function. Rust is similar to C in that we don't have exceptions and instead use return values to denote success or failure (read more in Rust Error Handling). Rust encodes the success or failure in the return value, requiring the caller to code defensively around possible errors. This has the effect of encouraging code to be more robust by handling both success and failure where they can occur.</p> An improvement over where we were in C, but we still have a lingering question: does this function have any permanent effect on the system? To be able to answer that question, we go one example deeper down the rabbit hole.</p> Haskell</h2> writeToFile :: String -> IO () writeToFile name = do </code></pre> Since this might be unfamiliar let's take a second to breakdown what we're seeing:</p> We start by declaring the type (which isn't terribly dissimilar from a forward declaration in C)</li> The type has two parts: We take one argument that has a type String</code></li> We return an IO ()</code> [see below]</li> </ul> </li> After the type, we write out the actual implementation of the function</li> Just as in Rust, Haskell is immutable by default. When we say we pass in a String, we're also implying that this string can't be changed or deleted.</li> </ul> Also similar to Rust, in Haskell there are no exceptions, so you must handle all possible values when you call the function. By saying IO ()</code>, this might at first seem similar to the Result<()></code>. In some sense, it is, in that IO carries along any possible errors.</p> It also goes a bit further. In essence, it says: "I'm going to go change the world, and when I come back you need to carry that truth on." In Haskell, if you call a function that performs I/O like our writeToFile function, you must also carry on that fact in your own return type. You have to be fully transparent all the way through the call stack of who is changing the world, even indirectly. This lets you clearly document who is modifying the world and who isn't.</p> Dark side of transparency</h1> But, it can't be all roses, right? Let's look at what I see as the dark side of transparency: encapsulation (or its lack thereof).</p> As good software engineers, we know an important piece of building a system is having good encapsulation. Any module or object worth its weight exposes only necessary functionality to the world and hides the rest. This encourages loose coupling, so that components don't depend on each others' implementation details.</p> In a way, transparency in the type runs counter to information hiding in that it encourages more coupling in the code. Some of this is natural. For example, in the C code, we have to remember what the return values are. If we add a new one, we may have to change the code handling the call as well. Likewise, in Python, if we change the types of parameters we allow, we may have to go update the callers.</p> On the other hand, knowing that a function may fail may be sufficient. In the Rust case, code that handles the call may need to be updated if you change how the function fails, even if it only cares about if it fails.</p> Likewise, in Haskell you may want to drop some probe into your system to catch the result of a function while you're debugging. Such changes can have a ripple effect in your codebase as this probe may change a series of function calls to reflect that I/O has been performed. (Not withstanding workarounds with fun names like unsafePerformIO</code>)</p> All a balance</h1> Types are there to help. Leaning on them not enough means encoding more information in your tests and code comments. Leaning on them too much may mean tightly coupling parts of your system. What the right balance is may change from project to project, or developer to developer.</p> Just as in our relationships with other people, it makes sense to not tell everyone we meet everything about us. It's a balance. Finding the right amount of transparency that encourages flexible, resilient systems is an art.</p> In the last few weeks, in playing with Rust more I've grown to appreciate its pragmatic set of trade-offs. Yes, you don't get to know if the system was changed, but if something can impact the rest of the program, you're encouraged to handle it. Not that there aren't workarounds or ways to easily opt-out, but just as in life, it's all a balance.</p> Lessons from the first 12 Euler problems in Rust 2015-10-09T00:00:00+00:00 !! Spoiler Alert !!</strong></p> I'm going to be talking about solutions to select problems from the first 12 Euler problems. If you haven't solved these for yourself yet, I highly recommend taking the time to solve them on your own first.</em></p> Background</h1> Before I went to Microsoft for a stint in JavaScript</a>, I did my fair share of systems programming in C++. I've built compilers, head-tracking software, signal processing software, and emulators and even did some of my first programming language projects in it and for it</a>.</p> So when word got out four years ago about a new systems language called Rust</a>, I was intrigued. I mean, it was new</em> and it was called rust</em>. Obviously there was some cheekiness involved :)</p> While I've dabbled with it before, this week was the first time I really gave it some dedicated time to learn it. TL;DR:</strong> hey, this is kinda neat!</p> In this post I talk through solving some of the Euler problems in Rust, both de-rusting my programming skills and learning what this Rust thing was all about.</p> If you're interested, you can grab the source code to my solutions</a>.</p> Problem 1</h1> "sum of all multiples of 3 or 5 below 1000"</p> fn main() { let mut answer = 0; for x in 0..1000 { if x % 3 == 0 { answer += x; } else if x % 5 == 0 { answer += x; } } println!("{}", answer); } </code></pre> My solution to problem 1 is pretty straightforward, but you can already start to see some of the characteristic earmarks of Rust. On line 2</strong>, there's a let keyword which creates a variable. After that is the mut keyword for saying that the variable we're creating is mutable, as variables are immutable by default.</p> Line 4</strong> shows off using an iterator. Rust's iterators are lazy, meaning they generate values only as needed. This gives them some cool properties, which we'll see later, but it has the property of being relatively cheap to iterate over them, since we don't generate a full range before iteration starts.</p> Line 5</strong> reminds us that we're not in C. The if doesn't require parens around its condition. Indeed, if you put them there, the compiler warns you that you don't need them.</p> Line 13</strong> shows how to print out a value. Note the '!' after println. If you're like me, you're bound to leave this off once or twice and get an incomprehensible error message. Similar to C and many other languages, you pass println! a format string. Here, we just use "{}". The {} part means to use a default output of the argument we pass in, which in our case just writes out the number.</p> Problem 2</h1> "sum of even fibonacci below 4,000,000"</p> struct Fibonacci { curr: u64, next: u64, } // Implement 'Iterator' for 'Fibonacci' impl Iterator for Fibonacci { type Item = u64; // The 'Iterator' trait only requires the 'next' method to be defined. The // return type is 'Option<T>', 'None' is returned when the 'Iterator' is // over, otherwise the next value is returned wrapped in 'Some' fn next(&mut self) -> Option<u64> { let new_next = self.curr + self.next; self.curr = self.next; self.next = new_next; // 'Some' is always returned, this is an infinite value generator Some(self.curr) } } // Returns a fibonacci sequence generator fn fibonacci() -> Fibonacci { Fibonacci { curr: 1, next: 1 } } fn main() { let mut sum:u64 = 0; for i in fibonacci().take_while(\|&x\| x < 4000000).filter(\|&x\| x % 2 == 0) { sum += i; } println!("sum: {}", sum); } </code></pre> Lest you think that I somehow jumped from beginner level to advanced in a few minutes, rest assured that I used the tried and true method of the copy/paste</a>. In trying to find out how to make an iterator so I could iterate over a fibonacci sequence, I found the source to do just that. Ah, the internet.</p> There's a fair bit going on here, so let's break it down by line.</p> Lines 1-4:</strong> Create a simple struct to hold the current value to be returned as well as the next value. Fibonacci needs at least these two values to continue to calculate each iteration.</p> Line 7:</strong> Now Rust is showing off the fancy. What is going on here? In Rust, a type, like our Fibonacci struct from lines 1-4, can implement a trait. What's a trait? Think of it as a small contract with the compiler. If you can show how your type satisfies all the requirements of the contract, then wherever that trait is required, the compiler knows how to use your type. In this example, we want to use our type in the for..in</code> language feature. In some languages, this wouldn't be possible. In Rust, if we can satisfy the Iterator trait, the compiler has enough information to allow that type to be used in the for..in</code> feature.</p> Line 8:</strong> Ah the problem with copy/paste. As I started writing this blog post, I had no idea what that line was doing. Docs to the rescue. It turns out that this line is saying what the 'associated type' is. With some traits, there's going to be both a) the type that is implementing the trait (here: our Fibonacci struct) and b) an additional type that needs to be known to get the rest of the story. For us, since we're creating an iterator over the fibonacci sequence, we need to let Rust know what the type of each "turn of the crank" on the iterator will output. Each fibonacci number will be a 64-bit unsigned integer, so we use the Rust short-hand type u64</code>.</p> Lines 12-20:</strong> Here's where we describe how to turn the crank for each step of our iterator. We calculate what the next turn will output and then return the current value. This allows us to keep one in reserve ready to be used to calculate the next value, and so on. You can see on line 19 that instead of just returning a u64</code> directly, we instead say that it's optionally a u64</code>. While in our case we iterate forever, using Option</code> here gives us a way to shut off the valve and finish the iterator if we ever wanted to.</p> Notice, too, on line 19</strong> the value is said last and the return statement is elided. This is just shorthand for returning a value from the function.</p> Lines 24-26:</strong> Create a simple function to give us our starting value to begin our fibonacci sequence. You can see a couple Rust-isms here, too. As before, a value by itself is an implied return value. It also shows how to create an instance of the struct without a constructor.</p> Line 30:</strong> We're finally in the main! I've forgotten what problem we were solving by now. Ah right, "find the sum of even fibonacci numbers less than 4,000,000". Let's look at the solution. Since there's a lot going on in this fluent-style approach, let's break it up into steps:</p> fibonacci()</code> - we call our function and get out the default value to start iterating with. Since we already told Rust how to use values of this type as iterators, it gets all the capabilities that iterators have.</p> .take_while(\|&x\| x < 4000000)</code> - that didn't take long, we're already jumping in and using the functionality of the iterator to check each turn of the crank, and if we exceed 4,000,000 stop.</p> The \|&x\| x < 4000000</code> is a simple function that takes in a reference to each element we bind to x, and then we check the value we're handed in the body of our function. If you're coming from C++, this reference style might look a little strange. Aren't you risking someone updating the value? Turns out Rust's references are immutable by default</a>. The reference here also shows us the performance characteristics of .take_while</code>. By this I mean, when we look at the .take_while</code> call, we see it will only pass references, so know ahead of time if we're iterating over big structures, we're not paying the price of copying each one with each step of the iterator.</p> Speaking of performance, it may look like .take_while is going to be exceedingly expensive. Is it consuming all of our fibonacci sequence until we have all the items we need to get to 4,000,000? Luckily, it isn't. Instead, .take_while</code> itself returns an iterator that you can continue your fluent calls on. I think of it like a series of machines in a factory:</p> -> [ ] -> [ ] -> [ ] -> </code></pre> If I start the conveyor up and I pull from the right hand, I'm moving whatever was on the far left all the way through the system. In our case, we're starting up the fibonacci iterator and getting that machine running. It spits out a single fibonacci-shaped number for us. This number chugs along and enters the next machine, which is our machine that checks if it's too big. If it's not, the conveyor chugs along and out drops a single fibonacci-shaped number of the acceptable size.</p> As these machines work in lock-step, we're getting just what we need when we need it rather than trying to run ahead and doing a bunch of work in advance. This allows you compose a lot of steps together and still stay efficient.</p> .filter(\|&x\| x % 2 == 0)</code> - Just like .take_while</code> above, this one checks each number as it passes through. This time, it only lets the number through if it's divisible by 2, and hence even.</p> Now we have all the steps in place, each time we turn the crank of the whole iterator, only fibonacci numbers that are less than 4,000,000 and even will pop out.</p> Line 31:</strong> We've already done all the hard work. Now we just have to sum those numbers together.</p> With that, we're done. While it was quite a mouthful, we can see the compositional nature of Rust, a bit of the type system, and how iterators work in more detail.</p> Problem 3</h1> "highest prime factor of 600851475143"</p> fn is_prime(num:u64) -> bool { for i in 2..(num / 2 + 1) { if num % i == 0 { return false; } } return true; } struct Prime { curr: u64, } impl Iterator for Prime { type Item = u64; fn next(&mut self) -> Option<u64> { let mut new_next = self.curr + 1; while !is_prime(new_next) { new_next += 1; } self.curr = new_next; Some(self.curr) } } // Returns the primes fn primes() -> Prime { Prime { curr: 1 } } fn main() { let mut num:u64 = 600851475143; let mut highest_prime_factor = 0; for i in primes() { if num % i == 0 { highest_prime_factor = i; } while (num % i == 0) && (num >= 2) { num /= i; } if num == 1 { break; } } println!("num: {}", highest_prime_factor); } </code></pre> Very similar to Problem #2 above, I create an implementation of Iterator, so I can use it later. This time, rather than creating a stream of fibonacci values, I create a stream of prime numbers.</p> Just like we had a struct for Fibonacci in Problem #2, I create one here for Prime numbers. Since we don't need the previous one to calculate the next one, I only keep the current value around and then start with that number + 1 when trying to find the next prime.</p> Once we have our iterator, we take each prime, and then try to divide out all its multiples from our give number. What's left should be our highest prime factor. Easy peasy.</p> PS: I know it's probably already too late, but I should have warned you to avert your eyes from my inefficient prime number skills :)</p> Problem 4</h1> "largest palindrome product of two 3-digit numbers"</p> fn is_palindrome(num: u64) -> bool { let s = num.to_string(); for (i1, i2) in s.chars().zip(s.chars().rev()) { if i1 != i2 { return false; } } return true; } fn main() { let mut largest = 0; for i in 100..999 { for j in 100..999 { if is_palindrome(i * j) && ((i * j) > largest) { largest = i * j; println!("Largest: {}, {} x {}", largest, i, j); } } } } </code></pre> Problem #4 also uses some iterators to find the palindrome product (line 13</strong> and line 14</strong>). If it finds a palindrome product, and that new product is larger than what it saw before, it replaces it with the new one.</p> There's one fun trick in this solution on line 3</strong>. We need to check if the number is a palindrome. While you can do this numerically</a>, I couldn't help but use some more iterators. This time, after converting the number to a string, we create an iterator over the characters in the string (using .chars()</code>). We want to see if the number is a palindrome, so one way to test this is that if we reverse the string and if still get the same value, we know we've got one.</p> That's exactly what we do. Here, I use .zip(s.chars().rev())</code> to combine the iterator that goes forward over the characters with one that goes in reverse. The .zip()</code> call brings both iterators together and iterates over both iterators for us, returning a pair (or tuple) of values. Think of it running both machines side by side, and with each turn of the crank one value pops out of each and zip returns both values tied together with a bow.</p> Now that we have a single pair, we immediately use it. At the beginning of line 3, you can see the for (i1, i2)</code> in part of the line. Just as before for..in</code> will run over the iterator. We know that the iterator is going to be outputting pairs of values. We could have just grabbed the pair as a single value and used it later, but here I go one step further by taking the pair the iterator gives me and pulling it apart so I can use it. This pulling apart, called destructuring, is a handy way of working with structured data when you're interested in its constituent parts rather than the structure itself. In this case, this lets me more easily compare the left side to the right side, so I destructure the pair into two values: i1</code> and i2</code>.</p> Problems 5 through 7</h1> Problems #5</a>, #6</a>, and #7</a> are all straightforward and don't show off anything new in the language. Let's jump ahead to the next interesting one.</p> Problem 8</h1> "largest product of 13 adjacent digits"</p> fn main() { let input = "73167176531330624919225119674426574742355349194934\ 96983520312774506326239578318016984801869478851843\ 85861560789112949495459501737958331952853208805511\ 12540698747158523863050715693290963295227443043557\ 66896648950445244523161731856403098711121722383113\ 62229893423380308135336276614282806444486645238749\ 30358907296290491560440772390713810515859307960866\ 70172427121883998797908792274921901699720888093776\ 65727333001053367881220235421809751254540594752243\ 52584907711670556013604839586446706324415722155397\ 53697817977846174064955149290862569321978468622482\ 83972241375657056057490261407972968652414535100474\ 82166370484403199890008895243450658541227588666881\ 16427171479924442928230863465674813919123162824586\ 17866458359124566529476545682848912883142607690042\ 24219022671055626321111109370544217506941658960408\ 07198403850962455444362981230987879927244284909188\ 84580156166097919133875499200524063689912560717606\ 05886116467109405077541002256983155200055935729725\ 71636269561882670428252483600823257530420752963450"; let mut largest = 0; let input_bytes = input.as_bytes(); let mut largest_string: &[u8] = &input_bytes[0..1]; let span_width = 13; for i in 0..(input_bytes.len() - span_width + 1) { let mut sum = 1u64; for j in 0..(span_width) { sum = (input_bytes[i + j] - 48) as u64; } if sum > largest { largest = sum; largest_string = &input_bytes[i..(i+span_width)]; } } println!("Largest: {} is {:?}", largest, std::str::from_utf8(largest_string)); } </code></pre> In Problem #8, we see an example of working with strings, this time using a new feature called a 'slice'. We start with the block from line 3</strong> to line 22</strong> that gives us this giant input string. Notice that I use the backslash '' to continue the line to the next line. Rust is smart enough to let us indent without introducing new whitespace into the string.</p> Now that we have our string, on line 25</strong> we turn it into a array of bytes we can search through. Notice we didn't use .chars()</code> this time. Recall that .chars()</code> gave us an iterator. Rather than iterating over one value at a time here, we want the flexibility to look across spans in our string. To do that, we use .as_bytes()</code> to get an array (technically, a byte slice).</p> Line 26</strong> introduces the slice we'll use. A slice is a window into values in memory. Since we want to look at a span of values, we create a holder for this slice. We'll later set it to spans of our 13 adjacent digits. PS: you'll notice I set it to a dummy default value, which I later throw away. This is to get around the compiler warning about uninitialized values.</p> The rest of the search follows fairly naturally. We do an iterator over the indices in our input, stopping short of the length of the span. We use that index to calculate the product at that span (line 32</strong> to line 34</strong>). Once we find a match, we save it off, using a slice to save off the winning span (line 37</strong>).</p> Finally, we use another format string {:?}</code>, which can print out our winning span for us using the debug formatter. The {:?}</code> format is a handy builtin which can handle a wider range of types with a default formatter.</p> Problem 9</h1> Problem #9</a> is another solution that's simple and doesn't show off any new features. Moving on.</p> Problem 10</h1> "sum of all primes below 2 million"</p> fn main() { let mut sum = 0; const SIZE: usize = 2000000; let mut slots: [bool; SIZE] = [true; SIZE]; slots[0] = false; slots[1] = false; // We calculate the primes using a simple stride and marking off multiples for stride in 2..(SIZE/2) { let mut pos = stride; while pos < (SIZE - stride) { pos += stride; slots[pos] = false; } } for (idx, pr) in slots.into_iter().enumerate() { if pr { sum += idx; } } println!("Sum: {}", sum); } </code></pre> Reading the problem description, you might have guessed that I would use the iterator solution from earlier. The astute reader probably already noticed that doing so would be. very. slow. Especially, as we look at primes above a million, where each step is itself taking hundreds of thousands, if not millions, of calculations. While I could let it run and heat up my apartment, it's better to do a more direct approach.</p> We trade space for time.</p> On line 4</strong>, I create my first fixed size array that will hold whether or not the number in that position is prime. Once we have the variable, it's initialized using another cool shorthand: [true; SIZE]</code>. This creates an array of boolean true values of the size given on the right of the ';', here the SIZE constant I defined. For example, you can create a 500-element array of zeros using [0; 500]</code>. Neat.</p> Once we have our slots, we loop over them using any possible multiple and check that off the list by setting that position to false. Once we're done, we have an array that tells us where the primes are.</p> The only trick remaining is how to get the numbers back out again. To do this, we use the iterator on line 17</strong>. Similar to our fluent iterators before, this time we turn our array of slots into an iterator, and then call .enumerate()</code>. The .enumerate()</code> call is a special kind of zip that gives us the index at that point in the iteration paired with the actual value. Just as before, we destructure the index and the boolean that indicates if it's a prime number separately.</p> On line 18</strong>, we check if the number is prime, and if so we add it to our sum. You'll notice the pr</code> call here. Just as in C, this lets me dereference and get at the value at that location in memory. I could have instead written for (idx, &pr)</code> in and just gotten to the value that way, using destructuring instead of dereferencing. With that change, the line would have been if pr { ... }</code>, and I could get at the value pr</code> without needing to dereference.</p> Problem 11</h1> "find the largest product in a grid"</p> let input = "\ 08 02 22 97 38 15 00 40 00 75 04 05 07 78 52 12 50 77 91 08 \ 49 49 99 40 17 81 18 57 60 87 17 40 98 43 69 48 04 56 62 00 \ 81 49 31 73 55 79 14 29 93 71 40 67 53 88 30 03 49 13 36 65 \ 52 70 95 23 04 60 11 42 69 24 68 56 01 32 56 71 37 02 36 91 \ 22 31 16 71 51 67 63 89 41 92 36 54 22 40 40 28 66 33 13 80 \ 24 47 32 60 99 03 45 02 44 75 33 53 78 36 84 20 35 17 12 50 \ 32 98 81 28 64 23 67 10 26 38 40 67 59 54 70 66 18 38 64 70 \ 67 26 20 68 02 62 12 20 95 63 94 39 63 08 40 91 66 49 94 21 \ 24 55 58 05 66 73 99 26 97 17 78 78 96 83 14 88 34 89 63 72 \ 21 36 23 09 75 00 76 44 20 45 35 14 00 61 33 97 34 31 33 95 \ 78 17 53 28 22 75 31 67 15 94 03 80 04 62 16 14 09 53 56 92 \ 16 39 05 42 96 35 31 47 55 58 88 24 00 17 54 24 36 29 85 57 \ 86 56 00 48 35 71 89 07 05 44 44 37 44 60 21 58 51 54 17 58 \ 19 80 81 68 05 94 47 69 28 73 92 13 86 52 17 77 04 89 55 40 \ 04 52 08 83 97 35 99 16 07 97 57 32 16 26 26 79 33 27 98 66 \ 88 36 68 87 57 62 20 72 03 46 33 67 46 55 12 32 63 93 53 69 \ 04 42 16 73 38 25 39 11 24 94 72 18 08 46 29 32 40 62 76 36 \ 20 69 36 41 72 30 23 88 34 62 99 69 82 67 59 85 74 04 36 16 \ 20 73 35 29 78 31 90 01 74 31 49 71 48 86 81 16 23 57 05 54 \ 01 70 54 71 83 51 54 69 16 92 33 48 61 43 52 01 89 19 67 48"; let input_split = input.split_whitespace(); let input_as_num: Vec<i32> = input_split.map(\|x\| match i32::from_str_radix(x, 10) { Ok(v) => v, Err(u) => { println!("Garbage in input: {}", u); 0 } } ).collect(); </code></pre> I'm only showing the interesting part of the solution since it's a bit long, but you can read the whole solution</a>.</p> Here we do a bit more string manipulation using some new methods. The .split_whitespace()</code> method on line 23</strong> lets us turn our input string into an iterator of strings, splitting at any whitespace.</p> Once we have this iterator, line 24</strong> takes this iterator and maps a function over it. This function attempts to convert each substring to a signed 32-bit integer i32</code> using the .from_str_radix(x, 10)</code> call. This call returns a Result</code>, a way of handling either a success or failure condition of an action that might fail. As is the case with converting from strings to numbers, if you happen to pass in something that can't be converted to a number, there needs to be a way to signal back out that the conversion failed. Some languages do this through exceptions, which break out of normal execution and give you an error you can handle. Rust uses a simpler, more functional approach and treats errors as just any other value. With this method, the type system encourages us to be vigilant and always have code available to handle errors.</p> To see which of success/failure is returned, we use the match keyword. Just as in other languages with pattern matching, the match keyword lets us ask which of the possible values is in the Result: Ok</code> or Err</code>. Pattern matching lets us destructure and find the success value or error message.</p> Finally, on line 29</strong>, we use the .collect()</code> method to run through the iterator and create a vector of i32</code>.</p> Problem 12</h1> "triangle number with over 500 divisors"</p> fn main() { let mut num = 0u64; // Cache the primes we'll be using let first_primes: Vec<u64> = primes().take(1000).collect(); for i in 1..1000000u64 { num += i; let mut num_div = 1u64; let mut num_tmp = num; for &x in &first_primes { let prime = x; if (num % prime) == 0 { let mut exponent = 1; while (num_tmp % prime) == 0 { exponent += 1; num_tmp /= prime; } num_div = exponent; } if num_tmp == 1 { break; } } if num_div > 500 { println!("num: {}", num); return; } } } </code></pre> Again we'll trim down to the interesting parts, since we've seen the prime iterator before, but you can read the full file</a>.</p> This looks pretty familiar. On line 5</strong>, we run our primes iterator, take enough to use later, and turn that into a Vec<u64></code> we can use a bunch of times.</p> The other interesting line is line 13</strong>. If you've played with vectors in Rust before, you know that you can iterate over them, like this:</p> {% highlight rust %} fn main() { let x: Vec = (1..10).collect();</p> for i in x { println!("{}", i); } </code></pre> } {% endhighlight %}</p> But if we try to do this in our Euler problem, we get a new error:</p> error: use of moved value: `first_primes` </code></pre> What is a moved value</em>?</p> It turns out that a moved value is part of Rust's ownership system. Just like the first time a programmer sees a pointer in C or a monad in Haskell, ownership in Rust is one of those things that's so uniquely Rust that it takes getting used to. Once you begin to understand it, you'll be able to see how the type system is trying to help you describe your intentions a bit more clearly.</p> If you're scratching your head, here's an example: imagine malloc'ing an array in C and then passing that malloc'ed array into a function. Who is responsible for cleaning up that memory, the function you called or the function that created the array? In C, you don't know by looking at the code, you only know as the writer of the code what your intention was. Maybe the function is called 'cleanup' at which case, yes, you do want it to free that memory. Or maybe the function is called 'inspect' and you decidedly don't</em> want it to free the memory. This is why Rust makes this very explicit. By looking at the code, you can follow who is responsible for what. The set of checks that do this make up Rust's ownership system.</p> There are a lot of resources</a> to understand ownership,which do a good job of explaining ownership in detail. For us, ownership is like trying to answer questions like "who is responsible for this thing?" and "when does this thing get removed?" Ownership can be passed from variable to variable, and its strictly checked by the compiler.</p> When you first encounter the above error, if you haven't already read about ownership, you might be in for a few hours of reading to get caught up. But we have some hints here that we can get started with to help figure out what's happening.</p> In our small example, we could iterate over a vector and all was well. When did things start to go wrong in the Euler problem?</p> On line 13</strong>, we want to loop over a vector, and when we look out to line 7</strong>, we see this is actually an inner loop being used by an outer loop. That means that if the inner loop takes control of our vector, the next time around the outer loop we've lost the ownership.</p> We don't want the inner loop to take ownership and then lose the ability to use the vector again in the next iteration of the loop, so we have to 'borrow' ownership. As the name implies, by borrowing, we're only taking temporary ownership for a time, and we have to follow all the rules stated when we borrow. With that in mind, let's look at line 13</strong> again:</p> for &x in &first_primes </code></pre> We borrow first_primes</code> instead of taking permanent ownership using the &</code> operator. This operator says "I would like to borrow first_primes, and I can not edit the contents". The compiler will then be able to check that we're using what we've borrowed correctly. You might be curious borrowing and also being able to mutate. To do that, we could've used the &mut</code> operator assuming the original vector was mutable.</p> Finally, we use the same destructure trick we did earlier to get at the values of x</code>.</p> PS:</strong> In this example, since all we were doing was iterating, we could have used the .iter()</code> rather than borrowing, though that won't always be the case in other examples that need more direct interaction with the vector.</p> Cargo</h1> After I got enough of these problems together that I got tired of building them by hand, I moved over to using Cargo. Cargo is Rust's build/package tool that is quite handy.</p> [package] name = "eulerinrust" version = "0.1.0" [[bin]] name = "ex1" path = "src/ex1.rs" </code></pre> An example of Cargo.toml</em></p> Cargo lets you describe dependencies that your project needs, which it fetches from crates.io. You can also control whether you're doing a release or debug build, and quite a bit more. I'll no doubt start directly with Cargo for any future Rust projects.</p> Summary</h1> These are just tiny examples created while playing around in a fairly simple set of problems, and yet there's actually quite a lot of Rust's richness shining through. Rust has a powerful iterator story that feels composable and clear. Rust's error checking is keeping us vigilant about handling error cases, matching types, and remembering who is responsible for values in the system. Overall, it feels like a very well thought-out language that holds together well.</p> Off to new adventures 2015-10-05T00:00:00+00:00 A few weeks ago I posted about moving teams inside of Microsoft. As sometimes happens when you make a change, you realize that you actually need to make a bigger change. A few weeks into the new team, and I could tell that it was time for me to move onto new adventures.</p> Microsoft has been a great place for me to "grow up" as a program manager. When I started as an intern, I was mentored by some of the most capable PMs I've ever met. They took a keen interest in me, and I benefited greatly from their years of experience and patience.</p> From a little bird of a PM, I grew to being able to lead the great TypeScript team. I got to meet countless numbers of people and make some good friends along the way. It was a wild adventure to be on the team, and I'm excited to see what they come up with next.</p> But as little birds grow, there comes a time to leave the nest and to take more risks. There are many adventures to be had, and to have them, you have to ride out the storms in search of the blue skies.</p> My first step will be to take some time to travel the world and see friends I haven't seen in years. My goal: recharge, explore, and take stock. After that, who knows.</p> Part of the adventure is not seeing the way ahead and going anyway.</p> Programming languages are a failure 2013-02-17T00:00:00+00:00 After spending years working in programming languages, helping out with some big-name projects and trying to generally make programmers' lives easier, I've decided something.</p> Programming languages are a failure.</p> Okay okay, before you write this off as a troll for attention, hear me out for a minute longer.</p> Calling a programming language a 'language' is false advertising. Think of what comes to mind when I say 'language'. A language, as in human-to-human language, has some pretty interesting properties:</p> It can be used to express intent</li> It can transmit information even if one (or even both) of the speakers are non-native</li> </ul> The act of programming is inherently not like speaking. Imagine having a conversation with someone in the same way you program.</p> Say what you want the other person to understand</li> Say it again, this time with slightly different wording</li> Say it again, with yet more slightly different wording</li> (Rinse, repeat. Time passes...)</li> Test the other person on what you've said to be very sure they understood</li> Move on to the next thing you want the other person to understand</li> </ul> Of course, the reason we have to do this in programming is because we want what we say to be precise, so the computer exactly what to do, right? Human language is filled with all that icky imprecision, which is why we use a programming language. The special kind of language that precisely describes what we mean to the computer, so long as we're patient enough to explain ourselves just so.</p> What we've actually created is something akin to legalese for programming rather than legal documents, a sort of programese. It's a one-way conversation with the computer in a style that removes as much room as possible for misinterpretation. The dirty secret is that programming languages, like many legal documents, never really finish going through refinement, even after years and years of refinement.</p> Why not embrace the ambiguity? Better yet, why not make programming a two-way communication that more closely resembles people talking?</p> Instead of this:</p> > The bird sits on the pole. Error: I don't understand 'bird'. I don't understand 'pole'. > The bird, an animal, sits on the telephone pole. Error: The animal 'bird' can not "sit". > The bird, an animal, stands on the telephone pole. Okay. </code></pre> Why not have this:</p> > The bird sits on the pole. Question: By "sits" do you mean "stands"? > Yes Understood. </code></pre> Let the programming system ask you when it truly doesn't understand. The difference is, where it can infer an understanding based on context and background knowledge, it does.</p> This opens up further possibilities. What the programming system could be doing for you behind the scenes is fill out what it understands, changing the text much like Telescopic Text. This would replace "The bird sits on the pole" after you type it in to be a telescopic text you could click on "[The bird] [sits on] [the pole]"</code>. If you clicked on [The bird]</code>, the text would telescope in one level, giving you: "[The bird, an animal][sits on][the pole]"</code>.</p> Imagine how much easier</p> Promising new programming languages 2011-09-22T00:00:00+00:00 It seems like these days there are lots of programming languages springing up. A combination of the web, parallel programming, and just outright curiosity seems to be fueling creativity towards new mixtures of classic features and experimental new ones.</p> Here are a few new(-ish) projects I'm keeping my eye on:</p> Rust</a> (Mozilla Research) - Concurrent language that only allows sharing of mutable state if it's handed off between threads. Uses typestates to enforce this ownership. Expected to be used in Project Servo, Mozilla's experimental parallel browser engine. Recent talk slides look promising. Still experimental (haven't reached 0.1 yet).</p> Kotlin</a> (JetBrains) - Impressively clean/lightweight syntax language for the JVM in the spirit of a Scala light. Perhaps may be to Scala what Firefox was to Netscape, but it's not yet release so it's hard to say for sure. Still, their presentation at the JVM Language Summit is worth a watch.</p> ooc</a> - You have to like the infectious audacity of this project. Hobbyist putting together a language, then proceeding to write one compiler for it in itself and then another.</p> Dart</a> (Google) - Mentioning it here because I'm curious, but the hype has already overshadowed it. We're still weeks away from knowing anything about it. Still, Gilad Bracha is like a "curiously strong mint", so we at least know it won't be flavorless.</p> These are just the ones on my radar. There are certainly others (Ceylon, Gosu, Mirah - and those are just the ones on the JVM), but they haven't quite made it to my twitter follow list.</p> Will parallelism go implicit? 2011-03-08T00:00:00+00:00 I've been spending quite a bit of time recently with Implicit Parallel Programming in pH</a>, which luckily the school library had a copy of. It's quite a good textbook to introduce programmers to functional programming, perhaps even one of the better ones (next to the original Introduction to Functional Programming</a> by Bird, Wadler that I mentioned in an earlier blog post). What turned me on to it, though, wasn't the treatment of functional programming - it was its far more audacious titular goal.</p> Which leads to the obvious question. What is implicit parallelism?</em></p> If you bring up the term around PL geeks, you might get a few questioning looks and some comments that hint at automatic loop parallelization. Even more confusingly, this second kind of parallelism is sometimes referred to as automatic parallelism. It refers to taking sequential codes and reasoning about where parallelism could occur without changing any results. Implicit parallelism is a more holistic approach - what if the only sequentiality in your program comes from the data dependencies? Then the rest, it stands to reason, can be done in parallel.</p> I mentioned this as an audacious effort. The first and foremost reason, to my engineering mind, is how in the world will you make this efficient?</em> I can just imagine the compiler identifying tens, perhaps even hundreds, of tiny tasks that can be done in parallel at any one timeslice. How do we schedule these tasks in such a way we don't pay a high overhead for synchronization and task switching?</p> I haven't dug into the implementations of the languages yet to see how they address this issue, but as I was brainstorming this morning it hit me. Implicit parallelism isn't so different from implicit (or automatic) memory management. When garbage collectors first came out, they weren't the lean mean things they are today. That's part of the advantage of having decades to polish an approach, and now fewer and fewer applications programmers use languages that don't have gc support. It saves more headaches than it causes, and that's generally a winning combination.</p> Not always, of course. The HPC community has regularly shunned gc in their languages, where high performance codes are hand-tuned Fortran or C with manual memory management. Still, I would argue that HPC is boutique, a cottage industry focused on maximal efficiency.</p> Most everyday programmers don't need these levels of efficiency, and I imagine, will instead opt for automatic support for parallelism. This saves them the headache of separately tuning for the same application to run comfortably on either 16 or 128 cores. From the consumer's point of view, this is also a win. Buying that new 256 core machine actually will make these apps run faster.</p> Some final thoughts</strong></p> Perhaps it's time to focus on introducing more forms of automatic and implicit parallelism into our programming languages and runtime systems. Our early attempts, taking from the experiments of the past could kick it off, and once industrial interest has turned to it, I'd expect steady progress in efficiency (just like we've seen in gc).</p> Is there some practical limit to the amount of parallelism in code? There's a tension between Amdahl's law</a>, that maximum speedup through parallelism is limited by the amount of sequential code, and Gustafson's law</a>, which says that the increased computing power lets us tackle larger problems. To continue to gain we may have to change how we approach problems, or even reframe the problems themselves.</p> Thoughts on actor-based languages 2010-09-07T00:00:00+00:00 (or "Minnow, a post-mortem")</p> I've been fortunate to work on both Minnow</a>(a shared-nothing actor-based language) and Chapel</a>(concurrent global-view language). I wanted to give a few thoughts comparing the two approaches.</p> First, when we talk about concurrency or parallelism, without going into the pedantic definitions of each, we should answer the question "why not stay serial?" There are a lot of reasons to do so, if we can. Serial code is going to be easier to write, easier to debug, and easier to reason about. Generally, going parallel is just going to complicate our lives.</p> I see two main reasons why going parallel would be required:</p> Reliability</strong> - Does the system have to tolerate failures of its parts?</li> Performance</strong> - Is serial too slow for your application?</li> </ul> In Erlang</a>, reliability is king. Its shared-nothing actor-model style allows actors to fail and be restarted, which allows for more advanced features like hot code swapping.</p> For areas where performance is king, C and Fortran still reign. In high-performance scientific computing, it's common to remove abstractions and program as "close to the metal" as possible. Though this doesn't preclude reliability, that reliability must either be done by the programmer or by the underlying architecture. It is, as you can imagine, very fast.</p> When I created Minnow, I set out to create an actor-based language that could perform as fast as C but potentially gain the benefit of reliability from the actor abstraction. Focusing solely on single node, shared-memory machine, I was able to optimize:</p> Message sends, down to a few atomic operations</li> Number of actors possible, using light-weight threads and continuations</li> Run-time overhead, even in the presence of fairness counters for cooperative multi-tasking, by minimizing extraneous work in loops and function calls</li> </ul> In short, Minnow was quite fast.</p> Was it fast enough? As it turns out, the answer is no</em>. Though Minnow avoiding copying where possible, it did not allow the user to see a large matrix and operate on pieces of it. Doing so would go against the notion of shared-nothing, we might even say that shared-nothing runs counter to the goal of performance. Algorithms where we could bifurcate data into chunks, for example, wouldn't suffer, but in general we can no longer use contiguous memory in a natural way.</p> The modern batch of concurrent languages (Chapel, X10, Co-Array Fortran, and others) return to looking at memory in a contiguous way. In fact, using partitioned global address space</a> (or PGAS), the notion of contiguous memory is extended to encompass the memories of all the nodes in a cluster.</p> The question of where reliability should live, and if we should push it into the language and sacrifice performance, appears to be solved issue in the high-performance community. It remains to be seen if the larger programmer community will embrace this approach.</p> Digging into scheme/racket 2010-09-05T00:00:00+00:00 I spent some of last night and this morning digging into Racket, the modern version of the venerable Scheme language. Every time I return to a Lisp variant, I find the parentheses less annoying that I did the last time. With proper indenting and some syntax highlighting, is there all that much difference?</p> (define factorial (lambda (n) (if (= n 0) 1 (* n (factorial (- n 1)))))) </code></pre> Arguably the trailing parentheses are an eye chard.</p> With the download of Racket you also get DrRacket, which is an updated version of DrScheme, a learning tool aimed at programmers of different experience levels (and to my understanding different ages as well)</p> I've just begun to experiment with DrRacket to see what it can do. To get familiar with the Scheme style, I've been working through Teach Yourself Scheme in Fixnum Days</a>.</p>

The case for Nushell

2023-08-30T00:00:00+00:00

Recently, I had a chat with some of my friends about Nushell and why they stuck with traditional shells like bash/zsh or the "new" hotness like fish rather than using Nushell. After chatting with them, my brain kept bubbling away at the state of how folks were using their terminals and the end result is this blog post.

In this post, I make the case for really taking a hard look at Nushell and also for generally asking the question: "can the state of shells be improved enough to overcome the inertia of sticking to what you know?"

The lay of the land</h2>
Let's take a look at some of the offerings out there that people are using everyday.

Bash/zsh</h3>
Bash, originally a set of improvements on the Bourne shell, has grown to be the default shell for almost all Linux distros. That's generally people's first experience when they hit the terminal. It's what they see when they log into a remote machine. It's reached the definition of ubiquitous.
I also throw 'zsh' in here as well. Apple's macOS switched from bash to zsh, an operationally similar shell but created a bit more recently.
Bash at this point has become so well known that people often confuse support for bash-isms as part of the POSIX standard, but we'll talk about that later.
Pros: it's everywhere. Learn once, run anywhere.
Cons: as a language, bash/zsh feels a bit too retro. It doesn't offer any of the modern programming language style, tool support, etc folks would be used to from other languages. In truth, bash was never really meant for writing the kind of large scripts that people are maintaining today.
Example for loop in bash:
#!/bin/bash for i in `seq 1 10`; do echo $i done </code></pre>Fish</h3> As fish's website says: "Finally, a command line shell for the 90s" It's enough to elicit a smirk, because you know it's a bit true. The bash/zsh style shells are getting left behind by something that feels a bit nicer, has nicer completions, looks nicer (you can get similar improvements out of bash if you work at it, but fish ships with them out of the box) Fish also bravely steps away from the shell scripting form of bash to something a bit more readable. Example for loop in fish: for i in (seq 1 10); echo $i; end </code></pre> Pros: the interactive experience of fish does feel quite a bit nicer that bash/zsh out of the box. Scripting is a bit nicer. Cons: As it says on the tin, it's a shell for the 90s. It ain't the 90s anymore. PowerShell</h3> Coming into 1.0 at 2006, PowerShell is one of the first shells to really draw a line in the sand to say "enough, we're going to do things differently". The unix style of pipelines, where commands communicate via text to each other was replaced by a .NET engine that passed objects between commands. The impact wasn't immediately obvious but as devops folks (and others) discovered what was possible when you have ways to work with data directly a fanbase grew. Example of a for(each) loop in PowerShell: foreach ($i in 1..10) { echo $i } </code></pre> PowerShell came with an opinionated design that focused on verb-noun naming, improvements to shell syntax, and a vast set of functionality drawn from the .NET ecosystem. Pros: it's a structured shell - you can actually work with objects rather than text. Powerful set of tools and capabilities taken from .NET. Cons: I'll go ahead and say it: PowerShell was never really designed to be a language first. The verb-noun convention forces a style that feels very awkward coming from other languages. Worth a mention: earlier versions of PowerShell were Windows-only and modern crossplatform support lacks some of the features of the earlier versions. Other shells</h3> When I was coming up, there were a lot of other shells, including the csh/tcsh family. Having said that, I don't know anyone who is using any of the other family of shells. Bash/zsh and to some extent fish really dominate the developer mindshare. Hold up, we really need to talk about POSIX</h2> We really need to take a minute and talk about POSIX before we continue. A lot of folks have leveled "but it's not POSIX" as an argument against using Nushell, but I'd like to turn that around and ask the question: "What's so good about POSIX?" Most folks when asked would likely point to it as a common ground that code can be ported to. In reply, I'd like to quote a few bits of the POSIX standard. The following are reserved words in the POSIX standard for shell scripting</a>: case</li> do</li> done</li> elif</li> else</li> esac</li> fi</li> for</li> if</li> in</li> then</li> until</li> while</li> </ul> Yes, really. fi</code> and esac</code> are a joke that never found their end. No one would design a language that did that with a straight face these days. Let's take a quick look at the number of flags common Unix commands ship with. These are on my macOS system, so ymmv. command</th> number of flags</th></tr></thead> ls</td> 45</td></tr> man</td> 15</td></tr> ps</td> 29</td></tr> </tbody></table> If you look through the flags of ls</code> to see why it has so many, notice how many are configuring what ls</code> is displaying. In a real sense, this is going against the underlying philosophy of unix pipelines. Rather than composing a pipeline to get the display you want, you're learning a language of flags for each command to configure the display. Let's talk about exit codes. Actually, wait, I already did that</a>. As I point out in the post, the standard says: "The value of status may be 0, EXIT_SUCCESS, EXIT_FAILURE, [CX] [Option Start] or any other value, though only the least significant 8 bits (that is, status & 0377) shall be available from wait() and waitpid(); the full value shall be available from waitid() and in the siginfo_t passed to a signal handler for SIGCHLD. [Option End]" </blockquote> Sure - we all still use 8-bit machines while flipping through a manual we printed trying to find the exit code description to understand why a command failed. I hear you: "but, look, it doesn't matter how archaic this stuff is, if we all agree to use it things keep working." I dunno - these arguments just don't hold up. We'd still be using C as our main systems language because it's the most documented, most portable, etc. But, by and large, we don't. We're increasingly choosing other languages. The truth is, in 2023 if someone asked us to design a system, we wouldn't design POSIX. If, in 2023, someone asked us to design a shell language, we wouldn't design bash/zsh. This matters. Show us the money</h2> I make some pretty bold statements in the above. There is a heritage of technology that got us to this point. While that heritage is important, it's not without its drawbacks. Are there better ways of doing it? Why structure matters</h3> Before we get into talking about Nushell, let's talk about why structured data matters. In the Unix pipeline way of thinking, text passes between commands. This is very flexible, but has a major problem: both the outputting command and the inputting command have to agree what shape that text will take so the info can be passed. This locks representation to presentation, disallowing commands from evolving their output over time. As we showed earlier, it also encourages a proliferation of flags. That's annoying. Does separating the structure from its presentation help us? > ls | where size > 10kb </code></pre> I always start with this example when showing off Nushell, because not only is it immediately readable, we didn't have to dig through any flags to figure out what we needed to pass to ls</code> to get that. We also aren't parsing anything from ls</code>. Instead, the data is passed directly to where</code>, which handles it directly. Commands already know this structure, why not make use of it? The same where</code> command works on other things. For example, we can process the output of the ps</code> command: > ps | where cpu > 40 </code></pre> Or open a csv</code> file and processing its rows: > open fields.csv | where area > 5 </code></pre> And so on. It's the same where</code> regardless of where the data is coming from. It also gives us the freedom to present the data however we want. Why Nushell matters</h2> Nushell is designed to be a language</h3> I had the good fortune of being a part of some prominent programming language teams, including helping to create TypeScript</a> and helping create Rust's error messages as part of the Rust team in Mozilla</a>. Designing languages to be easy to use, easy to read, easy to scale up, and easy to debug is something I care about and have worked on for many years. To that end, Nushell is designed with an eye towards being readable at a glance. Let's do a for</code> loop in Nushell: for i in 1..10 { print $i } </code></pre> (aside: "but why do variables have dollar signs?". Turns out the flexibility of shell programming allows paths to not use quotes, so it's nice to tell a difference between cd foo</code> and cd $foo</code>) This eye towards usable design shows up in many ways. Working with data is improved by not only having structure, but also being able to pattern match against it. Here's an example of pattern matching a list in Nushell: match $list { [$one] => { print "one element list" } [$one, $two] => { print "two element list" } [$head, ..$tail] => { print $"the tail of the list is ($tail)" } } </code></pre> In a way, working in Nushell should feel at home both interactively as a shell and as a full scripting language. We've had folks write COVID reporting software in Nushell, research experiments, even entire shells for well-known database services</a>. Nushell is typechecked</h3> Since Nushell doesn't treat all data as text, you can represent tables, records, numbers, booleans, etc directly in the language. As a result of this, Nushell is fully typechecked. Common errors can be caught early and shown to you before the script even runs. Taking what we learned from TypeScript - the types also feed into another important tool. Nushell has IDE support</h3> The types, autocompletion, and early error reporting feed into an engine in Nushell that knows a lot more about your code. As a result, you can write scripts and then work with them using the IDE support Nushell provides. Seeing errors, jumping to definitions, getting documentation on hovers, etc are all part of the Nushell experience. Nushell has nice errors</h3> In Nushell, we make extensive use of remembering where data comes from, as well as what caused an error. Simple errors, like division by zero, are shown clearly: A more complex error may need to show more to help track down where a mistake came from. Let's say you've accidentally put a string in your list of numbers, and then tried to process it: Nushell has a SQL-like style</h3> When you start using Nushell to compose pipelines, you'll notice that it has a distinct SQL-like style. Data flows through each stage, and you build up what you want to do to it as you add more commands. This gives Nushell a distinctive design that encourages experimentation and exploration. Nushell is, and has always been, crossplatform</h3> An important decision we made from day 1 was to be crossplatform. You can run Nushell on Windows, Linux, and macOS (and BSD, and Android) and get the same experience. You can easily write scripts in a way that they can be run across different platforms. Everything that you learn transfers between OSes without friction. Is Nushell good enough to overcome the inertia?</h2> I distinctly remember going to a SIAM conference many years back and giving a talk on the Chapel programming language</a>. Even back then, it was a clever language. In a couple lines, you could write code that could distribute and process a matrix across a network of computers. Coming from a lineage of array languages, it ate up data parallel processing. The equivalent code in other languages looked verbose in comparison. I went through my talk, hoping I'd done a decent job of conveying the main points, and at the end, someone in the audience stood up and said "but I can do all this in C++". He proceeded to explain that if he could recreate many of the techniques we showed all as part of a C++ library that people could use. At this time, I wasn't sure how to respond other than "but you don't have to, we already built this language" but he couldn't be swayed. If it wasn't C++, he didn't want it. Fast forward a couple years, and I'm standing in front of a JavaScript audience giving a similar talk, this time promoting TypeScript. I remember the kind of politely confused looks on people's faces as I showed off the features TypeScript offered. There was a similar sense of "why do we need to leave JavaScript?". To answer whether Nushell can overcome this kind of inertia, I'll pose two questions: Is Nushell compelling enough for a single person to adopt it?</li> Would adopting Nushell broadly as a community move the needle?</li> </ul> Let's tackle the first question. Time and again, as people try Nushell, they come back with quotes like "this is the most excited I've been about tech in 15 years". It has a fanbase that loves it, and that fanbase is growing. It reminds me of the early days of Rust, just after hitting 1.0. To the second question: would adopting Nushell broadly actually improve things noticeably? Without a doubt. I say this without any reservation. Thinking of our shells as structured, interactive processing engines opens up the doors to a much wider array of things you can do with them. The commands would be far simpler than their POSIX equivalents and would compose far better. They'd benefit from the full knowledge of the data being shared between them. Adaptors could be made to connect to all parts of the system, allowing you full, structured interaction with everything you have access to. In essence, as the saying goes, we'd be building a skyscraper starting from the 15th floor instead of the 1st. It's time to be honest about what Nushell is</h2> It's time I come clean about what Nushell is. Nushell isn't exactly a shell, at least not in the traditional Unix sense of the word. Nushell is trying to answer the question: "what if we asked more of our shells?" Nushell is really an interactive, data-focused scripting language with shell capabilities. It merges these three things into one: A fully-typed scripting language</li> An interactive shell</li> A data processing system (that can also handle large data loads via dataframes)</li> </ul> Rather than these being three separate ideas glued together, in practice it feels like Nushell is treating everything you interact with as data. This allows you to pull together different kinds of data, and know that the same commands will work over them. You might look at that list and think you don't need all that, but the way I might frame it is this: it's nice to have it when you need it. I don't need to do heavy data processing everyday, but it's nice to not have to shift what I'm doing at all when I need to do it. I don't have to download new utilities or switch languages. It's all right there. Need to write a script to load some files and handle some directory processing? Still right there. Need to throw together some web query that outputs the top download results for a github repo? You guessed it, all still right there. This is just scratching the surface, really. Nushell has a plugin system that allows more capabilities to be added based on your needs. We already have plugins that add a variety of additional file formats, querying capabilities, and more. It's okay to have nice things</h2> Nushell was built with a simple idea: working in the shell, writing code, and processing data should be fun. To that end, we work hard to make Nushell feel nice. You can write readable scripts that come with their own documentation, and then come back to them 6 months later and still understand what they're doing. You can sit in the shell and play with pipeline ideas until one grows into a scripting project and then effortlessly transition your experiment into a full script. That's it</h2> That's my case. It's okay to have fun. It's okay to write attractive, well-documented code. It's okay to leave the designs of the past behind when they no longer fit the present day. It's okay to move on to better ways of doing things. Happy pride! I'm trans! 2023-06-26T00:00:00+00:00 Hi, I'm trans. Let's talk about it! Note: In this blog post, I talk about the medical side of my particular kind of transgender experience. If that's not your thing, feel free to skip. A long time in the making</h2> The first time I looked up anything trans-related was in my early 20s. I didn't know it at the time, but I was actually looking for a way to help address a set of symptoms I was having. For most of my adult life, I experienced a cluster of issues: dissociation, derealization, depersonalization, and others. I thought it was normal to feel disconnected from your body, and for parts of your brain to feel disconnected from other parts of your brain. What I didn't understand, but have come to learn this year, is that I have biochemical dysphoria</a>. In short, this means that the hormone that my body used to regulate, testosterone, was not the correct regulating hormone for me. Think running an engine on the wrong fuel: it kinda works but it's really not good on the engine and you'll have side effects. Switch to the correct one, and you'll notice a big difference. The fix, as things go, is relatively simple. Remove the hormone you're starting from and replace it with the other hormone (using a process often called HRT). If you have the condition, you'll likely start feeling noticeably better rather quickly. If you don't, you'll likely feel worse. The first few weeks of HRT</h2> Before I talk about my experiences on HRT, I want to say that I'm just sharing my experiences rather than giving medical advice. Everyone is different, and there's really no telling how anyone else would respond. I did this through a doctor, as I luckily had access to trans-friendly medical care. It's my sincere wish that in the future everyone will have that kind of access. The early impact of HRT for me was certainly eye-opening. Mental health symptoms I'd had for decades started to lift within the first 2 weeks. I no longer felt internally disconnected. Instead, internally I had more flow, comfort, and a feeling of openness. I think my reaction could be summed up with "what... in the world?" in a very good way. I remember walking around my apartment mumbling to myself "I hope this doesn't stop, because this is great." Luckily, so far, it hasn't. Along with the drastically improved mental health I had two other changes: emotional changes and mood changes. Emotional changes</h3> The emotional side also began to change rather quickly. I think this could be described as someone handing me glasses for the first time. Whereas before emotions felt like they were piecemeal and difficult to understand, now emotions feel clear. I could cry, and smile, and feel each more deeply and more connected to my inner world. I love it. I'm able to speak more of the language of the inner world, which is helping me not only understand myself, but also communicate better with other people. Mood changes</h3> Since November last year, I've been daily logging my mood on my phone. I was hoping to find some patterns or some kind of good data. Without really realising it, I managed to log months of data before starting HRT as well as the months following. With it, I got a pretty clear picture of how HRT affected my mood. Before starting HRT, my average mood around around 6 out of 10. After starting HRT, within a couple weeks, it had risen to 8 out of 10. It wasn't just on the app. Friends who talked to me who hadn't seen me in a while would remark "whatever you're doing, keep doing it." The first three months</h2> While I won't list all the changes, as honestly something should probably be private in this blog post, let me highlight a few more changes: Week 3</h3> At week 3, I had my first big emotional wave. I've never felt something like it before. Imagine being a small boat on a very large sea and the wave effortlessly lifts you up and down. It took a bit of practice (and no doubt something I'll continue to learn) to ride this wave. Along with this, though, I noticed that these swells brought with them a kind of sharp focus when noticing things that are wrong/unjust/etc. You can see and feel when something isn't right, and it can be confusing when other people don't seem to notice or don't seem to feel it as intensely. Week 10</h3> Around week 10 my skin started to change. It changed to have a softer texture which is quite pleasant. Along with the above, my psoriasis started clearing up. Patches which had been there for years shrunk, and are continuing to shrink. The skin changes also got me starting to look into different fabrics and clothing, as the coarse men's clothing honestly doesn't feel all that great on the skin as it gets more sensitive. It's nice to put on something and feel soft and cozy. And that brings us to today. Today</h3> I've just gotten back from my doctor appointment for my 3 month check-in. Clean bill of health, so on we go (thankfully). My doctor was very pleased with how I was taking to the treatment, and clearly I am as well. Questions one might have</h2> Okay, so there are probably a few questions that could come to mind reading this, and no doubt a few others I haven't thought of. Let's see if we can address those. Trans... what, exactly?</h3> Being transgender means being a part of a rather large umbrella. Many folks know that there are trans women and trans men. For me, though, my trans-ness means to leave maleness and go off into the unknown. I don't really have a goal. Hmm, I take that back. My goal is to be healthy, and I try to put the effort in each day to make that happen. What that means may change over time, but I'm not rushing it. Aren't there body changes on HRT?</h3> Yes. And?</h3> Oh oh you mean, what will I do when they happen? Again, taking it a day at a time. I don't have all that figured out, and I'm okay with that. On a personal note</h2> It scares me how political trans-ness has become when, especially for my case, it's medical. It's something I'm working on between me and my doctor. I'm betting that's true for lots of folks: they just want to live their lives. It scares me that receiving medical treatment is difficult for a lot of people, especially seeing just how effective the medicine can be. The changes can be profound, and in some cases, also life-saving. To withhold that feels cruel. I'm sharing my experience here to both be a record for myself, and also to be something other folks can read to learn about what it's like to have biochemical dysphoria and to have it treated. I wish I could share the joy that I have now, the feeling of relief, for being able to address things I've struggled with for over the last 20 years. Do I wish I could have started sooner? Absolutely. Maybe this post will help other folks ask questions and get treatment sooner, too. Why I left Rust 2023-05-28T00:00:00+00:00 There's a lot of speculation about why I left Rust yesterday, so I want to set the record straight in this post. What happened</h2> A short timeline of what happened from my perspective, as this is relevant for later: A request went out to the interim leadership group for potential keynote speakers for RustConf.</li> JeanHeyd Meneide was proposed by myself and Manish as a keynote. JeanHeyd is one of the two editors of the official C standard</a>, a C++ expert, and a good speaker. We felt like he'd be a great invited expert with an outsider perspective.</li> A few days later it was held to a vote. Consensus in the interim leadership group (what the official blog calls the "leadership chat") are of those present in the group when the decision is made. In total five members voted, and all five votes supported JeanHeyd as the keynote speaker.</li> JeanHeyd was asked and accepted the keynote.</li> Before the schedule was posted, it was discussed in a team meeting that JeanHeyd had been selected as one of the keynote speakers.</li> A couple team members had strong opinions/discomfort against JeanHeyd being selected as a keynote speaker, as best as I understand it, because of the content of JeanHeyd's blog post on reflection in Rust.</li> This discomfort was brought to the interim leadership group as a problem that must be fixed immediately. There was no discussion of the ramifications of making a change at this point. There were complaints that this pushback was not sufficient to require change, but this feedback was ignored. The discussion focused on the discomfort of the team members and changing JeanHeyd away from being a keynote speaker as the only solution.</li> Two people in Rust leadership then, without taking a vote from the interim leadership group (remember, JeanHeyd was voted on and selected by Rust leadership), reached directly to RustConf leadership and asked to change the invitation. (note: this used to say one person, but I've now had multiple people independently confirm it was two people.)</li> RustConf leadership decided to wait a week before informing JeanHeyd, giving Rust leadership time to change its mind. The interim leadership group was not informed of this week delay.</li> After the week expired, JeanHeyd was informed of the decision to downgrade the talk. Understandably, he fully declined to speak at RustConf.</li> When I read JeanHeyd's blog post and learned of the declining of the invitation, I immediately resigned from the Rust project.</li> </ol> I'll also note that the above is the best I am able to understand and piece together. A full investigation needs to happen to uncover how and why the above happened. I also reserve the right to edit the above if new information comes to light. Why did I leave?</h2> To my mind, why I left is obvious. I just watched the Rust organization disgrace one of the experts in my field. As an organization, Rust and RustConf offered little resistence to Rust being wielded as a weapon against an individual because of some discomfort of a couple team members. What rang in my mind is: "a system is what a system does" Rust acted as a cruel, heartless entity that did not care about JeanHeyd and treated him as disposable. Easy to offer a place of respect and just as quick to snatch it away. That is what Rust is because that is what Rust did. I left because when I felt JeanHeyd's pain and disappointment at being mistreated and betrayed, my heart broke. I wept because of the cruelty. But I also wept because I helped create the system that could do this to someone. But it was just a downgrade</h2> I shake my head at people that say things like this. Clearly, they are not used to treating people - let alone experts in the field - with respect. Downgrading a keynote that you explicitly invited the speaker to give (and prior to this they weren't even going to speak at your conference) is extraordinarily disrespectful. I asked around to see if anyone had experienced this or even heard of it happening. No one had. Decisions are made with context, never in a vacuum</h2> I also felt the weight of the context of the decision. JeanHeyd isn't just a recent grant recipient of the Rust Foundation. JeanHeyd has important history with the Rust project. It was JeanHeyd who called Rust out for having no Black representation among Rust conference speakers. Rightly so, as both the Rust organization and the conferences had little to no Black representation. When I saw an organization that not only could act so coldly to an expert in the field, but also to one who was a vocal critic of Rust's lack of diversity, it was hard not to see the additional context. Systems have memory and biases. If the people that make up the system don't work to fight against these, they are perpetuated. As my buddy Aman pointed out, the context that this would have also been the first keynote by a person of color at RustConf should not be lost here. It's time for accountability</h2> I read back over how I handled the situation as it unfolded, and I could have done much better. My go-to tool is diplomacy, trying to build bridges, getting information, and finding compromises. But, after reading over what happened and how I acted, it's clear part of the problem is too much diplomacy in leadership. Too many people giving each other the benefit of the doubt. Too many people asking for answers and then being satisfied to let bad decisions slide. So let me do my part to fix my failings: What we need isn't diplomacy, it's accountability. We need people to take responsibility for what happened and to make amends. We need people to step back from leadership who abused it to perform cruel acts against an expert in the wider programming language community. We need an organization that does not act cruelly so that the project it creates and the organization of people behind it can rebuild the trust it has lost. Questions that need to be answered</h2> Rust, as an organization, is guilty of cruelty. It needs to be held accountable. Here are the first questions that come to mind that need to be answered: How was it possible for a decision to be made without a leadership vote?</li> Why was downgrading the talk the only path explored?</li> Why did RustConf leadership go along with this decision and not protect the speaker? Why wasn't Rust leadership notified of the time period in which to change the decision?</li> Who ultimately is responsible for turning some team discomfort into disgracing a public expert? How are we going to hold them accountable?</li> How do we put safeguards in place so that this doesn't happen again?</li> How can Rust not only be held accountable now, but maintain long-term accountability into the future?</li> </ul> Making a keyboard layout 2023-02-22T00:00:00+00:00 It all started innocently enough. I had gotten interested in more efficient typing at first by learning about different keyboards and what they offered. I was intrigued. Except there was one small problem: fancy keyboards are expensive, especially to import into New Zealand. So, I took a different route: alternate keyboard layouts</a>. These promised to help you type better than the standard qwerty layout. I dove in. I tried colemak, workman, and some of their variants. It was clear with practice you could type better with less hand strain. A normal person at this point might have picked something and stuck with it. Well okay, most people wouldn't have even got that far. But I digress. I on the other hand thought "you know what would be cool? Making a layout for myself!" And down the rabbit hole I went. Why do a new layout?</h2> It started from a simple idea that came from playing guitar and later made its way to professional Tetris competition: rolling</a>. The motion you might have used as a kid when you were bored, drumming your fingers on the table. As the fingers fall one after another, you can create high speeds. The trick seemed simple enough. Lay out the keyboard to try to get the maximum benefit from rolling without making the typing that doesn't roll harder than before. While it's still too early to declare victory, I'll tell you that the text you are reading now was written using the layout I describe below. Creating the layout</h2> I created a few programs with a relatively simple goal. I want to be able to generate layouts that matched a set of constraints. Once I had one, I'd try it out, change the parameters. Rinse and repeat. In the end, I had gone through hundreds of layouts. I learned a few things. You need to find a corpus of text close to what you type</h3> Look through the data to make sure it will fit your writing style and that common words are covered. Be careful about using text with high counts of an uncommon word</h3> When I first started solving for a good keyboard, I used the source for Nushell, a codebase I've spent a lot of time in. This was great in theory, but in practice the algorithm tried to fit words like "value" and "span", pushing other words out of the way. Your hands might help you find what works for you</h3> I would try out an idea and often revise after seeing how that idea worked in practice. I tried many shapes to roll and over time simplified it down to a set that felt I could hit with high enough accuracy. It is easy to over-fit your data</h3> Scoring the different moves ended up being a bit of an art. Setting scores high enough, but not too high, took many iterations of trial and error. Show us the layout already</h2> Inas keyboard layout Here's the Inas layout (named by the home keys on the left hand). Let's talk about how I scored typing movements, and then talk about some ways we might improve on it. Scoring</h2> I broke my corpus of text up into words, then found 1-, 2-, and 3-letter runs of letters. These runs would then make up the "roll" that would get higher points. I used the following scoring: Typing on the home row is preferred</li> Center row keys are to be avoided (I learned from guitar that the lift of the finger can be easier on it than lateral stretches)</li> Runs of two should be together where possible</li> Jumps between the top row and bottom row should be avoided</li> Runs of three can fall in a row or can be two in a row and then fall by one row (in the code I call this a j-roll)</li> Try to use the pinkies less (unless necessary for the above)</li> </ul> If you look at the keyboard above you can see quite a few patterns that fall out from that: Common three letter runs like "ing", "you", "the", and "can" are easier to type.</li> Longer words that build on runs are also easier: "there", "this", "think", "thing", "your", "than", "care", "other", "another", and so on. Mixing and matching runs while you type ends up being a fun puzzle.</li> Surprisingly, some common words like "and" don't run. I think this comes from how important "ing" is in English. This takes precedence over the "n" key for rolls.</li> </ul> But is it good?</h2> I've been working with this keyboard layout from the perspective of potential speed. It's still too soon to know how fast you can get with it. There are also other considerations. Namely, does this help your hands? Layouts like qwerty over time can hurt your hands. Does Inas help at all with it? The jury is still out. One good iteration on what I have so far would be to score movements that are more ergonomic higher. Inas has some of that on a basic level by preferring smaller movements and preferring the home row. Are there more ergonomic patters to include? Undoubtedly. If you give it a try, let me know how it goes for you. Build your own</h2> If you want to improve what I have or build your own keyboard you can try it out for yourself</a> YouTube addiction, one month sober 2023-02-13T00:00:00+00:00 Today marks my first month clean from YouTube</a>. It was the right choice, but that's not to say it was easy. Let's talk about it. Quitting</h2> When I decided to do something about my addiction, I had a few choices: Cut back the amount I was watching</li> Quit cold turkey</li> </ul> They both have their pluses and minuses, but by the time I figured out what was going I was done with being hooked on it. I was ashamed of getting caught up in it so badly. I chose going cold turkey. Would I choose that path again? Good question, let's explore it. Cold turkey</h2> It was pretty soon after I decided to quit that I blocked YouTube on all my devices. For most things, that meant finding the /etc/hosts</code> file or equivalent for the OS, and adding a line to set youtube.com</code> to 127.0.0.1</code>. That effectively shuts you out of YouTube the site. On my phone, I used AppBlock</code> to block the website. I would have preferred to edit /etc/hosts</code> like I did with my machines, but unfortunately that part of the filesystem is read-only, and can only be modified if you jail-break your phone. App it is. Once I blocked it everywhere I could, I tested it a few times to be sure it was working. Satisfied, I settled in to see what would change. Changes</h2> A few things changed right away. Now that I wasn't obliterating myself for 8-10 hours a day, I started noticing a lot of things. One was that on an average day prior to quitting YouTube, I consumed a lot of sugar. Multiple desserts, plus sugary snacks in between meals, that kind of thing. Before, when I was watching hours of YouTube, I'm guessing I would get up and find something to snack on and sit back down. Now, I don't have that kind of "permanently distracted" frame of mind, I notice that I eat fewer snacks and cut way back on the sugar. I also had time to walk more - something I've always enjoyed. I started doing a 5km (3.1 mi) walk every morning. It takes me almost exactly an hour, and it's a nice way to start the day. But the real changes started happening with my mental health. Mental health</h2> One of the things that made me try to find out what was wrong with me was my declining mental health. Part of me wanted to become a shut-in and lose myself, and another part of me wanted to fight this. I felt worse as the weeks passed. Once I stopped YouTube, I knew I needed a few things: I needed to understand why I got addicted in the first place, and I needed to work on healthier habits. This is still very early on for me, but I wanted to mention a few things I did: I talked to my doctor, who put me in for addiction counselling services. Here in New Zealand, you can get five free visits with an addiction counsellor as you begin your recovery. I put in for the services, but I didn't know how long it would take to get into the system, so I also looked around locally for a therapist who could also give some support. This was a private therapist, so there are costs associated with it. The contrast between the two could not have been more stark. The therapist was very gentle and sympathetic. The counsellor... well, let's say they have a specific task at hand. Why are you addicted and what can you do better?</h2> Addiction counselling, as best as I can figure, has a different goal that general therapy. They're trying to hold up a mirror so you can see something very important for you to see: you are not taking care of yourself. More than that - your brain, in your current state, is not your friend. These are some hard truths to swallow. You work on habits you can change: how to make small changes each day, how to think about rewards and how you've been making bad assessments of what rewards are for and what you should do with them. Did I mention that's all in the first half hour? Then we started talking about the "why", in the same way of holding up a mirror and talking through ways of working with this stuff. I won't share the personal details here, but we covered a decent bit of ground. In the end, as best as I could understand, is that while you may have past experiences that caused your instincts to protect you, they're just that. We all try to protect ourselves from pain and danger. This is a good thing, unless it goes too far and we lose the ability to take normal daily risks and live a healthy life. In a sense, the goal of building new habits isn't to only have new habits. It's to start rewiring the brain to process risks, pain, and danger in a healthier way. There will always be risks, like crossing a street, and we re-learn how to work with them. To some folks, that might sound like total common sense. If it does for you, that's great. For me, it got surprisingly easy to create a safe illusion that turned into an unhealthy way to cope. I needed to see my way out of that. Other changes</h2> After talking with my doctor, I also started cooking more. I'm not a particular good cook (actually I'm pretty bad, though getting better). The act of cooking and eating your own food, though, is therapeutic. You can see how healthy (or not) the food you make is. You know the ingredients that go into it. You can make enough for leftovers. By the time I knew I needed to quit YouTube last month, I'd given up on cooking. I might make a pizza here or there, but almost all my lunches and dinners came from delivery food. Now, this is reversed. I'll still get the occasional delivery, but almost all of what I eat these days I make. Cravings</h2> You may wonder how the actual quitting part is going. The short answer is that I have good days and bad days. It's surprisingly easy some days to fill a day with fun things to do, so that I don't even notice it passing. Other days, I might do some work on the computer and then lay in the floor thinking about stuff I want to watch, scheming ways I could justifying watching some YouTube. That voice that says "watch a few hours to scratch the itch". This, I know, would be a bad idea(tm). Content as an addiction</h2> In talking with lots of folks, at this point I'm convinced we've got content wrong. At least, how it's delivered to us. You can take most of them and understand the business model as "retention = profits". The more time you spend on the site, the longer your eyeballs are taking in the ads and related, the more money the site makes off of you. The way sites like TikTok and YouTube give you a feed of content means you can consume it in an unbroken fashion. It reminds me of the people you see at slot machines in casinos, pulling the lever to see what will happen. Maybe you'll get lucky and come across something awesome. The content also keeps you in a kind of trance - a state where you don't have to think if you don't want to. I feel like one of those people complaining about how televisions make us zombies. I chatted with my addiction counsellor about about social media platforms, and I'm sure you're not at all surprised that they're also addictive. I don't want this post to be just about the dangers of these platforms. There's more to it than I can really cover here, and those platforms can be used for good as well. The main thing, for me, is how they interact with our brains. Are we still able to live healthy lives? Form deep healthy relationships? Take daily risks of being our complete selves? Are our brains staying healthy? Reaching out</h2> Probably the most important thing, though, is to directly target what I needed most of all. Addiction is isolating. The particular addiction, YouTube, is a lot of wanting to connect to others in a very safe way. So, to combat this, the goal is to "reach out". To connect to real people. To get my 'cup' filled with real intimacy and connections with others. What in the world that means is something I'm trying to figure out. I know that my cup was very, very empty. Despite having so many friends, fun online interactions, game nights, etc. Those things are great, but they don't meet the need for real connection. I'm signing up for meetups, volunteering, and going and chatting with random people where I can to get face-to-face interactions again. Step at a time</h2> That's it. That's where I'm at. Figuring out what I can, making changes where I can, and learning as I go. To all the folks who reached out with their stories of struggles and recovery: thank you. Sincerely, thank you. The suggestions and words of encouragement have helped. Thanks for that. Here's to the hope my cup, and yours, will be filled. YouTube Addiction 2023-01-13T00:00:00+00:00 Hi, I'm a YouTube addict. I know what you're thinking. That sounds like a joke. That you want a funny punchline to go with it. Me too. First time I noticed</h2> Truth is, though, I've had a problem for a while. I first noticed it back in 2011. Back then, I could feel it growing. I would spend something like 3 hours at night on weekdays and maybe around 10 hours total on the weekends on YouTube. I was mostly watching "let's plays". Long, winding playthroughs of video games with a friendly voice. Each series might, in total, be something like 40+ hours of content. Of which, the most prolific Let's Play-ers have dozens, if not hundreds, of series. From one creator, you'd find another, and another. Already, back in 2011 watching my little niche, it was more content than anyone one person would ever be able to watch in a lifetime. I would nap watching a series, and wake up to that same familiar voice. Episode after episode, I watched myself go deeper into the rabbit hole. At the time, I told my roommate that I thought I might have a problem. He thought the entire concept of being addicted to something like YouTube was laughable. How could someone be addicted to watching some videos on a website? Maybe some of you reading this are thinking the same. Still, it felt like the right thing at the time to try to curb this growing habit. I bought a stack of books on things like meditation and self-help and read everything I could. It helped a little. Meditation helped me to see just how much stress I was taking on. At the time, my roommate and I were interning at Apple. The pressure was high to do well and get noticed. I made a promise to myself as I got into self-help and meditation that I would stop using YouTube as a way to soothe this stress. That I could find other ways. I remember when I slipped back into it. My favorite YouTube content creator - after not posting for over a year - had posted the start of a new series. By this point, I'd been off YouTube for almost a year. I was used to doing other things with my time. And yet, when they posted that video, and I got a notification, I didn't say no. Since then, I don't think I've ever taken more than a couple months break from it. When I slip back into it, it effortlessly returns to being part of my daily routine. What do you mean by addicted?</h2> Yesterday, I watched 572 minutes of YouTube. That's just over 9.5 hours. The day before it was 444 minutes. Just shy of 7.5 hours. The day before that was 510 minutes. 8.5 hours. Sometimes it's background sound to whatever I'm doing. But if I'm honest, most of the time I'm just watching, going from video to video. Not doing anything else. At this point, I have so many interests I'll watch content for. Climbing videos, computing history, endless amounts of video game footage, music videos, concert footage, interviews, wordless videos of food being prepared in the morning in Japanese restaurants. Anything that could catch my eye, I'd watch. It's gotten to the point where I don't really enjoy the video as much as the numb hypnotic state from consuming that much content. It's safe. It's always there. But I noticed something recently. I've been struggling with my mental health during the summertime, a time I traditionally have fewer problems. I wasn't recharging like I should. I was waking up tired, napping during the day, and generally not feeling at all prepared when the day started. I'd tried to dial back on my commitments to see if I could recover. Weeks later, and I'm still not recovering. It didn't really hit me until today, when talking to a close friend, that it might be the amount of YouTube I'm watching. That's when I took my calculator and added it up. A familiar story</h2> I've heard this before. I had an alcoholic roommate years ago describe in detail what it was like for them to be an alcoholic. Why they would reach for it. When they'd reach for it. How they'd plan their life around it. Get the work done and use it to celebrate. I feel like my use of YouTube has basically mirrored their alcoholism at this point. If anything, I spend more time in stupor than they did with alcohol. The truth is, I don't know how long I've been this bad. I keep clearing my YouTube history. Ostensibly, it's so I can get better recommendations. But the truth is probably that I'm also trying to hide how bad it's gotten from myself. My roommate, when they finally came out about their struggles, let us clean their bedroom. I just remember duffle bag after duffle bag of empty bottles. It took so long to clean out their room because there were so many. I couldn't fathom how anyone could drink so much. It had to be hundreds of bottles. I don't know how long I've been averaging over 8 hours a day, but I'm afraid it's been a long time. I remember telling myself years ago that as long as I was getting my work done it didn't really matter what I did. I remember my roommate saying the same thing. Quitting</h2> I always thought that it's easy to quit something. You just don't do it. So naive. This isn't going to be easy. There are so many temptations to pull me back in that it'll be a challenge to try to put this behind me. I mean, I'm a YouTube content creator now approaching 10k subs - a respectable number for a small channel. So much educational content is on YouTube. So much of how I get my name out is via YouTube. So much of what I do is attached to YouTube for good and for bad. It's really all of the above. It's true it has all these nice aspects. But for folks like me, who take it to an extreme, it very politely chews its way into your life and makes it far too tempting to sink deeper and deeper. I don't want to lose the audience I'm building, but I also want to be healthy. I'm afraid for myself and where I'll be five years from now. Will I be worse than I am now if I don't take steps? Will I become more withdrawn, stop working altogether, and live on my savings until I run out? I think about other things, too, like "How will I fill my days if I stop"? I honestly don't know. It's scary to think about having a hole and not being able to fill it with that comfortable "zoning out" feeling. I'm sure some folks are thinking "what if you just limit yourself to 30 minutes a day?" That's just it. If I could do that, I would have. Admitting it</h2> I don't know if I want to do a 12-step program or whatever. I honestly haven't really figured anything out at this point. But the first thing I wanted to do was come clean and talk about where I'm at. If you're someone reading this with similar patterns, you're not alone. I also wanted to write this all down to make a record of where I am. Maybe I'll come back and read this later. Maybe I'll be able to say something like "I remember being hooked on YouTube for over 12 years. I'm glad I cleaned up." Let's hope. Maybe then I'll think up a good punchline, too. The goal is to nourish 2022-12-23T00:00:00+00:00 I just finished watching the movie called Stutz, a movie created by Jonah Hill about his psychiatrist. Through the movie, Stutz gives bite-sized practices drawn from Buddhism, psychiatry, and his own experiments. One of the themes is to accept that connection is the goal (connection with others, ourselves, and our body) and we connect using, among other things, vulnerability. For years, I've thought about a conversation I had with a friend of mine. He had worked with David Schnarch, a well-known psychology that wrote books like Passionate Marriage. In my friends work with David, they worked towards an end goal where yes, we should recover from our pain and trauma, and that will be work we continue with, but that we also have a choice to put light back out into the world. I knew as I talked to my friend about this, that this was unshakeably the goal. I knew that I could put my fears and trauma out into the world, but I also knew that without the light, it wouldn't end up helping myself or others. Not in the sense of sugar-coating my experiences, but rather that there is a light within us that is part of who we are, and that we can share our experiences and share our light at the same time. We don't have to hide our light for others to see our pain. We can show up as ourselves. Showing up, being vulnerable, being willing to fall on my face has always been hard for me. I wanted nothing more than to the gifted kid that never made errors. I wanted people to look up to me. To see me as some kind of spiritual giant. Or great intellect. To do nearly super-human feats. It wasn't until I got older that I started to see those desires as part of wanting to hide. If I do super-human feats, perhaps people will be distracted from the fact that I'm still a very human human. In recent years, I've started to slowly understand how intimacy and vulnerable don't have to come with rejection. Sure, someone may respond poorly to what I'm sharing, which hurts my feelings and makes me want to hide. But I'm also there in the room as I'm sharing. I'm also able to be a light for myself in that moment. When you carry your own light, now others can have their experiences, too, and you can hold space for them. Maybe it's from playing too much Stardew Valley, but I just get this mental image of waking up every morning and watering the crops. It becomes so routine you almost don't think about it. It teaches a subtle lesson. That if you nourish other beings a little each day, you grow as a result. I don't mean giving when you have nothing to give, draining the last bit of blood from your body. I mean just like you breathe oxygen, exercise, and try to eat well, giving a little to others is part of a healthy life. This morning I struggled with derealisation. It's an uncomfortable mental state where everything feels unreal. You may feel like you're watching yourself or your surroundings, but rather than you being the focus, it feels like you're watching a movie. I didn't panic. I just told myself that in the coming hours that it may go away and that I should give myself time to check in a little later and see if it's improved a little. By afternoon, it had lifted. I don't recount that as a kind of "wow, I'm great" moment. Instead, I share it because it's interesting to contrast what it was like 10 years. Months of gray weather in Seattle was too much for me, and I would go through weeks of on and off dealing with feeling like I was in a movie. It drained me. I panicked, worried, and felt like I didn't really have any grounding. While I think part of being able to deal with it comes with practice, I also think there's another important component. Feeling more connected, feeling like you can still live your life and use what you have to connect to other people (even in small ways) meant I didn't have to solve the problem right then or risk losing any hope of love and connection in my life. The love and connection are there, whether or not I could see and feel them. And they'd be waiting for me. And that comes from the regular nourishing of those connections, both with myself and with others. I'm not entirely sure how best to end this post. It's not like there's a magic formula that if you follow it you never hurt and never feel alone. I just wanted to share that having moments of appreciation with other people - even during difficult times - has been an important piece of the journey for me. Autism is everywhere and in everything 2022-07-13T00:00:00+00:00 I've been thinking recently about being on the Autistic spectrum. But first, some trigger warnings: topics include some talk of sex and gender, emotional pain, burnout. I thought it was about sexuality, or gender, or...</h2> I've wanted to talk about this for a long time. Before coming out as non-binary, I was diagnosed in 2020 as being on the autistic spectrum. It hadn't even crossed my mind until a close friend recommended I look into it. As I sat reading my diagnosis I just kinda started to unzip the first outer coat of many coats I'd learned to wear for the first 40 years of my life. I listened to podcasts, read articles, listened to other folks, and just tried to understand anything. Even more fundamental than sexuality, or understanding gender, was this idea that I had be fighting extremely hard to look and appear normal to the world. I was exhausted. Exhausted of opening up to people only to have them leave when they couldn't handle the intensity that comes naturally when I let my guard down. Instead of punishing myself, though, I had a key to understand something fundamental about myself. The way I viewed the world and how the world interacted with me as an experience was profoundly different than it was for most other people. With each thought, I felt like I stripped off another outer coat and set it aside, until I could no longer understand how to communicate with people anymore. I still knew the tricks I'd learned. I still know how to "mask", to use patterns I'd seen other people use to appear more normal. But what I wanted to know was "what was I like when I didn't do that?". In essence, who was I? When I told my ex, who I'd recently separated from, that I had been diagnosed she laughed at me. Rather than getting angry, it confirmed for me what I'd already known - there's just a deep-seated difference in the needs I have and the needs she had. In hindsight, it also confirmed another sad aspect of being on the spectrum: you can have a very large tolerance for people having bad expectations of you because you learn at an early age that you must accommodate others in order to survive. You have no basis for what a good boundary is. Perhaps just as bad, in my case I've found it exceedingly difficult to feel when the other person loves me, respects me, or otherwise. So I kept working harder in the hopes that at some point in the future something will feel right and things would click. For months after my diagnosis, I worked backwards. I wanted to reach an earlier version of myself to see how long I'd been this way. You'll probably be unsurprised to learn I've always been this way. I remembered countless events that now made so much more sense. Times of overstimulation, panic, being unable to understand why my parents were punishing me, feeling completely lost as to why the kids at the new school did things and why they singled me out. In time, as I worked backward, I began to ask questions like "how does sexuality relate to autism?" and "how does gender relate to autism?". Though the research here is fairly new, I came across terms like "neuroqueer" that talked about how autism can "queer" your own perception of your gender and sexuality. Wait. What? I tested it on myself. Was that what was happening? The more I let myself just experience my inner world and not require myself to use images and concepts that I learned from outside, the more I saw sexuality and gender as basically formless goo. A kind nothing concept that carried value for others, but didn't seem to hold much importance to me. When I say I'm non-binary, I mean...</h2> ...gender doesn't make sense to me as an internal experience. I wear the clothes I wear because I own them and they fit. I'm not thinking about what I'm wearing or how I'm presenting myself outside of the house unless my clothes are stained and dirty. Gender, to me, feels like those clothes. What it is like</h2> Sex is nice, but it has nothing on the bliss of a hyperfocus. Losing track of time, where hours are there merely to serve as a dimensional tube to channel more of your topic into you. People are lovely and should be a part of life, but it's sad that many folks will never understand what it's like. But this coin is so easily two-sided. In the computer industry, many a company has noticed that as long as they keep people like me "entertained with hard problems" we're happy. That is, until we burnout because no one reminded us to take care of ourselves. What it's like for me: Imagine not knowing if your partner likes you or if they're just tolerating you. Imagine this going on for years.</li> Imagine being in a tumble dry cycle of ups and downs and feeling like you come out of every social situation worse than you started.</li> Imagine letting yourself enjoy a topic you find fascinating and the joy feels nearly limitless as you dig deeper. Imagine this lasting for days, weeks, months, or even years.</li> Imagine being told you're weird most of your life and finding the only way to feel "not weird" is to completely ignore yourself.</li> Imagine not having the ability to socially read the person in front of you until you've spent many hours with them, or they remind you of someone you knew 10 years ago and you hope that they're similar enough they'll react to the same things.</li> </ul> If this sounds like you, perhaps you're like me. If so, welcome to the club. Autism club</h2> In autism club, we embrace what we are and who we are. There are snacks, heavy blankets, endless libraries, friends to do projects with, calm lighting, soft sounds, and a universal welcome to any and all parts of you. You don't have to mask, and we won't ask you to. Autism club is welcome to all ages. It is our hope that young and old from all walks of life will join this club and bring who they are to it. We make this club for ourselves. Here you can feel your feelings in the way they come out, as intense as they are or even the mildest whisper. You don't have to cut them off. You don't have to ignore the urge to self-stimulate to help direct the energies you feel. Carrying the club with you</h2> I've been trying to learn how to carry autism club with me, wherever I go. Not only so I can be welcoming to others, but I can also be welcoming to myself. The term self-care feels like it almost gets overused these days, but it's still just as important. Checking in and understanding your needs -- even the ones you were taught to ignore -- opens the door to giving yourself space. I, for one, need that space. And I hope as I grow older, perhaps I'll also grow more capable of being a better support to myself and others. Perhaps I'll also find the right people who can be that to me in return. And then?</h2> As I write this, I don't have the perfect thought to end on. Just this idea that everything is ever changing, and that's us too. There isn't an endpoint to this journey because what it means to think and feel and experience the world through the way we process information is so central. I carried it with me into the start of this post. And, as I finish, I'll carry it with me from here. How we judge things 2022-06-23T00:00:00+00:00 I've been thinking recently... The good, the bad, and the average</h2> Do you ever wonder about how we judge things? By this I mean, let's say we go to a restaurant with a friend. The meal is going well enough, then towards the middle of the meal we try our friend's dish. Wham! It's amazing! How do we judge the restaurant? Do we say it's just okay because that's how we felt about our meal? Or do we say it's amazing because of the bite we had of our friend's meal? Say we go watch a movie. The movie is pretty good, but there's this 20 minute section in the last third that just feels like it goes forever. Was the movie good, since we enjoyed the majority of it? Or do we feel the movie was a big let-down because of the drag at its slowest point? Do we judge something by its best moment, its worst moment, or by its average moment? There's also the question to ask - do we judge something by our experience or by the experience of those around us? - but I'm more interested in our subjective experience for now. Noticing how we do this for various forms of entertainment, we can also turn our eyes toward how we judge people. People we've just met. People we're friends with. Relationships we're in and how people act in those relationships. Do we judge people by their best moment? Worst moment? Or average moment? It's not that these are set in stone. For example, in social media perhaps we follow people who we feel on average are entertaining. Maybe some tweets don't makes us laugh, but enough do that we stick around. Let's say something comes out of their past that casts a shadow over them. We may shift from judging their average moment to now judging them by their worst. What I've wondered for myself is this - do I know what style of judgement I'm choosing? And, is that the appropriate kind of judgement for this situation? Imagine you have a friend that's generally fun, but sometimes is cruel. If you judge them by their best moments, you might get a sense of them "having such potential" because you see the high points. If you judge them by their average, perhaps you seem them as "a fun person, with a few quirks". If, instead, you judge them by their low moments, you may think "you know, they actually can be pretty mean". It seems like, in a way, all of these are true at once. It's us, as the judge, who get to say how much weight we put on each type of judgement. Redemption</h2> Stories give us another way to see how we view things as they tend to exaggerate some aspect of the human existence. One aspect of some stories is the "redemption arc". Can we see this in the context of the judgement from earlier? If we judge Darth Vader, we might get three different answers: By their best moment: a hero! By their average: a pretty bad person By their worst: a monster Stories like to tug on our emotions by playing against these different feelings, but what about how this translates to how we interact with people in the real world. Are we able to judge correctly if someone has redeemed themselves? Person or situation</h2> I was recently listening to a podcast, and the speaker mentioned a bit of advice they had gotten: if you are leading a team, do not be afraid to fire people. If working with them is taking energy, and that energy isn't making the product or you stronger, cut them off. It felt so cold, and yet, there's something decisive about it. They're looking at what is happening and judging not just the person but the situation as a whole. Are things going in the right direction? Even if someone is a star engineer on the team, if the situation shows that their average is that the whole team drags as a result of their actions, then perhaps that judgement is more important than their star contributions? A personal note</h2> I was raised to give people second chances. To be forgiving. That forgiveness was part of what you did. In hindsight, this seems to be asking us to focus only on the best moments, or perhaps to focus on the average at the expense of the bad. Instead, we should respect all of our judgements of both people and situations we find ourselves in. To weigh them all, so that we understand them, and then act based on all we see without letting ourselves be pre-disposed to any one style. Scripting languages of the future 2021-10-31T00:00:00+00:00 I've been thinking recently about what the future for scripting languages could look like. We've had a good run with Python, JavaScript, and Ruby for the last few decades. I wondered, what might the next few decades look like. What is the scripting language of the future? How should it work? Here are some thoughts. Scalability</h2> Working on TypeScript</a>, something that became a constant refrain for us was that "TypeScript is JavaScript that scales". I might be a bit biased, but this is actually incredibly valuable. If your team invests time into writing an app in a scripting language, there shouldn't be an impedance mismatch that grows as the app gets larger. For JavaScript, we'd watch teams grow their apps only to have to increase testing and onboarding training to prevent bugs from creeping in. And yet, they still crept in. TypeScript gave tools a chances to catch these errors earlier, even in well-tested codebases</a>. Back in 2012, this was something viewed with a healthy dose of skepticism, and today over third of professional developers use TypeScript</a>. What TypeScript showed is that you could join together the idea of a flexible lightweight (and optional!) type system onto an existing programming language, and do so successfully. The question then is - what if you created a programming language from the start to have this kind of support? No need for transpilers, while also having stronger engine integration and maybe even better runtime errors? It's a tempting mix of possibilities and one well worth being a part of the scripting language of the future. Tune-ability</h2> I've known multiple people who were hired to rewrite a piece of infrastructure originally written in a scripting language. Just like scripts need to be able to scale to larger sizes, scripts need to be tune-able. The choice to go dynamic shouldn't be a hard stop on real performance tuning and maintenance. The work on numpy, scipy, pandas and others for Python is one possible blueprint. Instead of having to leave your language of choice, powerful libraries build up around it to support the high performance needs of different disciplines. There will likely be some resistance to this point, as there seems to be just an inevitable trade-off between performance and the flexibility that a scripting language provides. In all honesty, this feels like an older, and perhaps outdated, way of looking at this trade-off. For years, JavaScript developers have looked to techniques like monomorphism</a> to tune the performance of JavaScript code. They're structuring code in a way that's more "VM friendly". The VM has to do less work to understand the types and structures in the code, and as a result guesses correctly the first time when doing runtime code generation. The ability to tune code can come from other sources, too. Type decorations is one tempting direction to mention, though so far no established languages (that I know of) have used the technique to general type hints to speed up code. Unless type hints are used across hot paths, and with a sound type system, the transformations risks actually slowing performance. There is, however, a large exception. Dataframe systems like pandas structure operations across arrays of known types. While the setup code is all in Python, the code that actually runs is happily in systems-level tuned native libraries. In a way, Python is merely orchestrating these operations. Future languages could take advantage of this pattern in other ways. The associated standard libraries of these scripting languages could encourage more orchestration and less manual processing in the language itself. Easy parallelism</h2> On a related point to the one above: future languages need easy parallelism. By now, we've heard it a million times. The "free lunch is over", tomorrow's CPUs will have a zillion cores and will run at 1hz. Okay, maybe not quite that bad, but still the writing is very clearly on the wall and very much already happening. It's not far-fetched to imagine that today's 8-16 core machines over the next few decades grow to be dozens or even hundreds of cores. Any language that hopes to have a strong footing, even scripting languages, need to be able to handle this. This isn't without its own challenges. As has been quotes a few times recently, Matz - the creator of Ruby - famously said "I regret adding threads"</a>. Having now used a lot of different kinds of parallelism systems in a lot of languages, I can easily see why. Traditional threading, and things like it, honestly feel like goto to me. Just like goto, it's easy to write code that's hard to read and, over time, hard to maintain. Luckily, traditional threads aren't the only game in town. Work-stealing task pools</a> like popular libraries in Rust use are very fast and can be built to work with other abstractions like parallel iteration and reduction operations. Making abstractions like this part of the language gives more opportunity to use more of the machine across all scripts written in that language. All this without, we hope, running afoul of the design choices that make language creators mourn years later. IDE support</h2> This might sound completely alien to some folks, but the future for scripting languages includes strong IDE support. Gone are the days of most scripting being written in a simple text editor. Today, more than 86% of JavaScript is developed in an IDE</a>. Future languages are competing against established languages which have tooling and IDE support polished over many years. Any new language that wants to compete against that will have to swing for the fences. At a minimum, support for the language server protocol</a> these days in a must. Its broad support across editors gives languages much more leverage than ever before to get in front of developers. Looking ahead</h2> There's definitely a large space to explore for scripting languages of the future. The points I list above are important, as are additional points that we'll learn as we go. Perhaps we'll be able to bring the performance that more modern JS engines enjoy to a broader range of languages through an effort similar to what LLVM did for lower-level languages. I, for one, am looking forward to this future and what future languages it will bring. Let's talk about exit codes 2021-10-16T00:00:00+00:00 Before starting a shell</a>, if you would have asked me what an exit code was, I would have quickly replied "0 for success, non-zero for failure" without batting an eye. That's what it meant, right? Today, someone filed an issue on the Nushell repo</a>. In it, they ran git log</code>, exited, and Nushell reported the command to have failed. To know why it happens, we've got to do some digging. First, let's take a look at the standards to understand what we should expect. What the standard says</h1> Bash</h2> One of the first places you'll learn about exit codes is the Bash manual. It says: "A successful command returns a 0, while an unsuccessful one returns a non-zero value that usually may be interpreted as an error code. Well-behaved UNIX commands, programs, and utilities return a 0 exit code upon successful completion, though there are some exceptions." (emph. mine) POSIX</h2> The POSIX Open Group Base standard says: "The value of status may be 0, EXIT_SUCCESS, EXIT_FAILURE, [CX] [Option Start] or any other value, though only the least significant 8 bits (that is, status & 0377) shall be available from wait() and waitpid(); the full value shall be available from waitid() and in the siginfo_t passed to a signal handler for SIGCHLD. [Option End]" On the same page it says: "The functionality described on this reference page is aligned with the ISO C standard. Any conflict between the requirements described here and the ISO C standard is unintentional. This volume of POSIX.1-2017 defers to the ISO C standard." ISO C</h2> Okay, so what's the ISO C standard say? "If the value of status is zero or EXIT_SUCCESS, an implementation-defined form of the status successful termination is returned. If the value of status is EXIT_FAILURE, an implementation-defined form of the status unsuccessful termination is returned. Otherwise the status returned is implementation-defined." "an implementation-defined form of the status unsuccessful termination is returned" feels unsatisfying, but at least we know that non-zero means unsuccessful, right? The bug</h1> The original report was about git log</code>. Let's run the steps the user reports: run git log</code>, quit with q</code>, and then check the exit code. I'll do this in bash, since that's the shell everyone will have access to: > git log > echo $? 141 </code></pre> Sure enough, a non-zero exit code. Bash is reporting that git log</code> exits with exit code 141. This clearly feels intentional. It's not a 0 or 1, but a number specific to the other 254 possibilities. Turns out that some folks dug into this 5+ years ago</a>. They write a bullet list with their findings: Git pipes the logs into less</li> I quit the less process, which sends a SIGPIPE signal (13) to the underlying git process streaming the logs</li> git catches the interrupt and exits prematurely and per POSIX convention returns 128 + the SIGPIPE status ==> 141 to indicate that it was terminated by signal 13.</li> </ul> When failures aren't failures</h1> Why is the user being told about the SIGPIPE in this case? I'm sure there's a perfectly reasonable set of events that take us to this point. Git is calling out to less</code>, and less</code> wants to report the user quitting it ends the pipe prematurely. And yet, it definitely feels to fall short of the spirit of the specifications. Non-zero errors should mean we've exited in an unexpected way. The steps above exit us in about the only reasonable way to exit interactively viewing the log. As it turns out, git log</code> isn't alone. Here's a discussion on Stack Overflow about grep -q</code> returning 141</a>. In it, the responses point out yet other commands, like head</code>, also return 141 in their regular course of action. There are standards, and there are standards</h1> Something folks tend to assume is that the POSIX standard covers a lot of how Unix-like systems work. While this is true at some level, there are cases like this one where we quickly shift to having to lean more on conventions than standards. As the Bash manual says "though there are some exceptions". Using pipes and command composition leads to having to follow one part of the standard that in turn leads to really confusing situations for the user, like the situations listed above. A command like head</code> or less</code> doesn't take all of its inputs. As it closes, it passes along that "hey, technically the pipe I was reading from isn't going to be able to write to its output because I'm closing". From the user's perspective, though, exiting early is exactly what's supposed to happen. It's not actually an error. It kinda makes you wonder - if we could do this over again, how would we do it differently? Retiring 2021-03-03T00:00:00+00:00 Yesterday was my last day at Microsoft. I've been fortunate to have worked in an industry that paid well over my 20+ year career and to have grown up with frugal parents who passed that on to me. Starting the 4th of March, I'll be retiring to work on open source and teaching. So what's next? I have a few projects in mind: Get Nushell to 1.0</h2> We're currently working on the plans for what it will take to get Nushell ready for 1.0. For the last year and a half, Nushell has been quite the passion project for me, and being at the place where we can talk about 1.0 and how to get there has me excited. We'll publish these plans once they're ready so we can get your feedback, but suffice to say there's lots of cool things we'd like to do. Finish the Rust beginner series</h2> I've been enjoying working on the Rust beginner series</a>, where I work through the concepts you'll need as a beginner in Rust. It's more of an overview of the language and how language features fit together. I'm having fun with it, and hearing some good feedback, which is always nice. Work on the Rust intermediate series</h2> I recently asked on Twitter</a> about intermediate Rust content, which inspired me to start an intermediate Rust series. You can find out more information on this series in this blog post</a>. Keep doing Systems with JT</h2> I've been having a blast making videos about various OSes</a>. What started as a "it'd be fun to make some videos on these two hobby OSes" has turned into a weekly video, and I'd like to keep this going. How can we help?</h1> Probably the best thing is encouragement! Say hi on Twitter</a> or come by the YouTube channel</a> or Twitch stream</a>. If you have the funds and don't mind chipping in, I have Patreon</a>, too. This will help me upgrade hardware, improve video editing, and more. If you'd specifically like to help Nushell</a> hit 1.0, I'd be happy to show you around. We've got a discord</a> where you can join us and chat about what's happening and where to jump in. There are lots of places to help with code, docs, designs, and more. We'll also soon be posting the proposal for getting to 1.0, which will include lots of areas that you could help with. Conclusion</h1> I'm both nervous and excited. Though I've planned as best I can and I live as simply as I can, it feels like taking a leap into the big blue unknown. But I'm all for following the heart and seeing what happens next. Intermediate Rust series 2021-02-12T00:00:00+00:00 Hi all! A few days ago I posted on Twitter</a> asking for topics that people wanted to see covered for intermediate Rust content. The response was awesome and it's inspired me to make a series where I go through each topic and do my best to cover it as a video and/or blog post. Here's the current list of topics. If you see something missing, you can drop an issue on this repo</a> or ping me on Twitter</a> and I'll try to add it. Rust features</h2> Associated types</li> Working with lifetimes</li> Pin</code> and Pin<Box<Stream>></code></li> Macros, proc macros, custom attributes</li> Concurrency</li> Async/await</li> Threads/tasks</li> MPSC theory</li> Futures</li> Send/Sync</li> Unsafe</li> FFI, shared mem across FFI, closures/function pointers over FFI</li> FFI to C and C++</li> C interop, opaque structures, repl(c), working with null, alloc/free, bitflag export to C</li> What tools to use for C++</li> std::sync</code> (Arc, Rc, RefCell, etc)</li> Raw pointers</li> Impl trait, dyn trait, and trait objects</li> Fn traits</li> Field sensitivity, field sensitivity + smart ptrs, field sensitivity and closures</li> Why do we cast from Box<Struct></code> to Box<dyn Trait></code></li> How to use [patch]</code> in Cargo.toml</li> std::mem</code>, swap, replace, take</li> </ul> Crates</h2> Tonic</li> Tokio</li> Deeper dive into existing frameworks/libraries</li> Channels/async/par_iter (concurrency vs parallelism)</li> </ul> Projects</h2> Basic SQL database</li> Interpreters and parsers</li> Tokio + io_uring for CLI apps</li> Filesystem + FUSE</li> Seda or actor based project</li> Wasm + wasm bindgen</li> </ul> Patterns and practices</h2> Like http://railstutorial.org but for Rust</li> Functional composition</li> Async functional composition</li> Dependency injection (and alternatives)</li> Design patterns</li> Lessons from Rust in production</li> How to teach Rust to teams</li> Testing patterns and strategies</li> https://rust-lang.github.io/api-guidelines/checklist.html</li> Zero copy processing best practices</li> Error handling. One big error enum or something extensible? Anyhow?</li> Debugging Rust</li> Modelling classic data structures in Rust</li> Logging</li> Shared data structures in long-running apps</li> Profiling</li> Improving compile times</li> </ul> Rust background</h2> How do nightly features stabilize?</li> </ul> Looking ahead</h1> As you can see, there's a lot of area to cover. My plan is to make steady progress on the list above, covering topic by topic until we cover as many of the topics as possible. People were also upvoting tweets with likes, so I may juggle this list around so that I'm sure to cover the hot topics sooner. Excited to dig into this! If you'd like to follow along to enjoy the content as it comes out, you can follow me on Twitter</a>, you can follow my YouTube channel</a> where I post my technical content, and follow me on Twitch</a> for when I stream a topic. Nushell 0.5.0 2019-11-05T00:00:00+00:00 Nushell, or Nu for short, is a new shell that takes a modern, structured approach to your commandline. It works seamlessly with the data from your filesystem, operating system, and a growing number of file formats to make it easy to build powerful commandline pipelines. Today, we're happy to announce the 0.5.0 release for Nu. We've got lots of new features, including some long-requested ones, in this release. Where to get it</h1> Nu 0.5.0 is available as pre-built binaries</a> or from crates.io</a>. If you have Rust installed you can install it using cargo +beta install nu</code> (or if you want all the features cargo +beta install nu --all-features</code>). Nu as a login shell (sophiajt)</h1> One of the key features that landed in 0.5.0 is set of new capabilities that work together to allow you to use Nu as a login shell, completely independent of bash or other hosting shells. For this, we've built in support for querying and updating the environment variables and the path. To get started, you'll need to first copy the environment you're using into the config. Luckily, we've also made some improvements there: > config --set [path $nu:path] > config --set [env $nu:env] </code></pre> Once these values are set, you'll be able to use Nu as your login shell. New variables</h2> As you saw above, we've added a few new built-in variables. These will let you know what the current values that Nu can see are for your environment, path, and the config itself. > echo $nu:env > echo $nu:path > echo $nu:config </code></pre> Adding paths to your PATH</h2> One of the first things you'll notice is that the new $nu:path is structured. If you run the echo above, you might see something like this: > echo $nu:path ━━━┯━━━━━━━━━━━━━━━━━━ # │ <value> ───┼────────────────── 0 │ /usr/local/sbin 1 │ /usr/local/bin 2 │ /usr/sbin 3 │ /usr/bin 4 │ /sbin 5 │ /bin 6 │ /usr/games 7 │ /usr/local/games 8 │ /snap/bin ━━━┷━━━━━━━━━━━━━━━━━━ </code></pre> We've added two new commands: prepend</code> for adding items to the start of a table and append</code> for adding items to the end of a table. With these commands, we can now query out the path, update it, and save it back. > echo $nu:path | prepend "/my/new/directory" | config --set_into path </code></pre> Adding variables to your environment</h2> You can use a similar set of steps to add new variables, or change existing variables, in your environment. > echo $nu:env | insert GREETING hello_world | config --set_into env </code></pre> Note: the previous add</code> command of previous releases has been renamed insert</code> to remove confusion with mathematical functions. On-going improvements</h1> We're continuing to improve the commands we currently ship as part of Nu. Here are a few you might find helpful: Substrings (Flare576)</h2> The str</code> command now supports being able to retrieve a substring from the strings given, so you could return, for example, the first 5 characters and stop after that. > ls | get name ━━━━┯━━━━━━━━━━━━━━━━━━━━ # │ <value> ────┼──────────────────── 0 │ target 1 │ CODE_OF_CONDUCT.md 2 │ .cargo 3 │ src 4 │ features.toml 5 │ rustfmt.toml </code></pre> > ls | get name | str --substring "0,3" ━━━━┯━━━━━━━━━ # │ <value> ────┼───────── 0 │ tar 1 │ COD 2 │ .ca 3 │ src 4 │ fea 5 │ rus </code></pre> Recycling (jdvr)</h2> Ever wish you could rm</code> things, but not forever? You can now tell rm</code> to send items to your platform's recycle bin rather than deleting them forever. As with our other commands, this works across all the platforms that Nu supports. > rm myfile.txt --trash </code></pre> Parameter descriptions (sophiajt)</h2> We're also continuing to improve the built-in help system. New in this release are descriptions for the flags and parameters that the command uses. For example, here's a look at what the help for rm</code> now looks like: > help rm Remove a file Usage: > rm <path> {flags} parameters: <path> the file path to remove flags: --trash: use the platform's recycle bin instead of permanently deleting --recursive: delete subdirectories recursively </code></pre> New commands</h1> In addition to the new append</code> and prepend</code> we mentioned earlier, we've added a few new commands to Nu. Average (notryanb)</h2> Growing our set of mathematics functions for working with numbers, we now have an average</code> command which will take the average of values given to it. > ls | get size | average </code></pre> Read (sophiajt)</h2> We've also introduced a new command to load in strings as tables. This new read</code> command will take a pattern that describes the columns, their names, and where they are in each row of the string. > open .editorconfig root = true [*] indent_style = space indent_size = 4 charset = utf-8 trim_trailing_whitespace = true insert_final_newline = false end_of_line = lf </code></pre> > open .editorconfig | read "{variable} = {value}" ━━━┯━━━━━━━━━━━━━━━━━━━━━━━━━━┯━━━━━━━ # │ variable │ value ───┼──────────────────────────┼─────── 0 │ root │ true 1 │ indent_style │ space 2 │ indent_size │ 4 3 │ charset │ utf-8 4 │ trim_trailing_whitespace │ true 5 │ insert_final_newline │ false 6 │ end_of_line │ lf </code></pre> Bugfixes (sophiajt, JesterOrNot, thegedge, andrasio, wycats, notryanb, Detegr, t-hart)</h1> As always, we've had lots of bugfixes. A huge "thank you!" to folks who reported issues, fixed issues, and just generally shared their experience with the shell. It's much appreciated and helps to continue making Nu that much better. Survey</h1> If you haven't already taken it, we'd love to hear your feedback in a quick (roughly 3 question) survey</a>. Looking forward</h1> There are a bunch of areas we're currently working on to make Nu more stable and feature complete. Until this work is finished, please consider Nu to be somewhere in the pre-alpha to alpha quality level. That said, we're excited to merge this work and continue to take steps towards a more full-featured shell. Soon to be coming will be Nu working on stable Rust(!!), some improvements in the Nu internal engine which will make it possible to have better streaming and cleaner commands, and features like aliases. Nushell 0.4.0 2019-10-15T00:00:00+00:00 Nushell, or Nu for short, is a new shell that takes a modern, structured approach to your commandline. It works seamlessly with the data from your filesystem, operating system, and a growing number of file formats to make it easy to build powerful commandline pipelines. Today we're happy to announce the 0.4.0 release of Nushell. The 0.4.0 release</a> marks a continually maturing shell that is now starting to show signs of stability. And, of course, a few fun features along the way. Where to get it</h1> Nu 0.4.0 is available as pre-built binaries</a> or from crates.io</a>. If you have Rust installed you can install it using cargo +beta install nu</code> (or if you want all the features cargo +beta install nu --all-features</code>). What's new</h1> New Colors (wycats)</h2> New colors! With 0.4.0, we're adding some new colors to show off different types of the command, where errors might be happening, and if the command is internal or external. The coloring is just the tip of the iceberg, as it were, and builds from a reworking of the parser to make it more accurate, more stable, and more feature-complete. Streaming table (sophiajt)</h2> Streaming tables Up to this point, we've taken a few shortcuts with how streams worked in Nushell. We knew that a table, in theory, was a stream of rows. In practice, though, this wasn't the case as the code has lots of assumptions about how tables worked. With the 0.4.0, we're taking a big step in the direction of full support for streams. You'll see that now data will start being printed as it becomes available, with tables printing pages of 50 rows at a time (by default). This allows you to get deep listings of files (eg using ls **</code>), work with streams from external sources, and more. Better ctrl-c support (sophiajt, landaire)</h2> As part of having streams of tables, it's always helpful to be able to say "stop!" when you need to. With 0.4.0, using ctrl-c to stop the output should be a lot more responsive and will take you back to the prompt. Textview gets VI keys (JonnyWalker81)</h2> The built-in text viewer now lets you use vi keys for navigation thanks to JonnyWalker81's work. Onto the Beta compiler (est31)</h2> For those of you wondering if and when Nu was going to be usable on stable Rust, you need but wait just a little longer. Contributer est31 has been hard at work at moving Nu onto stable Rust and has now successfully moved Nu off of nightly and onto beta. As beta matures and becomes the next release of Rust, we'll be able to move Nu onto stable. This should help people who are working to distribute Nu on various distributions. Brazilian Portuguese book (marcelocg)</h2> Contributor marcelocg surprised us with a translation of the Nu book into Brazillian Portuguese. Obrigado, Marcelo! New commands</h2> from-ssv (t-hart)</h3> To follow along with our from-csv</code> (comma-separated-values) and from-tsv</code> (tab-separated-values), we now have a new way of handle importing content from commands that output their own tables: from-ssv</code>. Like its siblings, it looks for values separated by a separator, using the spaces to show where the breaks between cells should be. match (rnxpyke)</h3> We're continuing to evolve our support for matching string data. Recently, we began experimenting with a match</code> command that can find matches in strings for you. We plan to continue experimenting and exploring what's possible in this space in the future. Lots of new documentation (sdfnz, twe4ked, chhetripradeep, mfarberbrodsky, cristicismas, Charles-Schleich, coolshaurya, notryanb, pema99, gilesv, yahsinhuangtw, JesterOrNot, nalshihabi, marcelocg, sorrell, pizzafox, iggy14750, DrSensor, mlbright, BradyBromley, andrasio, piotrek-szczygiel)</h2> Happy hacktoberfest! Lots of people wanted to jump in and help document all of our commands, help fix docs that are already there, and generally give things a bit of polish. Much appreciated everyone who came by and contributed. Tons of stability fixes (thegedge, sophiajt, wycats, JonnyWalker81, vsoch, pka)</h2> As always, there were a bunch of stability and correctness both in Nu and in the CI that support Nu's development Looking ahead</h1> We'll continuing filling out the features that will let folks use Nu as their everyday shell. Coming up are better support for the environment, path, and for command aliases. Nushell 0.3.0 2019-09-24T00:00:00+00:00 Nushell, or Nu for short, is a new shell that takes a modern, structured approach to your commandline. It works seamlessly with the data from your filesystem, operating system, and a growing number of file formats to make it easy to build powerful commandline pipelines. We're happy to announce that today we're releasing Nushell 0.3.0. Nu has seen numerous bugfixes, performance improvements, and features added since its initial public release (which was only a few weeks ago!) Where to get it</h1> Nu 0.3.0 is available as pre-built binaries</a> or from crates.io</a>. If you have Rust installed you can install it using cargo +nightly install nu</code> (or if you want all the features cargo +nightly install nu --all-features</code>). What's new</h1> New table design (Porges, sophiajt)</h2> One of the most striking differences is that Nu now uses UTF-8 box drawing to draw its tables. The table is also configurable. If you feel like the table is a bit too heavy, you can also configure it to work in light mode using this command: > config --set [table_mode light] </code></pre> New file formats (pmeredit, est31, andrasio, sophiajt)</h2> Nu now natively supports new file formats, including: bson, tsv, sqlite, and url-encoded strings. New commands (incrop, ramonsnir, ijt, sophiajt, andrasio, JonnyWalker81, chhetripradeep, pka)</h2> Since the 0.2.0 release, Nu has gained a set of additional commands, including: help</code> - built-in help system</li> reverse</code> - reverse a table</li> last</code> - like the first</code> command, but working from the bottom of the table, return n rows</li> embed</code> - create a new table using the current table as a start</li> fetch</code> - get the contents of a URL (this originally lived in open</code>)</li> post</code> - post to a URL and get the results</li> pwd</code> - print the working directory</li> env</code> - give access to a number of important paths and environment settings</li> pivot</code> - pivot a table so that rows become columns and vice-versa</li> echo</code> - a built-in echo command</li> </ul> Error improvements (sophiajt)</h2> We've made steady progress on improving error messages. Recently, we added "did you mean?" errors to help when you mistype the name of columns. Nu 0.3.0 also has gone through a few passes to generally polish the errors to include more information when an error occurs. Shell features (pka, wycats, twe4ked, andrasio, iamcodemaker)</h2> Nu now supports ~</code> to refer to your home directory as part of a path. You can also use cd -</code> to return to a previous directory you were working in. The shell itself also got a few updates. You can now enable 'vi' mode, if you'd prefer vi-like bindings (by running config --set [edit_mode vi]</code>). On non-Windows platforms, you can also use Ctrl-R to enable Sublime-style history searching (we're hoping to enable Windows support for this soon). Plugin improvements (sophiajt, andrasio)</h2> Plugins have continued to mature, and it's now possible to use plugins to extend Nu with support for new file formats. We've also added documentation on how to write Nu plugins in both Rust and Python</a>, with info on Ruby coming soon. Docker support (vsoch, DrSensor)</h2> If you're interested in using Nu and Docker together, be sure to check out the new Docker support</a>. Book progress</h2> The Nu book</a> is now available in three languages: English, Spanish, and Japanese. We've also created a Nu contributor book</a>, which will help developers who want to contribute to Nu learn about its philosophy, design, and how to create plugins. Acknowledgements</h2> est31, pmeredit, twe4ked, DrSensor, vsoch, pka, jankoprowski, Porges, max-sixty, ijt, djc, vthriller, taiki-e, oskarskog, iamcodemaker, JonnyWalker81, yaahc, tim77, svartalf, ramonsnir, orf, lesichkovm, ineol, incrop, eoinkelly, devnought, chhetripradeep, aidanharris, GuillaumeGomez, and BatmanAod for contributing to the Nu codebase. Also a big thanks to our issue reporters! ymgyt - for the Japanese version of the Nu book! mistydemeo, romanlevin, ralvessa, pka, lord, kkalyan, jankprowski, and boisgera for helping with the book. Looking ahead</h2> We're hard at work on the upcoming milestone release</a>. If you'd like to help, feel free to jump in! We have an active discord</a> you can join for both users and contributors. Introducing nushell 2019-08-23T00:00:00+00:00 Today, we're introducing a new shell, written in Rust. It draws inspiration from the classic Unix philosophy of pipelines, the structured data approach of PowerShell, functional programming, systems programming, and more. It's called Nushell, or just Nu for short. We have a book</a> (¡también se habla Español</a>!). We have a repo</a>. This release was made by Sophia Turner (me), Yehuda Katz, and Andrés Robalino, with contributions from Odin Dutton. But why?</h1> Many of us have gotten used to bash (or zsh/fish/etc), and don't understand why you would need another kind of shell. That was me, too, a few months ago before I started working on this. My friend Yehuda had discovered PowerShell and was going on and on about how amazing it was to do more with the shell, but until he actually gave me a demo, I didn't really believe him. Then he talked me into joining him on an idea he had. What if we could take the ideas of a structured shell and make it more functional (as opposed to object-oriented)? What if, like PowerShell, it worked on Windows, Linux, and macOS? What if it had great error messages? I fell in love with the project ideas, made a few new friends, and many nights and weekends later I'd like to show you what we've made. In this post, I'll talk about how a few simple ideas drive how Nu works, what Nu can do with them, and where we hope to go in the future. Simple ideas</h1> To Nu, everything is data. When you type ls</code>, you're given a table of information about the directory you're listing: Rather than having to remember different flags to ls</code>, we can just work with the data it gives back. We can find the files greater than a certain size: Or we could choose to sort it by a column, or only show directories, or more. That by itself is fun but perhaps not compelling enough. Where this simple concept - that everything in Nu is data - starts to shine when we try other commands and realize that we're using the same commands to filter, to sort, etc. Rather than having the need to remember all the parameters to all the commands, we can just use the same verbs to act over our data, regardless of where the data came from. Nu pushes this idea even further. Nu also understands structured text files like JSON, TOML, YAML, and more. Opening these files gives us the same tables we saw with ls</code> and ps</code>. Again, this lets us use the same commands to filter our data, explore it, and use it. Working with the outside world</h1> The above approach could be fun, but if we're not careful, it could become a walled garden. What happens outside of the commands Nu comes with? First, let's take a look at working with a file that Nu doesn't understand. > open people.psv Octavia | Butler | Writer Bob | Ross | Painter Antonio | Vivaldi | Composer </code></pre> To work with this in Nu, we need to do two steps: figure out where the rows are, and then figure out what the columns are. The rows are pretty easy, we just have one record per row: > open people.psv | lines ---+------------------------------ # | value ---+------------------------------ 0 | Octavia | Butler | Writer 1 | Bob | Ross | Painter 2 | Antonio | Vivaldi | Composer ---+------------------------------ </code></pre> Next, we can create our columns by splitting each row at the pipe (|</code>) symbol: > open people.psv | lines | split-column "|" ---+----------+-----------+----------- # | Column1 | Column2 | Column3 ---+----------+-----------+----------- 0 | Octavia | Butler | Writer 1 | Bob | Ross | Painter 2 | Antonio | Vivaldi | Composer ---+----------+-----------+----------- </code></pre> That's already good enough that we can work with the data. We can go a step further and name the columns if we want: > open people.psv | lines | split-column " | " firstname lastname job ---+-----------+----------+---------- # | firstname | lastname | job ---+-----------+----------+---------- 0 | Octavia | Butler | Writer 1 | Bob | Ross | Painter 2 | Antonio | Vivaldi | Composer ---+-----------+----------+---------- </code></pre> But what about working with commands outside of Nu? Let's first call the native version of ls</code> instead of the Nu version: > ^ls assets Cargo.lock docs images Makefile.toml README.md rustfmt2.toml src tests Cargo2.toml Cargo.toml extra LICENSE open readonly.txt rustfmt.toml target </code></pre> We'll use the same commands we used on data to bring it into Nu: ^ls | split-row " " file ----+--------------- # | value ----+--------------- 0 | assets 1 | Cargo2.toml 2 | Cargo.lock 3 | Cargo.toml 4 | docs 5 | extra 6 | images 7 | LICENSE 8 | Makefile.toml 9 | open 10 | README.md 11 | readonly.txt 12 | rustfmt2.toml 13 | rustfmt.toml 14 | src 15 | target 16 | tests ----+--------------- </code></pre> Or maybe we want to work with the native ls -la</code>: ^ls -la | lines | split-column " " ----+------------+---------+----------+----------+---------+---------+---------+---------+--------------- # | Column1 | Column2 | Column3 | Column4 | Column5 | Column6 | Column7 | Column8 | Column9 ----+------------+---------+----------+----------+---------+---------+---------+---------+--------------- 0 | total | 296 | | | | | | | 1 | drwxr-xr-x | 13 | sophia | sophia | 4096 | Aug | 24 | 03:24 | . 2 | drwxr-xr-x | 21 | sophia | sophia | 4096 | Aug | 22 | 17:00 | .. 3 | drwxr-xr-x | 2 | sophia | sophia | 4096 | Aug | 3 | 05:39 | assets 4 | drwxr-xr-x | 2 | sophia | sophia | 4096 | Aug | 21 | 19:29 | .azure 5 | drwxr-xr-x | 2 | sophia | sophia | 4096 | Jun | 23 | 05:09 | .cargo 6 | -rw-r--r-- | 1 | sophia | sophia | 2963 | Aug | 22 | 20:17 | Cargo2.toml 7 | -rw-r--r-- | 1 | sophia | sophia | 201255 | Aug | 24 | 03:24 | Cargo.lock 8 | -rw-r--r-- | 1 | sophia | sophia | 3127 | Aug | 24 | 03:24 | Cargo.toml 9 | drwxr-xr-x | 2 | sophia | sophia | 4096 | Jun | 17 | 15:32 | docs 10 | -rw-r--r-- | 1 | sophia | sophia | 148 | Jun | 17 | 15:32 | .editorconfig 11 | drwxr-xr-x | 4 | sophia | sophia | 4096 | Aug | 22 | 19:29 | extra 12 | drwxr-xr-x | 8 | sophia | sophia | 4096 | Aug | 24 | 03:24 | .git 13 | -rw-r--r-- | 1 | sophia | sophia | 58 | Aug | 10 | 11:08 | .gitignore 14 | drwxr-xr-x | 2 | sophia | sophia | 4096 | Aug | 24 | 03:24 | images 15 | -rw-r--r-- | 1 | sophia | sophia | 1085 | Jun | 17 | 15:32 | LICENSE 16 | -rw-r--r-- | 1 | sophia | sophia | 614 | Jun | 17 | 15:32 | Makefile.toml 17 | -rw-r--r-- | 1 | sophia | sophia | 0 | Aug | 23 | 04:58 | open 18 | -rw-r--r-- | 1 | sophia | sophia | 11375 | Aug | 24 | 03:24 | README.md 19 | -r--r--r-- | 1 | sophia | sophia | 0 | Jul | 4 | 03:51 | readonly.txt 20 | -rw-r--r-- | 1 | sophia | sophia | 37 | Aug | 23 | 04:54 | rustfmt2.toml 21 | -rw-r--r-- | 1 | sophia | sophia | 16 | Aug | 1 | 19:45 | rustfmt.toml 22 | drwxr-xr-x | 10 | sophia | sophia | 4096 | Aug | 24 | 03:24 | src 23 | drwxr-xr-x | 4 | sophia | sophia | 4096 | Aug | 22 | 19:22 | target 24 | drwxr-xr-x | 4 | sophia | sophia | 4096 | Aug | 22 | 04:15 | tests 25 | drwxrwxr-x | 2 | sophia | sophia | 4096 | Jul | 19 | 15:18 | .vscode ----+------------+---------+----------+----------+---------+---------+---------+---------+--------------- </code></pre> After a bit of experimenting, we might come up with a command like this: > ^ls -la | lines | skip 1 | split-column " " perms files group user size month day time name ----+------------+-------+----------+----------+--------+-------+-----+-------+--------------- # | perms | files | group | user | size | month | day | time | name ----+------------+-------+----------+----------+--------+-------+-----+-------+--------------- 0 | drwxr-xr-x | 13 | sophia | sophia | 4096 | Aug | 24 | 03:24 | . 1 | drwxr-xr-x | 21 | sophia | sophia | 4096 | Aug | 22 | 17:00 | .. 2 | drwxr-xr-x | 2 | sophia | sophia | 4096 | Aug | 3 | 05:39 | assets 3 | drwxr-xr-x | 2 | sophia | sophia | 4096 | Aug | 21 | 19:29 | .azure 4 | drwxr-xr-x | 2 | sophia | sophia | 4096 | Jun | 23 | 05:09 | .cargo 5 | -rw-r--r-- | 1 | sophia | sophia | 2963 | Aug | 22 | 20:17 | Cargo2.toml 6 | -rw-r--r-- | 1 | sophia | sophia | 201255 | Aug | 24 | 03:24 | Cargo.lock 7 | -rw-r--r-- | 1 | sophia | sophia | 3127 | Aug | 24 | 03:24 | Cargo.toml 8 | drwxr-xr-x | 2 | sophia | sophia | 4096 | Jun | 17 | 15:32 | docs 9 | -rw-r--r-- | 1 | sophia | sophia | 148 | Jun | 17 | 15:32 | .editorconfig 10 | drwxr-xr-x | 4 | sophia | sophia | 4096 | Aug | 22 | 19:29 | extra 11 | drwxr-xr-x | 8 | sophia | sophia | 4096 | Aug | 24 | 03:24 | .git 12 | -rw-r--r-- | 1 | sophia | sophia | 58 | Aug | 10 | 11:08 | .gitignore 13 | drwxr-xr-x | 2 | sophia | sophia | 4096 | Aug | 24 | 03:24 | images 14 | -rw-r--r-- | 1 | sophia | sophia | 1085 | Jun | 17 | 15:32 | LICENSE 15 | -rw-r--r-- | 1 | sophia | sophia | 614 | Jun | 17 | 15:32 | Makefile.toml 16 | -rw-r--r-- | 1 | sophia | sophia | 0 | Aug | 23 | 04:58 | open 17 | -rw-r--r-- | 1 | sophia | sophia | 11375 | Aug | 24 | 03:24 | README.md 18 | -r--r--r-- | 1 | sophia | sophia | 0 | Jul | 4 | 03:51 | readonly.txt 19 | -rw-r--r-- | 1 | sophia | sophia | 37 | Aug | 23 | 04:54 | rustfmt2.toml 20 | -rw-r--r-- | 1 | sophia | sophia | 16 | Aug | 1 | 19:45 | rustfmt.toml 21 | drwxr-xr-x | 10 | sophia | sophia | 4096 | Aug | 24 | 03:24 | src 22 | drwxr-xr-x | 4 | sophia | sophia | 4096 | Aug | 22 | 19:22 | target 23 | drwxr-xr-x | 4 | sophia | sophia | 4096 | Aug | 22 | 04:15 | tests 24 | drwxrwxr-x | 2 | sophia | sophia | 4096 | Jul | 19 | 15:18 | .vscode ----+------------+-------+----------+----------+--------+-------+-----+-------+--------------- </code></pre> Because Nu let's you manipulate your data until it's how you want it, there's a feeling of playing with your data. You get used to using the verbs, and then you can use them on anything. When you're ready, you can write it back to disk. Oh, before I forget - I wanted to quickly show how to get data from Nu back out into the outside world. Here's an example of calling echo</code> on each filename in a directory: > ls | get name | echo $it </code></pre> You can see that we can mix-and-match commands that are inside of Nu with those that are outside, and data will still flow between them as expected. But Nu is more than just a pipeline. More than a pipeline</h1> As we built Nu, we realized we could experiment with other parts of how a shell works. The first of these experiments lead us to an observation: if everything is data in Nu, we should be able to view this data. We've seen the tables. Nu also supports opening and looking at text and binary data. If we open a source file, we can scroll around in a syntax-highlighted file. If we open an xml, we can look at its data. We can even open a binary file and look at what's inside (hint: there's even a fun easter egg if you open certain kinds binary files, especially if you've installed Nu with the optional rawkey</code> feature). Being able to view data is helpful, and this kind of polish extends to other aspects, like error messages: Nu takes heavy inspiration from the error messages in Rust</a>. As much as possible, draw your eyes to the problem. Combined with the pipeline, some pretty interesting errors are possible: You might wonder how does that even work. Nu has a metadata system (still early!) that you can read about in the Metadata chapter</a> of the Nu book</a>. Let's just take a quick peek at it: > open Cargo.toml ------------+--------------+------------------+----------+---------- bin | dependencies | dev-dependencies | lib | package ------------+--------------+------------------+----------+---------- [11 items] | [object] | [object] | [object] | [object] ------------+--------------+------------------+----------+---------- > open Cargo.toml | tags ----------+------------------------------------------ span | origin ----------+------------------------------------------ [object] | /home/sophia/Source/nushell/Cargo.toml ----------+------------------------------------------ </code></pre> Data that flows through the pipeline gets a set of additional metadata tagged to it. We can use this later to figure out how to display the contents, show a better error message, and more. Shells, plural</h1> Let's say you're in a directory, but you'd really like to flip back and forth between it and one or two others. You could open up multiple tabs, multiple terminals, if you're on a Unix system you could use "screen", and probably even more than that. What if the shells were just built in? In Nu, we can enter</code> a directory, which adds it to a ring of shells we can bounce between: > enter ../rhai/ /home/sophia/Source/rhai(master)> shells ---+---+------------+------------------------------- # | | name | path ---+---+------------+------------------------------- 0 | | filesystem | /home/sophia/Source/nushell 1 | X | filesystem | /home/sophia/Source/rhai ---+---+------------+------------------------------- </code></pre> Using n</code> and p</code> we can jump back and forth between the shells. exit</code> gets us out of a shell. You might noticed that name</code> column in the shells</code> table. Why's that there? Oh no... oh yes. > enter Cargo.toml /> shells ---+---+--------------------------------------------+------------------------------- # | | name | path ---+---+--------------------------------------------+------------------------------- 0 | | filesystem | /home/sophia/Source/nushell 1 | | filesystem | /home/sophia/Source/rhai 2 | X | {/home/sophia/Source/nushell/Cargo.toml} | / ---+---+--------------------------------------------+------------------------------- </code></pre> That's right, we're in the file. Can we cd</code>? Oh yes, we can: /> ls ------------+--------------+------------------+----------+---------- bin | dependencies | dev-dependencies | lib | package ------------+--------------+------------------+----------+---------- [11 items] | [object] | [object] | [object] | [object] ------------+--------------+------------------+----------+---------- /> cd bin /bin> ls ----+----------------------+--------------------------- # | name | path ----+----------------------+--------------------------- 0 | nu_plugin_inc | src/plugins/inc.rs 1 | nu_plugin_sum | src/plugins/sum.rs 2 | nu_plugin_add | src/plugins/add.rs 3 | nu_plugin_edit | src/plugins/edit.rs 4 | nu_plugin_str | src/plugins/str.rs 5 | nu_plugin_skip | src/plugins/skip.rs 6 | nu_plugin_sys | src/plugins/sys.rs 7 | nu_plugin_tree | src/plugins/tree.rs 8 | nu_plugin_binaryview | src/plugins/binaryview.rs 9 | nu_plugin_textview | src/plugins/textview.rs 10 | nu | src/main.rs ----+----------------------+--------------------------- </code></pre> Plugins</h1> Nu can't come with everything you might want to do with it, so we're releasing Nu with the ability to extend it with plugins. There's more information in the plugins chapters</a>. Nu will look for these plugins in your path, and load them up on startup. All because of Rust</h1> Nu would not have been possible without Rust. Internally, it uses async/await, async streams, and liberal use of "serde" to manage serializing and deserializing into the common data format and to communicate with plugins. We also heavily leveraged crates.io</a>. The ability to load numerous file formats, display messages, draw tables, and more all came from the hundreds (thousands?) of generous developers who wrote the crates we use in Nu. A huge thank you to everyone who contributed to Nu without ever knowing it. What's next?</h1> Nu is just getting started. In this release, we have a foundation to build on. Next, we'll work towards stability, the abilty to use Nu as your main shell, the ability to write functions and scripts in Nu, and much more. If you want to give it a spin, the installation instructions</a> will help you get started. If you want to chat come by our discord</a> and github</a> Creating crossplatform Rust terminal apps 2019-04-27T00:00:00+00:00 Look Mom, Pikachu running in Windows CMD! I've been wanting to play around with the cool spinning Pikachu demo</a> everyone was talking about. Sadly, it used termion to do its magic, which meant that unfortunately it wouldn't work for me. Termion has been a boon for Rust, with lots of folks using it to create terminal applications. Unfortunately, as a Windows user, I know there's a good chance that if the crate depends on termion that's the end of the line for me, as termion apps just don't work in Windows. Surely, I thought, there must be a better way, but I never managed to find one. Enter crossterm</h1> Yesterday, I discovered a crate I have wanted for at least a year: that holy grail of something like termion that also worked crossplatform. The crate is called crossterm</a>, and it is exactly that. Rather than being an exact copy of termion, I'd call Crossterm a termion-like crate. The API surface feels very similar between the two crates, but it's not identical. That said, often, the translation is 1:1 (or close to it). In this post, I'll detail what I had to do to port to crossterm. You can take a look at my repo</a> and give it a spin. Moving to crossterm</h2> Like termion, crossterm is a single crate to depend on that pulls in what you need for interacting with the terminal: -termion = "1" +crossterm = "0.9.3" </code></pre> Next, is setting up the terminal: - let mut stdout = AlternateScreen::from(stdout().into_raw_mode().unwrap()); // Raw output is clean output - let mut stdin = async_stdin().bytes(); // Async in so input isn't blocking + + let crossterm = Crossterm::new(); + #[allow(unused)] + let screen = RawScreen::into_raw_mode(); + let input = crossterm.input(); + let mut stdin = input.read_async(); + let cursor = cursor(); + + cursor.hide()?; </code></pre> There are a few more steps here, and I took a slightly different approach than termion. In the original, the author uses an AlternateScreen, a way to easily switch back to the previous terminal with the application exits. The crossterm package supports this, though there were is an issue</a> that's being worked on. Using an AlternateScreen is optional, so I just left it off. The next step was to set up raw mode. Raw mode lets us get user input directly without the normal terminal buffering. The order of operations in crossterm above is important. Lastly, I hide the cursor so it doesn't mess up the animation. The event loop is similar, though a little more verbose in my version: - let b = stdin.next(); - if let Some(Ok(b'q')) = b { - break; + if let Some(b) = stdin.next() { + match b { + InputEvent::Keyboard(event) => match event { + KeyEvent::Char('q') => break, + _ => {} + }, + _ => {} + } </code></pre> The event systems between termion and crossterm, while similar, are different enough that I preferred to do two-step match to make sure I understood it. Events for both key and mouse come in through the event stream, which is a nice benefit. Drawing colors to the screen is also very similar between termion and crossterm: - format!("{}{}{}{}", color::Fg(color::Rgb(bg.0, bg.1, bg.2)), - color::Bg(color::Rgb(25,25,25)), - src, - color::Fg(color::Reset)) + print!( + "{}{}{}", + Colored::Fg(Color::Rgb { + r: (pixel.1).0, + g: (pixel.1).1, + b: (pixel.1).2 + }),^M + Colored::Bg(Color::Rgb { + r: 25, + g: 25, + b: 25 + }), + pixel.0 + ) </code></pre> The change from color::Fg</code> to Colored::Fg</code> is straightforward enough I did most of these changes with search/replace. Gotchas</h1> The translation is not without its own set of hiccups. The main one of which you get a hint at here: - pub frame_buffer: Vec<String>, + pub frame_buffer: Vec<(char, (u8, u8, u8))>, </code></pre> Originally, the whole framebuffer was turned into a vector of strings. On most Unix platforms, these would include the ANSI commands to change the color or move the cursor. It's a powerful tool, and being text-based, you can prebuffer the whole frame via string manipulation. Unfortunately, assuming everything is string-based doesn't work in Windows. When crossterm outputs using println!, it's calling into the Windows Console API between each piece of the println. This allows the same code to work, assuming it doesn't buffer the string. To work around this, I stopped buffering the string and instead stored the pixel value in the buffer. When it was time, I drew the whole frame using print! statements. When I first did this, it was noticeably slower than the termion version(roughly 5-10 fps). This is because calling into the Console API that often (once per character) is going to pull down performance. Luckily, I could work around this by just checking if we were already using the color I wanted to render. If we were, I didn't set the color again. Performance between the termion version and the crossterm version now look the same to me, which I call good enough :) Conclusion</h1> We have a way to move more of our terminal apps to being crossplatform, so that they work out-of-the-box on Windows as well as Unix-based systems. It was fun to write the port, and I did the port in a couple hours without ever having used crossterm before. Also, big thanks to Mitchell Hynes</a> for writing the original! The Fallow Year, my Rust2019 post 2018-12-08T00:00:00+00:00 Definition: fallow: (of agricultural land) Ploughed but left unseeded for more than one planting season. Rust is an amazing project. It's unlike anything I've ever seen. Not only are we seeing an ever-growing number of big name users of Rust, we also continue to see leaps in productivity and functionality. Yesterday, the Rust 2018 edition shipped</a>. The culmination of three years of work, it shows off what the community is capable of: new features, backwards compatibility, new ergonomics without sacrificing performance, and the list goes on and on. Which is why it might comes as a little bit of a surprise that I suggest we let the field rest for the year. More specifically, that we should let one field rest while we plant another. Rather than focusing on new designs, I suggest we turn our attention outside of RFCs to other areas of Rust. Examples</h1> The compiler: The compiler team has been understaffed while also being heavily taxed by the effort of keeping up with new designs. I think they've done a stellar job with the resources they have, but they need help -- and lots of it. RFCs have gone unimplemented simply because not enough people are contributing. Compile times need work. The compiler team is generally too busy to implement core functionality that it hoped to get done this year, like compiler-drive IDE completions. Let's clear through the backlog of the growing list of accepted RFCs. Let's take the time to do the deep refactoring that we haven't had time to do because we were stretched too thin. Let's give ourselves space to experiment with new ways of doing things. Let's get the compiler ready to serve both build and IDE scenarios. RLS and related tools: We've had big success with projects like rustfmt, and it would be great to mirror that success in the Rust Language Server, or RLS, which powers Rust's IDE support. Unfortunately, the current RLS has to rely on many hacks to work with what is available from the compiler. By freeing up the compiler team, we also give space to the dev tools team to collaborate and finish the RLS, removing the need for hacks that cause incomplete functionality and unpredictabilty. The community: That's right, the community needs a pause too. Everyone loves new features, but we need a bit of time to let all the new ones sink in before we start designing more. What design patterns will arise in practice? What are the best ways to work with async code at scale? On and on. Getting all this experience will let us see more clearly where to design next. Documentation, videos, and other materials: Along with the growing knowledge the community is getting by more using the current Rust more deeply, we can also translate this knowledge into more documentation. The need is strong for mid-level Rust developer documentation. What are the Rust design patterns? How does one get the most benefit from Rust quickly? How does one translate C++ knowedge to Rust? And so on. Libraries: We can use the knowledge we have to create new and better libraries. Existing libraries could continue to mature and adopt Rust 2018 idioms. Areas like GUI development need some clear direction and focus. Outreach: We've worked on some outreach to companies who might be interested in Rust, but there's more we can do here. New whitepapers, new documentation about how to use Rust in specific sub-fields, like finance, data mining, machine learning, etc. The list</h1> As it turns out, these aren't my ideas. They're yours. This year's Rust survey</a> results highlighted these areas of improvement (sorted by number of comments): the need for better library support</li> a more improved IDE experience</li> the need for broader adoption of Rust generally</li> a richer ecosystem of tools and support</li> an improved learning curve</li> the need for important language features and crates to be stable and supported</li> support for async programming</li> support for GUI development</li> better documentation</li> improved compile times</li> </ul> You'll notice that only one of these, "support for async programming", implies adding new functionality to Rust itself. Rather than asking for new features, users wanted better documentation, tools, library support, IDE support, learning materials, and overall stability. The idea</h1> Let's finish what we started. As much as is possible and practical this year, let's set aside new designs and ship what we've already designed. Let's tackle challenges we haven't had time to give our full attention. In short, let's grow Rust where it needs it most. A Snapshot of Rust's Popularity in July 2018 2018-07-28T00:00:00+00:00 Talking about a language's popularity is traditionally a tricky topic. How do you measure popularity? How do you compare one language to another when they're focused on different styles and different audiences? So, rather than having one or two charts, I'm going to look at a number of "slices" into Rust's growth to see it front different angles. There are also many ways we could measure popularity. For example, looking at the growth of StackOverflow, internal forums, Twitter traffic, Google Trends, or the various language popularity reports. Each of these measures has its own strengths and trade-offs. For our purposes in this post, we're going to focus on a set of data points that should help show growth of usage rather than growth of buzz (that is, the amount people talking are about Rust). If people are using a language to do real work, that's the best indicator of a language's success. The first of place we'll look for Rust usage is in open source, and with that, let's look at GitHub. Growth on GitHub</h1> GitHub offers a few ways to measure how well a particular language is growing. GitHub itself uses the number of pull requests in a given language, which it uses in the GitHub Octoverse</a>. If we were to run this query for 2018 so far, the top fifteen languages (by number of PRs) would be: JavaScript: 1736476</li> Python: 804790</li> Java: 703649</li> Ruby: 560430</li> PHP: 359040</li> C++: 319324</li> TypeScript: 311229</li> Go: 258131</li> C#: 246513</li> CSS: 236795</li> Shell: 168301</li> C: 160889</li> Swift: 67664</li> Scala: 67188</li> Rust: 52936</li> </ol> Rust comes in at 15th place as part of the group of Swift/Scala/Rust. This is pretty good company to be in. All of the languages on this list are known for doing heavy lifting in real applications, rather than being hobby or boutique languages. Being 15th is also significant because, if the Octoverse were held today, Rust would make their top languages list for the first time. Moving on, one gotcha of focusing on PRs is that it lets a relatively small number of highly successful projects skew the results. The popularity of the projects drives high numbers of PRs, and as a result the language looks far better. To compensate for this, let's also look at "active projects". For our purposes, let's define "active projects" as the number of projects that a) are larger than a "hello world", b) have at least 1 star, and c) have been worked on in the last 30 days: JavaScript: 56437</li> Python: 37554</li> Java: 27912</li> C++: 16787</li> PHP: 16451</li> Shell: 12245</li> C#: 11503</li> C: 10662</li> Go: 8816</li> CSS: 8781</li> TypeScript: 8447</li> Ruby: 7518</li> Swift: 3828</li> Objective-C: 3054</li> Rust: 2604</li> Kotlin: 2343</li> Scala: 2072</li> </ol> Another respectable showing for Rust. Again, we see it as part of the languages for doing serious work. The GitHub numbers tell one story, but do we see the same story if we look in other places? With this kind of popularity, we'd expect to see visible use of Rust in industry. Rust's Commercial Users</h1> Rust has seen a surprising amount of growth in commercial usage over the past year alone. Here's a partial list of names people may know: Amazon - Building tools in Rust</a>. Atlassian (makers of Jira) - Using Rust in the backend</a>. Dropbox - Using Rust in both the frontend and backend</a>. Facebook - Tools for source control</a>. Google - As part of the Fuchsia project</a>. Microsoft - Using Rust in part of their new Azure IoT work</a>. npm - Using Rust in some of the npm core services</a>. Red Hat - Creating a new storage system</a> Reddit - Using Rust in its comment processing</a> Twitter - As part of the build team support for Twitter</a>. You can see even more on the Friends of Rust</a> that include other familiar names like Baidu, Wire, Mozilla, Samsung, Cloudflare, Chef, Canonical, and more. Across the industry, software companies are beginning to adopt Rust at some level in their organization. As expected, these may be tentative at first, focusing on Rust for build tools, for backend processing, or targeting a specific platform like IoT. Early adopters like Dropbox and Canonical have now grown beyond initial investments to using Rust in an increasing amount of their codebase. All told, it's still early days for Rust. Large companies take time to try new technology before they adopt it more broadly. We're seeing Rust in that part of the adoption curve, and it's doing well here. Languages -- like Go and TypeScript -- that saw explosive growth in recent years also saw similar patterns at this part of their lifecycle. Because of Rust's closeness with C++, we should also look at industries that are traditionally strongly populated by C++ developers. One such industry is game development. Zooming into Games</h1> Despite the game industry's use of established development technologies, Rust is beginning to make in-roads with game developers. Here's a list of usage of Rust in some familiar names among game developers: Chucklefish Games - Using Rust in multiple upcoming games</a>. Electronic Arts - Using Rust in SEED</a>. Frostbite Engine - Using Rust in backend processing</a>. Ready at Dawn Studios - All new development</a> will be in Rust. Unity - Using Rust in data engineering</a>. We're also seeing other indie developers</a> beginning to use it for their titles. Just as we saw in the "big name" usage, we see a similar pattern with game developers. Initial usage is targeted, perhaps at backend or data-focused projects. After this, there's a path to developing core technology, and even whole games, in Rust. Jobs</h1> When we ran the 2017 Rust survey</a>, one of the big themes that came up for people not using Rust was around their companies not using it. The jobs just weren't there. In 2018, with a growing numbers of companies and studios using Rust, has this situation improved? In short, yes, though there's still a lot of room to grow. A quick look at LinkedIn's job list shows a number of Rust opportunities from the likes of Amazon as well as a number of companies involved in blockchain, security, science/healthcare, and more. Still, when compared to the volume of job offerings for languages like Go or TypeScript, there's still a gap in terms of availability. Let's move on from Rust's use in companies to look at the growth of the community that supports Rust. Community and Governance</h1> In times past, a language could survive on being used solely for commercial development. These days, however, a language's community forms its foundation. It offers a way to ramp up new users, support new projects, and set the tone for the project going forward. Rust meetups now total over 130 around the world The in-person piece of the Rust community, the Rust meetups, have seen 30% growth in the last nine months and now total over 130 around the world. We've likewise seen steady growth in the number of conferences that users can attend for more training and networking. We've also seen some impressive growth in indicators of the online community as well since the last time I reported on them</a>. For example, the Rust reddit recently passed 40k users (up from the 29k of nine months ago), the GitHub repo is nearly at 30k stars (up from 24k), and the official Rust twitter has over 31k followers (up from 24k). This enthusiasm has also translated into the teams that govern Rust itself. Combined, the governing teams have doubled in size in the last 9 months to over 100 positions across the various teams</a>. The scope has also grown in tandem, with new teams that cover infrastructure, developer experiences, and dependency management. Conclusion</h1> These are just some data points about Rust's use in industry and in the software development community. It's difficult to get a complete picture, and even more difficult to predict what will come next. That said, it does seem to shine a promising ray of light on where Rust is today. Where will Rust go tomorrow? That is anyone's guess. I, for one, will certainly be watching and cheering it along. Starting the next thing 2018-07-24T00:00:00+00:00 It's that time again! July 31st will be my last day at Mozilla. Even though I was only at Mozilla for just under two and a half years, I had the good fortune to work on some pretty awesome projects. I joined the Rust team and worked with Niko Matsakis and Aaron Turon on where Rust could grow, building on my experience coming from TypeScript. We learned a ton from each other, and I'm really grateful for all those conversations and the close friendship that resulted. I also got to roll up my sleeves and write some code. The first project was to redesign the Rust error messages</a> with Niko. The next was to create the Rust Language Server</a> with Nick Cameron. Both have grown a life of their own, and it's been awesome to see! I was also lucky to help Jack Moffitt out with the Servo project. Under his guidance, it produced loads of good code and good ideas. These made their way into Firefox</a> and into the Firefox Reality</a> work. Learning from Jack and connecting with the Firefox org (and some of the awesome Firefox managers) was a treat. Finally, my last seven months have been helping create and run the Developer Technology team at Mozilla, a melding of the Rust and WebAssembly efforts under one umbrella. This allowed even closer collaboration between the two projects and has already resulted in lots of cool new technology, tools, and coding practices</a>. I'm excited to watch what they'll be able to come up with next. And speaking of "next", what's next for me? I'm moving on to create a new company. I don't have a ton of details to share, yet, but I hope to do so in the coming months as the ideas and technology start to come together. "Will it use Rust?" I can already hear many of you asking. Most assuredly. If there's a better solution for efficient, write-once-and-don't-worry-about-it software development, I've yet to find it. Thank you, Mozilla! Both for the experience and for the friends I've made along the way. PS: I plan to continue podcasting - it's fun :) How fast can we compile Rust hello world? 2018-05-03T00:00:00+00:00 Seeing Nick Nethercote's blog post about speeding up the compiler</a>, I started wondering just how fast could a Rust compiler be? How fast could we compile a simple example? How fast can we compile a Rust hello world? Starting out</h2> When you do a cargo new hello_rust --bin</code>, you get a simple Rust hello world: fn main() { println!("Hello, world!"); } </code></pre> Awesome. Let's see how long this takes to build: > time cargo build Compiling hello_rust v0.1.0 (file:///home/sophia/Source/hello_rust) Finished dev [unoptimized + debuginfo] target(s) in 0.52 secs real 0m0.548s user 0m0.468s sys 0m0.080s </code></pre> Half a second on my XPS 13 9360, with a i7-7500U @ 2.70GHz with 16 gigs of RAM. That seems, a bit long for a hello world. Maybe cargo is doing extra work? > time rustc src/main.rs real 0m0.378s user 0m0.289s sys 0m0.083s </code></pre> That's definitely better. We're just over 1/3rd of a second. What should we expect? If only there were another language we could compile a hello world for that's comparable... > cat hello_c.c #include <stdio.h> int main() { puts("Hello, from C!"); return 0; } > time clang hello_c.c real 0m0.082s user 0m0.021s sys 0m0.041s </code></pre> Woooaah! Wait a second. You're telling me that compiling C hello world is over 4.5x faster. That's... that's something. Maybe the compiler is doing a bit too much work for this small example? I mean, we don't need much from the standard library, just the println macro. You know, a reasonable person would probably stop here and walk away. Chalk it up to "Rust is more complicated" or "maybe if I knew the right commandline options" No, that's not us. Not today. Today, we're going to write our own Rust translator. We're going to see just how fast we can compile Rust to a working hello world. First stop: let's build a Rust->C translator.</h2> Here's my thinking: if we write our own translator, we can translate the Rust to C ourselves, and then compile the C. That way, we should end up with a compile time close to the C. > cargo new rabbithole --bin </code></pre> Writing a full compiler is complex work, but we're going for speed here. And speed means cutting corners. To that end, we're implementing just enough of a compiler to translate our Rust to C, then we're calling that good enough. First step, we need a parser. Luckily, there already is a crate that can parse Rust code for us. [dependencies] syn = {version = "0.13", features = ["full", "extra-traits"] } </code></pre> extern crate syn; use std::env; use std::fs::File; use std::io::Read; fn main() { let mut args = env::args(); args.next(); match args.next() { Some(fname) => { let mut src = String::new(); let mut file = File::open(&fname).expect("Unable to open file"); file.read_to_string(&mut src).expect("Unable to read file"); let syntax = syn::parse_file(&src).expect("Unable to parse file"); println!("{:#?}", syntax); } _ => { println!("Please supply the file to compile"); } } } </code></pre> > cargo run -- hello.rs File { shebang: None, attrs: [], items: [ Fn( ItemFn { attrs: [], vis: Inherited, constness: None, unsafety: None, abi: None, ident: Ident { term: Term { sym: main } }, decl: FnDecl { fn_token: Fn, generics: Generics { lt_token: None, params: [], gt_token: None, where_clause: None }, paren_token: Paren, inputs: [], variadic: None, output: Default }, block: Block { brace_token: Brace, stmts: [ Item( Macro( ItemMacro { attrs: [], ident: None, mac: Macro { path: Path { leading_colon: None, segments: [ PathSegment { ident: Ident { term: Term { sym: println } }, arguments: None } ] }, bang_token: Bang, delimiter: Paren( Paren ), tts: TokenStream [ Literal { lit: "Hello, world!" } ] }, semi_token: Some( Semi ) } ) ) ] } } ) ] } </code></pre> This might look a bit intimidating, so let's break it down. The top-most thing is a File. File</li> </ul> Each File continues zero or more Items. An Item are things like declarations. Function declarations, for example, are one kind of Item. File Item</li> </ul> </li> </ul> Since this file has one function, it has one item, an ItemFn. This holds all the information about our main function. You can see in here that we have the function's declaration (sometimes called the function prototype, if you come from C). This tells us how many parameters it takes, what its return type is, etc. Luckily for us, this is all empty because main has nothing interesting here. This lets us focus on the body of the function, here called 'block' File Item (ItemFn) Block</li> </ul> </li> </ul> </li> </ul> Blocks contain statements and expressions. This is really the meat of what actually does things, and which is where we'll actually spend our time writing the compiler. Let's take a closer look: block: Block { brace_token: Brace, stmts: [ Item( Macro( ItemMacro { attrs: [], ident: None, mac: Macro { path: Path { leading_colon: None, segments: [ PathSegment { ident: Ident { term: Term { sym: println } }, arguments: None } ] }, bang_token: Bang, delimiter: Paren( Paren ), tts: TokenStream [ Literal { lit: "Hello, world!" } ] }, semi_token: Some( Semi ) } ) ) ] } </code></pre> On closer inspection we see this block has only one statement. A something-something macro something or other. Oh right, println is a macro! Let's focus on just the macro invocation itself: mac: Macro { path: Path { leading_colon: None, segments: [ PathSegment { ident: Ident { term: Term { sym: println } }, arguments: None } ] }, bang_token: Bang, delimiter: Paren( Paren ), tts: TokenStream [ Literal { lit: "Hello, world!" } ] }, </code></pre> Now this is a little more like it. It's still a little cumbersome, but we can see the parts we need. There's a sym</code> thing that says what the macro is. Later on, we can see the string that we're printing is a literal string. We can ignore the rest. That's it. Make sure it's invoking println, and then we'll look for the string that's being printed. To make things a little easier on ourselves, let's make a quick-and-dirty intermediate representation. This will only have one command so far, but it'll be an easy way to de-couple the function that does the parsing from the one that does the code generation. enum Command { PrintLn(String) } </code></pre> With that, we're ready to make our parser that can take in the syn::File</code> and output our Command. There are a few ways we can do this, but to keep things pretty simple, I'm going to make a separate parse</code> function for each layer we talked about earlier: File, Item, and Block. You can think of this as us unpacking the useful information from what syn</code> gives us into much simpler data structures. extern crate syn; use std::env; use std::fs; use std::io::Read; #[derive(Debug)] struct File { functions: Vec<Function>, } #[derive(Debug)] struct Function { body: Vec<Command>, } #[derive(Debug)] enum Command { PrintLn(String), } fn parse_body(body: &syn::Block) -> Result<Vec<Command>, String> { let mut stmts = vec![]; for stmt in &body.stmts { match stmt { syn::Stmt::Item(syn::Item::Macro(ref im)) => { let macro_name = im.mac.path.segments[0].ident.as_ref(); if macro_name == "println" { match im.mac.tts.clone().into_iter().next() { Some(ref arg) => { stmts.push(Command::PrintLn(arg.to_string())); } None => return Err("Expected argument in function".into()), } } else { return Err(format!("Unknown macro: {}", macro_name)); } } _ => return Err("Unexpected statement in function".into()), } } Ok(stmts) } fn parse_fn(item: &syn::ItemFn) -> Result<Function, String> { Ok(Function { body: parse_body(&item.block)?, }) } fn parse_file(file: &syn::File) -> Result<File, String> { let mut functions = vec![]; for item in &file.items { match item { syn::Item::Fn(ref item_fn) => { functions.push(parse_fn(item_fn)?); } _ => return Err("Unexpected item in file".into()), } } Ok(File { functions }) } fn main() { let mut args = env::args(); args.next(); match args.next() { Some(fname) => { let mut src = String::new(); let mut file = fs::File::open(&fname).expect("Unable to open file"); file.read_to_string(&mut src).expect("Unable to read file"); let syntax = syn::parse_file(&src).expect("Unable to parse file"); let result = parse_file(&syntax); println!("{:#?}", result); } _ => { println!("Please supply the file to compile"); } } } </code></pre> Which results in: Ok( File { functions: [ Function { body: [ PrintLn( "\"Hello, world!\"" ) ] } ] } ) </code></pre> That's much nicer! It took a little work reading the syn</code> documentation, but once we got the hang of it, we can now work with real Rust source code and turn it into something we can use. Oh, before we go, let's clean one thing up. You might have noticed that extra set of quotes in the macro argument. No problem, we'll just turn: stmts.push(Command::PrintLn(arg.to_string())); </code></pre> Into: stmts.push(Command::PrintLn(arg.to_string().replace("\"", ""))); </code></pre> Which gives us: Ok( File { functions: [ Function { body: [ PrintLn( "Hello, world!" ) ] } ] } ) </code></pre> Beautiful. Now that we have it in this form, we have a much simpler form that we can hand to the other half of the translator. In this half, we'll be outputting the C code and compiling it. Since we only care about one function in one file, let's just make a quick-and-dirty function to generate the C: fn compile_to_c(file: &File) -> String { let mut c_output = String::new(); c_output += "#include <stdio.h>\n"; c_output += "int main() {\n"; for stmt in &file.functions[0].body { match stmt { Command::PrintLn(ref msg) => { c_output += &format!("puts(\"{}\");\n", msg); } } } c_output += "}\n"; c_output } </code></pre> That's it. Let's call it. println!("{}", compile_to_c(&result.unwrap())); </code></pre> Running it again, we get: #include <stdio.h> int main() { puts("Hello, world!"); } </code></pre> Woah! That looks like C to me. Dare we try to compile it? > cargo run -- ../hello_rust/src/main.rs > testme.c > clang testme.c > ./a.out Hello, world! </code></pre> It... works! We just made a compiler! > time ./target/release/rabbithole ../hello_rust/src/main.rs > testme.c real 0m0.014s user 0m0.010s sys 0m0.004s > time clang testme.c -o testme real 0m0.100s user 0m0.043s sys 0m0.041s </code></pre> Nice, by cutting lots of corners and making our own translator, we got the compile times down from 0.378s to 0.114s, less than 1/3rd of where we were. Should we stop there? Surely we can go further down the rabbit hole. Oh right, C is not the lowest we can go Deeper down the rabbit hole</h2> Let's cook up some assembly. First, let's test how fast we can assemble a hello world in assembly. > cat hello_asm.s .text .global main main: mov $hw_str, %rdi call puts mov $0, %rax ret .data hw_str: .asciz "hello asm" > time as hello_asm.s -o hello_asm.o real 0m0.019s user 0m0.008s sys 0m0.011s > time ld -o hello -dynamic-linker /lib64/ld-linux-x86-64.so.2 /usr/lib/x86_64-linux-gnu/crt1.o /usr/lib/x86_64-linux-gnu/crti.o -lc hello_asm.o /usr/lib/x86_64-linux-gnu/crtn.o real 0m0.044s user 0m0.036s sys 0m0.008s </code></pre> Wow, the assembly output cuts the compile almost in half! If you haven't seen x64 assembly before, that might look a little strange. Instead of working with C code with functions, we've dropped down much closer to the machine itself. The above represents the lowest we can go in Linux. Let's unpack what's happening: .text .global main </code></pre> Our assembly file starts with two things. One, a section that we'll be talking about the code itself (.text) and then the one symbol we'll be making visible (main). This will be the main function that the linker will use later. Next, main begins: main: </code></pre> What follows are series of assembly commands. The hardware keeps numbers and addresses in registers, similar to how you might store them in variables. mov $hw_str, %rdi call puts </code></pre> With that in mind, this doesn't seem quite so bad. The first line is going to set up our register that we'll use as the argument to the puts function (here, in Linux, %rdi). After the argument is ready, we call puts. mov $0, %rax ret </code></pre> After that, we return 0, just like we did in C. Here, in 64-bit assembly, we're returning the return value in %rax. That's it. Oh wait, one more thing. So this was the code section, we still haven't told it what the string is to print (which we called $hw_str earlier). .data hw_str: .asciz "hello asm" </code></pre> That's not bad. We have a data section that contains our strings. We have one string, called hw_str</code>, and we give it a null-terminated string to print. You may have been wondering about our linker line, too: > time ld -o hello -dynamic-linker /lib64/ld-linux-x86-64.so.2 /usr/lib/x86_64-linux-gnu/crt1.o /usr/lib/x86_64-linux-gnu/crti.o -lc hello_asm.o /usr/lib/x86_64-linux-gnu/crtn.o </code></pre> This bit of magic is actually the kind of tasks that your C compiler does for you behind the scenes. In Linux, for a program to run, it needs to be able to access things like the Linux program loader, the C standard library, and the Linux subsystems. The above takes the barebones object file we made with the assembler and gives it all it needs to be a complete Linux executable. Before we write our assembly-outputting function, let's take a quick pause and talk about Windows. We've been giving a lot of examples for Linux, but you can do the same kinds of things in Windows. Make sure you have a 64-bit nasm</a> installed. C:\Users\Sophia\source\fast_hello>type nasm_hello.asm global main extern puts section .text main: push rbp mov rbp, rsp mov rcx, message call puts mov rax, 0 pop rbp ret message: db 'Hello, Rust!', 0 C:\Users\Sophia\source\fast_hello>nasm -f win64 nasm_hello.asm -g -o nasm_hello.obj C:\Users\Sophia\source\fast_hello>link -out:b.exe -defaultlib:libcmt "-libpath:C:\\Program Files (x86)\\Microsoft Visual Studio 14.0\\VC\\lib\\amd64" "-libpath:C:\\Program Files (x86)\\Windows Kits\\10\\Lib\\10.0.15063.0\\ucrt\\x64" "-libpath:C:\\Program Files (x86)\\Windows Kits\\10\\Lib\\10.0.15063.0\\um\\x64" -nologo nasm_hello.obj </code></pre> You can see that the assembly looks different, though it's similar. We do a little more setup, but the core of the work is the same. We set up where the string is, we send it to the puts</code> function, and we return 0 via the rax</code> register. Likewise, after we have an object file from the assembler, we have to do a similar dance with the linker in order to have a fully working Windows executable. With that, we can see all the techniques we're using in Linux can work in Windows, too. Okay, let's finish this up and see where we are. Just like we wrote a C output function, we're going to write an assembly output function. This function will create two strings: one for the code and one for the data, and then stitch them together. fn compile_to_asm(file: &File) -> String { let mut asm_code_output = String::new(); let mut asm_data_output = String::new(); asm_data_output += ".data\n"; asm_code_output += ".text\n"; asm_code_output += ".global main\n"; asm_code_output += "main:\n"; let mut temp_num = 0; for stmt in &file.functions[0].body { match stmt { Command::PrintLn(ref msg) => { asm_code_output += &format!("mov $str_{}, %edi\n", temp_num); asm_code_output += "call puts\n"; asm_data_output += &format!("str_{}:\n", temp_num); asm_data_output += &format!(" .asciz \"{}\"", msg); temp_num += 1; } } } asm_code_output + &asm_data_output } </code></pre> > time ./target/release/rabbithole ../hello_rust/src/main.rs > hello_asm.s real 0m0.015s user 0m0.010s sys 0m0.005s > time as hello_asm.s -o hello_asm.o real 0m0.022s user 0m0.009s sys 0m0.012s > time ld -o hello -dynamic-linker /lib64/ld-linux-x86-64.so.2 /usr/lib/x86_64-linux-gnu/crt1.o /usr/lib/x86_64-linux-gnu/crti.o -lc hello_asm.o /usr/lib/x86_64-linux-gnu/crtn.o real 0m0.043s user 0m0.029s sys 0m0.011s </code></pre> From 0.114s to 0.08s. We dropped another 30% off the compile time! To recap, we started at 0.548s with Cargo, then 0.378s with rustc, 0.114 going via C, and 0.08 going via assembly. Roughly a 6.8x speedup. Can we squeeze anything else out? You might be wondering about keeping more things in memory and getting a bit more caching. It's a good hunch, let's try it! > time (./target/release/rabbithole ../hello_rust/src/main.rs > hello_asm.s && as hello_asm.s -o hello_asm.o && ld -o hello -dynamic-linker /lib64/ld-linux-x86-64.so.2 /usr/lib/x86_64-linux-gnu/crt1.o /usr/lib/x86_64-linux-gnu/crti.o -lc hello_asm.o /usr/lib/x86_64-linux-gnu/crtn.o) real 0m0.048s user 0m0.028s sys 0m0.021s </code></pre> That's another 40% off the compile time. We've now gone from 0.548s to 0.048s, a whole 11x faster. There's even more we can do. For example, we can squeeze out more by writing our own parser, outputting the .o ourselves, and more. But we'll leave that as an exercise for the reader. Conclusion</h2> I hope you had as much fun reading this post as I had in writing it. There are amazing benefits to all the features that cargo and rustc provide, and the Rust ecosystem builds on those benefits. At the same time, it's fun to play around with compiler technology and see that when we get down to it, it's just another Rust program that we can tune to our heart's content. You can see the rabbithole source</a> if you want to play with it yourself. Talking about how we talk about Rust in 2018 2018-01-10T00:00:00+00:00 This is an entry in the #rust2018 blogging efforts to talk about Rust in 2018 When you come to Rust, you're bound to hear a lot of phrases. "Wrestling with the compiler". "Rust evangelism task force". "Non-lexical lifetimes". "Systems programming". Some seem strange. Others are a bit scary. Some are just a common set you quickly need to learn to before you can even read the Rust news. In this post, I want to talk about how we talk about Rust in 2018. I think there are some ways we can change what we're saying and how we're saying it. These changes will help Rust be more approachable for beginners and an even nicer place in general. "Wrestling with the compiler"</h1> I think we've all been there. Especially when we're starting with Rust, or trying to do something new with it we haven't done before, we feel like the compiler is fighting us every step of the way. Sure, it's frustrating. On a bad day, we might even say "forget it". I wonder, though, is there a better way to talk about it? If you're a Rust user, do you trust the compiler? Are the error messages helping to clarify your thinking? Is it, as some people like to say, a pair programming partner for you? I suggest in 2018, we kick the idea of wrestling with the Rust compiler to the curb and focus on how it helps us rather than the idea of it beating us down. Niko mentioned error messages</a> here to help with that, and I think we should push hard there. While we can change how we talk about the compiler, we can also change how the compiler talks to us as well. By the end of 2018, no more wrestling. It's a tool to help us, and we should work hard both in how we talk and how it works to ensure this is true. "Rust evangelism task force"</h1> We've seen the jokes about Rust having its own task force that swoops in and says things like "you should rewrite it in Rust". Sometimes the swoop happens just as a new security exploit is released. Sometimes it comes out of nowhere. Or maybe you've seen this: someone on a thread, maybe on reddit, posts that maybe Rust isn't doing so well in a benchmark or in a particular scenario. Minutes later there are half a dozen comments dissecting what happened. It's intimidating. And really, it's not helpful to Rust. As a community, it's great that we have so much enthusiasm. I just want to be sure we're channeling that enthusiasm in a way that keeps Rust growing. There are ways we can be courteous by not piling on or jumping in with uninvited suggestions. That might seem like little things, but at scale I think they'll make a big difference. Some of you may be familiar with burntsushi's ripgrep tool. The author of a similar tool, ack, had a great quote</a> about the other day that applies to more than just search tools: "If someone uses ripgrep instead of ack, it doesn’t hurt me. It’s the difference between an abundance vs. scarcity view of the world. I choose abundance." </blockquote> We can adopt the same way of thinking. If someone wants to use Go or C instead of Rust, that's great! Perhaps that's a better fit for what they're working on. Maybe they'll try Rust again in the future. We should make it easy to come and go. "Systems programming"</h1> There are a lot of terms that we use when talking about Rust that I think are simply confusing. Have you ever had this conversation? Person 1: Have you heard of the Rust programming language? Person 2: I'm not sure, what is it? Person 1: It's a systems programming language. Person 2: What's a systems programming language? Rather than answering the question, we just made it seem even more complicated. Confusingly, even among people who have used Rust for a while there isn't even total consensus on what is meant by a systems programming language. Doubly confusingly, Rust isn't even just a systems programming language anymore. With the ability to create web applications and tie into other languages, it starts to feel more than just systems-level development. You can use Rust to build pretty much anything from a low-level operating system component to a JSX-like web app. In the future, we'll be in space</a>. Systems is both too confusing and too limiting for what Rust will be used to do. Maybe there's a better term that we can use, or a small phrase that captures the right feel? "Non-lexical lifetimes" and more</h1> Lots of the Rust community is pretty facile when it comes to computer science. Some of us have dabbled in Haskell. Others are old-hands at C++. We've seen programming from different angles and have a lot of vocabulary to use to describe Rust. This started out as pretty effective short-hand. "This is like typeclasses in Haskell" we might say. "Lifetimes are lexically-scoped". "You need to use a lifetime variable, they're just like generics..." Etc. We can use it to communicate, but the end result can be a bit of a mouthful. I'm not going to suggest we remove these terms from our vocabulary entirely, but I want us to think about how we can improve what we write publically. People we don't know will come by IRC, read our blog posts, and see our forum threads. They'll be coming from a variety of backgrounds, many of which will be different from our own. If where we're talking is public with a wide audience, let's keep that in mind as we communicate. Keep the door open for more people to join. Rust, at its heart, isn't intimidating. It's a technology that a wide range of programmers can use to target a wide range of applications. We don't need to make it sound or feel more complicated, and those extra few minutes to reword a sentence here or there can help. Improving how we improve Rust in 2018 2018-01-07T00:00:00+00:00 Here's my #rust2018 post to add to the batch. The past few years, I've been lucky to be part of the Rust community survey process. In 2016, this data helped show the need to improve usability. We had good foundations, but they were too hard to get at. In 2017, we saw a renewed need to reach out to companies, to improve documentation, and a continued importance of pushing to making Rust more usable. These surveys were a key part of seeing big issues, but it got me thinking. Are there better ways to get and use data? Can we see Rust more clearly -- for what we do well and where we need to improve -- in the future? When you do performance tuning, you benchmark and run perf profilers to get at the root of the problem. What if we did the equivalent for Rust as a whole? We already measure through the surveys. The problem is, we can only really measure once or twice a year. This results in waiting long periods of time to see if we successfully addressed an issue. The delay may mean we don't catch new issues in time or we may fix a problem only to find out we didn't address the underlying cause. Getting better data, sooner</h1> What if we had a way to have better data sooner? There are a few ways we could do this. Smaller surveys</h2> The first thing we could do is put out smaller surveys and use them as a quick check-in. They might target specific features in a "did this change solve your issue?" way. We do a bit of this in the RFC process already. People who know about a particular RFC can jump in and comment on whether or not it the approach works for them. While we want to continue using RFCs for this feedback, we could extend what we do here. For example, a blog post could talk about a group of RFCs working together to address one core problem. We could have a quick survey as part of the post to gather feedback on whether the combination of changes does what we hoped. This is still pretty manual, though, and is focused developers actively reading RFCs and blogs and volunteering their time, which may be limited. Are there ways we can get more people involved across a wider range of changes? Usability studies</h2> There's another avenue we could explore: usability studies. Work on software development in the past has shown there's naturally a distance between what people think they spend the most time on and what they actually do spend the most time on. Self-reporting often misses these insights. When you actually watch someone working, issues with usability become more obvious. With the rise of coding streams, we already have some usability data already. Perhaps there is a way to leverage the same tools so that we can perform studies remotely. The end result of these could more directly answer how workflows improve, and give us a clearer insight into both the Rust learning process and how more experienced Rust users use Rust in their work. Opt-in statistics</h2> What if Rust developers could opt-in to some statistics gathered from the tools themselves? This would let Rust developers help the Rust project as part of their everyday work. So it's clear, let me state I'm a strong privacy advocate. I have no desire to read people's private source code, to try to "guess" what they're doing, or anything remotely like that. Instead, here are some ideas we could be looking at that are helpful while not being invasive: What error messages are people hitting? How often?</li> Roughly how much time is spent working in Rust?</li> How long are build times compared to the lines of code? How much is incremental compilation helping?</li> Roughly how much are particular language features being used?</li> </ul> I believe there's a way to we can gather data like this in a way that is both opt-in and preserves the privacy of the developer. Transparency</h2> An important aspect of "better data, sooner" is transparency. If we're careful, we can and should make the data we gather public. It gives contributors a window into the Rust world in more depth than we've previously had. Imagine a live "board" where you could go and see Rust being used in real time, seeing what errors are occuring the most or if the latest fixes to nightly improved build times in the wild. This data could help not only people putting code in the compiler, but also folks writing RFCs, documentation, and more. Measure twice, cut once</h1> When I was growing up, my dad would do various woodworking projects around the house. When I'd help, he would remind me to "measure twice, cut once". You didn't want to waste the effort it took to cut the wood if you couldn't use the result. We can take a similar attitude for Rust in 2018. There are a lot of things we could do. Lots of initiatives. Lots of projects. Can we come up with ways to measure twice before we invest the time? I think we can, and Rust will be better for it. Here's to a happy, successful Rust in 2018! Fun facts about Rust's growing popularity 2017-10-30T00:00:00+00:00 There are now 100 friends of Rust</a> (Be sure to hover to learn about how each company uses Rust) </li> There are now 3 Rust podcasts: New Rustacean</a>, Rusty Spike</a>, and Request for Explanation</a> </li> First year had one conference, the second year had three, and this year had four</a>. </li> crates.io is not even 3 years old, but already the number of crates has passed CRAN(R) and Hackage(Haskell)</a>. </li> The Rust reddit is now over 29,000 Rustaceans strong</a>. </li> the #rust irc channel has grown to over 1300 people </li> The Rust gitter channel is over 1800 people</a>. </li> our first Rust survey had 3,086 responses. Our second had 5,368 responses, over 2,000 more than the first year. </li> Rust is a top 10 most discussed</a> project on GitHub. </li> Rust is a top 10 most reviewed</a> project on GitHub. </li> Rust is used on hundreds of millions of desktops</a> via Dropbox. </li> The "impl period" now has at least 37 different teams</a> helping implement new designs. </li> The popular @code, ranked 5th in this years Stack Overflow survey, powers its code search with ripgrep</a>, a search tool written in Rust. </li> Atlassian uses Rust in a service for analyzing petabytes of source code. </li> The new O'Reilly Rust book was the most popular on Safari</a>. </li> There are over 100 meetups</a> around the world discussing Rust. </li> npm rewrote performance-critical bottlenecks in the registry service architecture with Rust. </li> Popular indie game developer Chucklefish recently announced their new game will be written in Rust</a>. </li> The number of full-time Rust developers continues to grow. Over half are working on large Rust codebases</a>. </li> Rust was the most loved technology two years in a row</a> in the Stack Overflow survey. </li> @rustlang has over 24k twitter followers</a> 🎉🎉 </li> 367 applicants applied for the 12 spots in the Rust Reach initiative</a> </li> There are at least 5 Rust books in production, with releases over the next 18 monts. </li> Red Hat added support for Rust in their recent updates</a>. </li> IntelliJ officially supports Rust</a>. </li> Rust's reddit channel hit 100,000 uniques and 1,000,000 views in July </li> Amazon is hiring for developers with Rust experience</a>. </li> Google engineers are experimenting with Rust for low-level system modules</a>. </li> The official Rust YouTube channel</a> has over 90 videos on Rust. </li> Android Rust development grew 10x in 2017 (to 9.9% of devs)</a>. </li> iOS Rust development grew 10x in 2017 (to 5.3% of devs)</a>. </li> Rust has seen slow, steady growth in commercial usage. Now over 20% of Rust devs use it at work</a>. </li> More than 75% of respondents</a> said they feel comfortable in the Rust community, which includes even non-users of Rust. </li> Rust attracts about 18%</a>, or roughly 1/5th, of its userbase in new users every month. </li> The number of Rust devs with more than a year of experience grew to 42% in the last year</a>. That's a lot of experience! </li> More than 2% of all Rust devs work on Rust codebases totalling more than 100k lines of code</a>. </li> A growing number of Rust devs use Windows, which now makes up over 30% of all Rust users</a>. </li> This year we saw a huge increase, at 16% of responses</a>, in the amount and style of documentation requested to cover more learning styles </li> This year we also saw a shift in non-user reasons to be companies don't use it</a>. More folks now would learn it if it can pay the bills. </li> Speaking of module counts, at 16/day Rust is the 7th fastest growing package repo</a>. </li> The Rust internal forum</a> is a hotbed of activity, with popular posts regularly getting over 10k views. </li> A growing number of GNOME developers are using Rust, with ports underway for some important libraries</a> </li> Rust and GNOME teams have had one hackfest with another next month </li> Though early days, we're seeing more Rust support in the gstreamer media playback framework</a>. </li> Canonical uses Rust in everything from server monitoring to middleware. </li> Lots of System 76 tools and systems code</a> are written in Rust. </li> Principal Intel engineers are encouraging people to learn Rust</a>. </li> Facebook wrote a mercurial server in Rust</a>. </li> Facebook is also a sponsor for Rust Belt Rust</a>. </li> Over 2000 people</a> have contributed to Rust. </li> The github repo has also gotten over 24k stars, making it one of the most popular programming language repos. </li> Not to be missed, even the Rust book repo had over 180 contributors</a>. </li> Rust first shipped in Firefox 48</a>, almost a year and a half ago. </li> Rust is now used in five different modules in Firefox, with the next release having a new style system written in Rust. </li> Rust Bridge</a> continues to be a good outreach to underrepresented developers, now having its 7th(?) event at this week's Rust Belt Rust </li> Rust is supported by 11 regular subteams and 3 shepherd teams</a> that help mentor and lead discussion </li> Staff engineer at Tumblr mentions that Tumblr is using Rust</a>. </li> Rust is just behind Scala in active github projects (this month: 1263</a> vs 1376</a>) </li> If you haven't seen them yet, the Mozilla Hacks videos are some of Rust's most popular (with one over 50k views</a>) </li> Also in the survey, 16.8% of Rust devs are doing embedded work. </li> Jorge Aparacio</a> has been a long-time supporter for embedded and is writing a book for embedded Rust development</a>. </li> Rust got mentioned in a Batgirl comic</a>. </li> The new Chucklefish game is not the first in Rust. An even earlier game was Shar</a>. </li> Between Rust's first and second year, 56 RFCs were merged. </li> In the last ~3 months, 36 RFCs were merged. </li> Rust has had over 45 releases. Many since 1.0 have been well-received in programming news circles. </li> Another published Rust game is a Snake's Tale</a> </li> Speaking of popular posts, one Rust project, alacritty, received over 1000 points on Hacker News</a>. </li> crates.io has had over 240,000,000 downloads </li> standard Rust tools are voted/promoted by the community. RLS, rustfmt, clippy, and rustup each have over 1000 stars on GitHub. </li> for a bit of history, here's a post that helped start it all</a>. To this day, still a regularly referenced post. </li> Ever wonder about the en-US at the end of https://www.rust-lang.org/en-US/ ? The Rust homepage is in 13 different languages! </li> Language interop is an important piece of Rust's growth. We have C++ interop with bindgen</a>. </li> There's also node.js interop with Neon</a>. </li> There's good Ruby/Rust interop with Helix</a> and others </li> Python and Rust also have on-going interop work</a>. </li> In addition to interop, Rust has been an influence on Swift</a>. </li> Rust's focus on memory safety without performance loss has helped other languages as well, including D</a>. </li> Rust has also been inspired by other languages, including a revamp of errors</a> inspired by Elm. </li> Rust has also grown from a wide range of influences, including discussions with top language designers</a>. </li> Rust's 1st job was on creating browser tech. In making Servo, Rust focused on both safety and speed, w/o one excluding the other. </li> Rust focuses on productivity also, leaning on ergonomics</a> for getting more done with fewer language rough edges. </li> IEEE lists Rust as the 22nd most popular programming language</a>. </li> This year, according to Redmonk, Rust jumped from 47th to 26th most popular programming language</a>. </li> Likewise, Google trends shows Rust with (albeit a little noisy) comfortable growth over the last 5 years</a>. </li> Rust is global, with heavy growth in the United States and Europe. These are the meetup locations worldwide</a>. </li> Google trends also shows strong activity recently in parts of Asia</a>. </li> Just noticed the Rust reddit has had 16 posts in the last 24 hours. Not bad! </li> Rust just isn't a lang, but a community as well, with a focus on helpfulness. Growing this</a> is also important. </li> In addition to community, Rust-focused contractors like Integer 32</a> and Asquera</a> have sprung up. </li> Speaking of, there are a number of trainers available for Rust. The Rust community team</a> can help connect you. </li> If you can make it out to one of the conferences, there are great hands-on labs</a> there as well. </li> We've seen through multiple surveys that people stick around once they get used to Rust. It's a joy to watch. </li> I forgot to mention, there's also an unofficial Rust slack</a> channel. </li> As well as an unofficial discord server</a> for Rust discussions. </li> Rust is used commercially in a wide range of areas</a>, incl. embedded systems, a/v transcoding, servers, and more. </li> The "impl period"</a> is a reaction to Rust's growth. Discussions were at such a high volume that focus was needed. </li> The Rust team at Mozilla is spread across three continents, with most hosting meetups/talks. </li> Speaking of meetups, if you ever have a chance to go, check out the SF Rust meetup. Lots of great talks/ideas (also free pizza!) </li> A core piece of Rust's growth is a sense of courtesy. This comes from the code of conduct</a>. </li> Rust is growing into new areas. Recently, there's been a focus on WASM support, with new tools like wargo</a> </li> Anecdotally, I can say that I see new faces on Rust twitter regularly. More people are getting excited and sharing their adventures. </li> A big feature for commercial adoption just landed in cargo: alternative registries</a>. </li> The Rust community has its own poet that writes limericks on the news, including yours truly</a>. 😅 </li> Another fun survey fact: despite more users and releases every 6 weeks, only ~7% of users had upgrade breakage</a>. </li> Lin Clark</a>, who makes awesome code cartoons, wrote a popular one on the upcoming Firefox style system rewrite</a>. </li> Also, speaking of, there was a followup article on WebRender, another upcoming update to Firefox rendering</a>. </li> While some may see Rust as largely Mozilla-driven, the Rust core team is actually made of folks from four different companies. </li> On a personal note, I've been pleasantly surprised that each Rusty Spike episode has gotten over 500 listens</a>, even though it's new. </li> This is the place-holder for a lot of stuff we can't talk about being done by companies you might know. Wish we could talk about it! </li> Outreach to companies is exclusively grass roots by engineers in those companies promoting Rust and asking people to present. </li> Rust is growing in importance in infosec. Projects like Tor have already begun moving code to Rust</a>. </li> Awesome Rust</a> is well worth a shout-out. If you ever wonder about what's good in the Rust world, check it out. </li> </ul> RLS now available on nightly 2017-04-18T00:00:00+00:00 We've got some good news for Rust IDE support. We recently hit a milestone for the Rust Language Server, a tool that combines the power of the compiler with a fast autocompletion engine. Up to this point, if you wanted to use it you had to build it from scratch. The process was tedious, time-consuming, and error-prone. We're happy to report you can now install RLS from rustup</a>. Landing as a component in rustup means a few important things. First, this will make it easier to get up and running with the RLS. You no longer have to build the RLS to use it. Getting it up and running now just takes the time to run a few rustup commands: rustup default nightly rustup update nightly rustup component add rls rustup component add rust-analysis rustup component add rust-src </code></pre> You can find more helpful tips in the readme</a>. Second, this also makes it easier for IDE/editor plugins to automatically install the RLS and use its capabilities. This opens up the possibility for an even smoother user experience that is driven directly by the IDE plugin. We're looking forward to exploring this further. Lastly, it is now easier to stay up-to-date with the RLS as it improves. Once you install the component, it will update alongside your compiler. All you'll need to do is run the rustup update nightly</code> command, and the RLS and its dependencies will update with it. We've also been hard at work fixing bugs and improving the experience of using the RLS so that it's more stable and accurate than before. For larger projects, you may also notice that the RLS is a bit faster as well. We're excited to get this out to more people and hear your feedback</a>. There's still plenty of work to do, and if you'd like to jump in and help</a>, we'd love to have you. Using Rust in Windows 2017-03-28T00:00:00+00:00 As a Rust user, I was discouraged by not having all the good tips for using Rust in Windows in one place. This is an attempt to just that. Getting started</h1> Step 1: Install Visual Studio Community edition</h2> My recommendation for working with Rust in Windows is to first get Visual Studio 2017 Community</a> installed. Edit: Make sure you check the boxes to install C++ support when you instal Visual Studio, this provides the linker. The Rust compiler will use the linker that VS provides to translate our Rust code all the way to being an application. Having VS installed pays off in more ways than one, as we'll see later. Step 2: Install rustup</h2> The Rust toolchain (including the compiler and the cargo</code> build tool) are managed by a tool called rustup</a>. This is the easiest way get going with Rust. It also has a few advantages, like allowing you to more easily switch the style of compiler (stable/beta/nightly) as well as building for other targets, like embedded systems. Once Visual Studio is installed, rustup will detect it and let us install the compiler that works with the VS toolchain. Rustup on the initial install will let you pick the compiler, and you can select working with nightly or release. What you choose is up to you. Nightly gives you access to new features but hasn't been tested as thoroughly as the stable release compiler. Whether you choose Nightly or Stable, you'll have commands available to you at the terminal. You won't need a separate shell to get to them. The commands 'rustc' and 'cargo' are the Rust compiler and Rust package manager respectively. Step 3: Make sure everything is working</h2> In a source directory, you can test if Rust has been installed correctly by creating a simple project, compiling it, and running it. cargo new hello_world --bin cd hello_world cargo run (or "cargo build" followed by "cargo run") </code></pre> You can find out lots of helpful general getting started tips</a> in the Rust documentation. Writing Rust Code in Windows</h1> Currently, IDE support for Rust is still in its infancy. If you don't need fully intellisense-style IDE experience, pretty much all editors now have basic Rust support. Pick your favorite one, install their Rust plugin or syntax highlighter, and you're good to go. Many of these plugins come with support for a project called 'racer', which provides an autocompletion. Visual Studio users may be interested in Visual Rust</a>. For the more daring, there's a project called the Rust Language Server</a>, which is based on the Language Server Protocol</a> developed for VS Code and now part of a growing number of editors and IDEs. You can find steps for how to set up the RLS and use it with VS Code on its readme</a>. That said, the RLS is still pre-beta quality as of the writing of this post, so please exercise caution if you go this route. Debugging with Visual Studio</h1> Earlier we installed Visual Studio. It seems a waste to have this nice VS install sitting around and not getting more benefit from it it. Let's use it for debugging. Before we do some debugging, there are two steps we'll want to do beforehand: Step 1: Install rust-src</h2> When we're debugging, VS will sometimes complain because it can't find the source code to the standard libs. We can remove this announce by installing the rust-src, and helping VS find the source it's looking for (this is also a helpful step for racer</code> support) rustup component add rust-src </code></pre> This will let you find the source to the standard libs when you're debugging. After the command is run, it will be put in your .multirust directory. For me that directory is: C:\Users\Sophia Turner\.multirust\toolchains\nightly-x86_64-pc-windows-msvc\lib\rustlib\src\rust\src </code></pre> When Visual Studio asks you for a file from the standard library source, this should give you a good place to start looking for it. Step 2: Add natvis for Rust</h2> Rust has its own common structs, like String and Vec, that Visual Studio won't understand by default currently. To be able to view these data structures, we use native visualisation or .natvis files. Rustup will helpfully make them available for us, we just have to copy them into our personal directory. For me, the .natvis files are in: C:\Users\Sophia Turner\.multirust\toolchains\nightly-x86_64-pc-windows-msvc\lib\rustlib\src\rust\src\etc\natvis </code></pre> NOTE: you may need to install the nightly</code> toolchain to have this directory. You don't need to use the nightly compiler as your main compiler, though, to take advantage of them. I copy these files into: C:\Users\Sophia Turner\Documents\Visual Studio 2017\Visualizers </code></pre> If you don't have a Visualizers</code> directory, go ahead and create it and then copy the files into it. Once you start up VS, it should pick them up and give you a little better view of your Rust data. With that, debugging a Rust app works very similar to debugging other apps. There are a few steps you need to do: Compile your projects with debug info. The easiest way is to build using cargo build</code> and NOT cargo build --release</code>.</li> Run your application</li> Attach the VS debugger using the 'Attach...' feature. You should be able to find your application in the list. Once you attach, you can hit pause, set breakpoints, and step as normal.</li> </ul> Debugging with VSCode and MSVC</h1> You can also setup vscode as your development and debugging platform. There are some good tips on how to do this in this blog post</a>. Conclusion</h1> There is still plenty of room for improvement, but already there's enough here that we can write, build, and debug Rust applications without too much fuss. The Rust teams are working to continue to improve the experience of developing in Windows on Rust, but it's encouraging to see the progress and where it could go in the future. Rust Language Server Alpha 2 Release 2017-03-02T00:00:00+00:00 Today, we're happy to announce the second alpha</a> for the Rust Language Server, a project build to bring high-quality Rust IDE support to any IDE or editor. This release brings new features, better stability, and an easier installation than the first alpha. Support for new project types</h2> When a project first starts, it tends to begin life as a library or as an application. As projects grow, sometimes they take on aspects of both, being both a library and an application that can be executed. With alpha 2, we add support for this hybrid project type. Why, you might ask? Editing Cargo's source code With this change we can now work on Cargo in RLS. We're continuing to improve the experience and take on even larger codebases, but it's promising to see the RLS already able to work with real-world projects, like Cargo. Support for custom derive</h2> Rust projects that need serialisation often reach for the popular serde</a> crate to do the work. With alpha 2, we now support the new serde annotations that are based on the new custom derive capabilities, which is sometimes called "macros 1.1". Support for test code</h2> Editing a test in Tokio With alpha 2, you'll also be able to get intellisense for the #[test]</code> functions code in your project, allowing you to seamlessly switch between application and test code without having to change anything in your IDE or project. We'd like to extend this coverage more in the future. Better stability</h2> We've fixed dozens of issues in the RLS and related crates. As a result, the experience is much more solid. We continue to make stability a focus as we push towards the beta release, but we're already pleased with the progress since the first alpha. Easier installation</h2> When you look at the setup instructions</a> for the RLS now, you'll see we've got it down to four steps. Standard library documentation on hover We've also simplified getting access to the standard library documentation. In one step, you can get now all the types and documentation for the standard library to your RLS install, allowing you to get the same IDE experience with the standard library that you do with your own code. Part of the Rust language nursery</h2> The RLS, as it's matured, has proven that it's ready to take a step closer to being an official Rust project. Recently, the Rust tools team voted to bring the RLS in the Rust language 'nursery', a set of blessed projects that are working towards becoming an officially supported part of the Rust toolchain. We're excited to take this step and are committed to continuing to improve it towards the eventual 1.0 release. Looking forward</h1> We're hopeful this will let more people set up the RLS and get the best experience. In the future, we hope to transition the RLS over to the rustup tool, which should allow you to add RLS support with one easy command. This release is a stepping stone to that vision. Announcing Rust Language Server Alpha Release 2017-01-18T00:00:00+00:00 Today, we're announcing the first alpha release of the Rust Language Server (aka RLS)</a>. With this alpha release, this is the first time we're encouraging early adopters to try on real projects and send us feedback</a>. The RLS has now reached a level of maturity where it should be able to run against most Cargo-based Rust projects. But what exactly is the Rust Language Server? In a nutshell</h1> The Rust Language Server is a way of providing editors and IDEs with a range of functionality. Rather than leaving each editor plugin to have to parse and understand the types in your program and provide you with capabilities like refactoring, the RLS centralizes all this logic and provides it to the editor via a standard language server protocol</a>. In this alpha release, this allows editors and IDEs to share the following capabilities: auto-completion - allowing you to complete symbols and press '.' for member lists</li> goto definition - jump to the definition of a given symbol</li> goto symbol - jump to the definition of a symbol you know the name of</li> find all references - show all the locations a given symbol is referenced</li> rename/refactor - rename all instances of a given symbol to a new name</li> types on hover - get the type of a symbol</li> show errors - as the user types, get live analysis showing errors as they happen</li> </ul> The alpha release of the RLS has been run successfully on Linux, Mac, and Windows. How to get started</h1> Step 0: Make sure you have a recent nightly rustc/cargo</a>, git, python, node, and cmake installed Step 1: Check out the RLS: git clone https://github.com/sophiajt/rls </code></pre> Step 2: Set up your editor (here we show VS Code plugin</a>) git clone https://github.com/sophiajt/rls_vscode.git cd rls_vscode npm install </code></pre> Step 3: Set the RLS_ROOT environment variable to point to where you checked out the RLS: export RLS_ROOT=/Source/rls </code></pre> At this point, you can open the project in VS Code, and when you run the project, it will open a new instance of VS Code with the RLS plugin enabled. The plugin will build RLS for you. Note: the initial analysis of a project can take quite a long time, as metadata is built for all the project dependencies. Note 2: analysis currently ignores test code, so you won't get IDE support for it. We have a fix for this in the works. How does it work</h1> The current version of the RLS is built from a combination of two tools: racer</a> and the Rust compiler</a>. Racer allows us to get quick-and-dirty completions, allow you to get completion results in sub-second times. The trade-off this makes is that the results are not as accurate. For tasks that require higher accuracy, like safe refactoring or code navigation, the RLS uses the Rust compiler directly. Because this type of analysis, while more accurate, takes longer to complete, the RLS will tell the editor when the analysis has successfully completed. Help, it doesn't work!</h1> The RLS is closely tied to particular versions of rustc. If you're a rustup user, the DLLs it needs are managed by rustup. In this case, you'll get an error like this if you run the rls by hand: jturner-23759:rls jturner$ ~/.cargo/bin/rls dyld: Library not loaded: @rpath/librustc_driver-6eb85298.dylib Referenced from: /Users/jturner/.cargo/bin/rls Reason: image not found Abort trap: 6 </code></pre> If you see an error like this, you can run the rls by instead running the command cargo run</code> in the rls directory. In the future, as we improve the rls integration with the main Rust tools, this will no longer be an issue. What's next</h1> The RLS is still early in its life and needs to go through a stabilization period before it's ready for 1.0. This alpha release marks the first release in this series. The next release will be a second alpha</a> that includes another set of bugfixes and polish. This will be followed by a beta</a>. The beta will be the first that will be distributed as a binary and will be much closer to final 1.0 experience in terms of ease-of-install and ease-of-use. I want to help</h1> Great! There are a few ways you can jump in. Test the RLS - use the RLS on your projects and tell us how it feels. This will help us shake it out and make sure it's ready for production. Contribute to RLS: - we'd like to add more refactorings, more Rust-centric code navigation (eg find all impls for a type), and possibly more advanced features like lifetime visualization and macro debugging. Contribute an editor plugin - there are already at least six plugins</a> that follow the Language Server Protocol that the RLS uses, each in its own state of completion. If you have experience with one of those editors, or if you have a favorite editor you don't see on the list, you can help us by improving support. While the RLS will continue to grow and mature, it's really the plugin where the "rubber meets the road". It's the plugin that helps the developer get the most out of what the RLS provides. Hack on the compiler - ultimately, we want to drive everything in the RLS straight from the compiler. This means being able to have sub-second response times for interactive completions as well. The Rust compiler team has already started a series of refactoring</a> to make the transition to an interactive mode possible. If you have a background in compilers, IDEs, or static analysis tools and would like to help with this work, you can contact the compiler team at #rustc on irc.mozilla.org or through the Rust compiler project on GitHub</a>. Programming Language and Compilers Reading List 2016-10-08T00:00:00+00:00 This week, the talented Julien Fitzpatrick</a> (btw, you should check out their RustConf talk</a> if you haven't already) asked what a good list would be for people who are interested in programming languages and compilers. I took it as a good excuse to write a blog post with some of my recommendations. I'm going to attempt to put together a list that doesn't require a lot of background information, but as these things go, sometimes you'll have to re-read portions of books to get their benefit. Also, this isn't going to be an exhaustive list. I have never read through an exhaustive reading list before. Instead, this is definitely biased towards writing real code, getting things working, and learning by doing. Without further ado, let's get started! Getting started</h2> To kick things off, the first thing to do is to remove any fears we have about thinking that compilers are big complicated things that are only possible by the most heroic of efforts. Yes, some compilers can be quite complex, but that's true of many kinds of industrial software. At its core, though, the compiler just does a few relatively simple things. So let's demystify those things. First off, let's make a small compiler. @thejameskyle</a> has written a cool tutorial</a> that helps take you through the steps of creating a simple compiler. Rather than chewing your way through dusty tome after dusty tome before you write your first line of code, my suggestion would be to always have a code editor open at each step of the way. Play with the ideas as you learn them. Alright, next, after we've played around with our little compiler and made a few changes and see how our changes affect the output, it's time to start building up our knowledge. Fundamentally, how a programming language works can be described in abstractions that build on other abstractions. If we peel these abstractions away, we're left with a pretty simple foundation. One of the compiler books I liked that help build up these abstractions from a simple base is Jeremy Siek</a>'s compiler course</a>. The notes are laid out by chapter, and each chapter introduces a new concept that you'll add to the compiler. I enjoyed the hands-on approach, since you can think about how each feature affects each part of the compiler. It definitely gave me a "oh my god, I can't believe this works!" moments. You may also want to adjust to fit your particular style. For example, if something like the assembly piece feels awkward, you could instead output code to a language you're more familiar with, and work through the other steps of the compiler. On the programming language side, probably one of the better survey books I've seen is Programming Language Pragmatics</a>. I'd probably pair it with something like Masterminds of Programming</a> or Coders at Work</a>. These two books are interviews with programming language creators. Creating programming languages is definitely equal parts art and engineering, and bouncing between the creators talking about the art and seeing how the engineering works in practice might help see parts of the picture a bit clearer. Working some problems</h2> I used to hate doing homework. I mean, who didn't? But getting a little older, I realize just how useful doing some of the problems in the book can be. That's the time where you actually are solving little puzzles. And, if you're lucky, it's also a great chance to have one of those "a ha!" moments that unlock the next set of secrets. If you're like I was at the time, you've heard of "functional programming" about a million times but weren't sure why anyone should care all that much. It wasn't really until I was working through Introduction to Functional Programming by Bird/Wadler</a> that things started to click. The problems in this book are great. They're juicy enough that you might have to work at them a little, but you come away with a much better sense of the concepts in each chapter. Another book I enjoyed the problems in is the Dragon book</a>. The problem with the Dragon book, though, is that it definitely feels like a stuffy college textbook. That said, it's worth a skim. My favorite part about it are the problems in the earlier chapters. Working through those gave me a bit more confidence I understood the concepts. Getting more modern</h2> Perhaps another problem with the well-known dragon book is that the dragon is a bit long in the tooth these days. Modern techniques have moved on a bit from where they were a few decades ago. One thing to check out to see a bit more of how modern compilers work is the LLVM chapter</a> of Architecture of Open Source Applications (which may also be a fun read for someone interested in how things work generally). I don't know of a good book on this topic, but there's a good video that goes into some of the ways that modern compilers work differently</a> than those in the dragon book. For the brave</h1> When you've worked your way to this point and are feeling pretty confident, it's time to scale to the dizzying heights. To get there, you'll inevitably come across Types and Programming Languages</a> (which you might hear pronounced "tapple"). I'm not going to mince words, this book is brutal. It took months and months of working on it to crack it. I managed to finally get my head around it by methodically using a hands-on approach. I went chapter by chapter</a> and implemented each new language concept the author introduced. It took a little while, but it managed to turn the greek symbols into something a bit more concrete. At this point you have quite the toolbox. You've written a few small compilers, you've seen how language features work and how they interact with each other, you can read academic papers (if need be), and maybe you've even made a programming language of your own. Now you can branch out to almost anywhere. Helping with the Rust Errors 2016-08-03T00:00:00+00:00 I recently did some work with Niko Matsakis on a new compiler error format. You can try them out by setting RUST_NEW_ERROR_FORMAT=true</code>. While I put together a blog post talking about the design for the main blog, I also wanted to open it up for people to come help us move to the new errors. I've split up this post based on background for testers, doc writers, first-time compiler hackers, and experienced compiler hackers. I'm good at testing!</h1> Perfect! We've love your help to come up with tests that show where an error message is confusing or just plain wrong. If you'd like a little help getting started, there are a bunch of unit tests for the compiler that all fail intentionally. You can find these in the rust-lang repo under the src/test/compile_fail</code> directory. You'll notice we've already started a list of errors</a>, but that list is incomplete. We'd like to make sure every error code that needs to be updated makes it onto that list so that others can help update them. There are a bunch of good examples in the compile_fail</code> directory that explore various error codes. If you look around there you can also search for error messages and see if there is already a test. Running this test will help you see if the error has been updated to the new format. If you can find a test case that needs to be updated, file an issue like one of those listed in the "list of errors" link and cc @sophiajt. I can help get the error updated to the new format and tag the error so that others can find it. I love writing docs!</h1> Good, we need that too. There are a couple places where doc writers can help out, especially if if you already know a little Rust. The first place is helping us write clearer error messages. Even now, and especailly as we move to the new format, some errors are confusing and sometimes wrong. By helping write clearer error wording, and by experimenting with new ways to describe the error that is occuring, you can help us further improve the errors for the next developer. If you look at the list of errors</a> we have so far you can see what we're going for. The likely need here is that errors need new labels. The general style of the new errors: The error message text. This describes the general problem.</li> The primary error label (the '^^^' underline). This describes the 'what' of the error. It tries to summarize what went wrong in an fairly brief, yet approachable way.</li> The secondary error label (the '+++' underline). This describes the 'why' of the error. By reading these the user can see the order of operations that lead to the error.</li> </ul> You can also explore the src/test/compile-fail</code> of the compiler source code for a lot of unit tests that fail intentionally. By looking through these, you can help find errors that aren't update or that use confusing notes and labels. Worth mentioning too, if you're a writer, also keep a lookout for Stage 2 (which hopefully will be ready in the coming weeks). We'll need help in the future writing helpful extended messages that describe errors. A great place to jump into help out here and learn about other places doc writers can help is #rust-docs on irc.mozilla.org. I know Rust but I've never hacked the compiler!</h1> Excellent, a first-time compiler hacker. Let's walk through updating an error message to the new format. You can look through the list of errors</a> to see which ones need updating. Once you've downloaded the Rust compiler source</a>, run the command: > python src/bootstrap/bootstrap.py --step libstd --stage 1</code> This will build "Stage 1" of the compiler. After the first compile, this saves us a bunch of time as we test our ideas as it is the minimal compiler rather than a full install. Once built, we can run this by hand using (here, I'm on OS X): RUST_NEW_ERROR_FORMAT=true ./build/x86_64-apple-darwin/stage1/bin/rustc <rust file></code> Let's take a look at an error that needs to be updated. You can find a lot of error tests in the src/test/compile-fail</code> directory in the compiler source. For this example, I'll use src/test/compile-fail/borrowck/borrowck-match-binding-is-assignment.rs</code>. You can also see which error this is looking for the EXXXX number, which is the error number. This one is E0384. error[E0384]: re-assignment of immutable variable `x` --> src/test/compile-fail/borrowck/borrowck-match-binding-is-assignment.rs:50:13 | 50 | x += 1; | ^^^^^^ | note: prior assignment occurs here --> src/test/compile-fail/borrowck/borrowck-match-binding-is-assignment.rs:49:10 | 49 | [x,_,_] => { | ^ </code></pre> We can see that's still using some of the old logic because the underlines don't have a label. You can also tell because there's an attached note is simple enough to be that label. Let's change the note to be a label. To fix this, we first need to find it in the compiler source. Search tools to the rescue. I tend to use the error number (and sometimes part of the error message text) jturner-23759:rust jturner$ grep -r E0384 src src/librustc_borrowck/borrowck/mod.rs: self.tcx.sess, span, E0384, src/librustc_borrowck/diagnostics.rs:E0384: r##" </code></pre> We now have the line that creates the error and another that lists it for us. We only need to change the first of these. Looking at the code, sure enough, this is using the older style: struct_span_err!( self.tcx.sess, span, E0384, "re-assignment of immutable variable `{}`", self.loan_path_to_string(lp)) .span_note(assign.span, "prior assignment occurs here") .emit(); </code></pre> This is a pretty easy fix. We just move from .span_note</code> to .span_label</code> and create a simple label, like this: struct_span_err!( self.tcx.sess, span, E0384, "re-assignment of immutable variable `{}`", self.loan_path_to_string(lp)) .span_label(assign.span, &format!("prior assignment occurs here")) .emit(); </code></pre> Once we build and run the new compiler against the error message, we get something where the label is now part of the main snippet. error[E0384]: re-assignment of immutable variable `x` --> src/test/compile-fail/borrowck/borrowck-match-binding-is-assignment.rs:50:13 | 49 | [x,_,_] => { | - prior assignment occurs here 50 | x += 1; | ^^^^^^ </code></pre> That looks a little better, but we're not done. You'll notice that we don't have a label on our primary span (the ^^^^ part). In the new format, we like to have a label here because it's one of the first places your eye will look when you see an error message. The second is that the label "prior assignment occurs here". That feels a little awkward. Since the label is now part of our code snippet, we can use a better wording. Let's update the error to look like this: error[E0384]: re-assignment of immutable variable `x` --> src/test/compile-fail/borrowck/borrowck-match-binding-is-assignment.rs:50:13 | 49 | [x,_,_] => { | - first assignment to `x` 50 | x += 1; | ^^^^^^ re-assignment of immutable variable </code></pre> With a little bit of wording changes and the new primary label, that's not bad. Of course, as you get comfortable working with these errors you may find even better wording that's more clear. Next, let's update the code to output this style: struct_span_err!( self.tcx.sess, span, E0384, "re-assignment of immutable variable `{}`", self.loan_path_to_string(lp)) .span_label(assign.span, &format!("first assignment to `{}`", self.loan_path_to_string(lp))) .span_label(span, &format!("re-assignment of immutable variable")) .emit(); </code></pre> You'll notice that I reused the same span</code> that the main error message uses in my new .span_label</code>. When I do this, I'm telling the DiagnosticBuilder that this span_label is talking about the primary span and is the root of what is causing the error. Other labels are secondary spans because they're more informational. And with that, we're done! Well, almost. We can't forget to update the unit tests. To quickly check the unit tests that will fail because of our changes, we can run: python src/bootstrap/bootstrap.py --step check-cfail --stage 1</code> Compile-fail unit tests are set up with their own annotations that say what labels to expect and where. As we update errors, we'll need to also update the corresponding test the new expected labels. failures: [compile-fail] compile-fail/asm-out-assign-imm.rs [compile-fail] compile-fail/assign-imm-local-twice.rs [compile-fail] compile-fail/borrowck/borrowck-insert-during-each.rs [compile-fail] compile-fail/liveness-assign-imm-local-in-loop.rs [compile-fail] compile-fail/liveness-assign-imm-local-in-op-eq.rs [compile-fail] compile-fail/liveness-assign-imm-local-with-init.rs test result: FAILED. 2381 passed; 6 failed; 15 ignored; 0 measured </code></pre> Sure enough, we need to update some of the unit tests. Let's run these individually. To do this, we can use: python src/bootstrap/bootstrap.py --step check-cfail <test name> --stage 1</code> like so: python src/bootstrap/bootstrap.py --step check-cfail asm-out-assign-imm --stage 1</code> A fair amount of text comes out, but what we're interested in is: error: /Users/jturner/Source/rust/src/test/compile-fail/asm-out-assign-imm.rs:21: unexpected "note": 'first assignment to `x`' error: /Users/jturner/Source/rust/src/test/compile-fail/asm-out-assign-imm.rs:24: unexpected "note": 're-assignment of immutable variable' error: /Users/jturner/Source/rust/src/test/compile-fail/asm-out-assign-imm.rs:21: expected note not found: prior assignment occurs here </code></pre> These lines tell us what's breaking our test case. In this example we see there are notes being printed that it doesn't expect and one expected note not found. This makes sense, since we changed the wording in the error. Let's go ahead and fix this. We can change: x = 1; //~ NOTE prior assignment occurs here </code></pre> To: x = 1; //~ NOTE first assignment </code></pre> The //~</code> tells the test runner what the expected message is. Here, the message is a NOTE, which corresponds to the label we changed. One more to go. We also added a new label on the primary span. We can insert a new note where this goes: unsafe { asm!("mov $1, $0" : "=r"(x) : "r"(5)); //~^ ERROR re-assignment of immutable variable `x` //~| NOTE re-assignment of immutable //~| NOTE in this expansion of asm! } </code></pre> Notice: we use the //~</code> when the note is by itself, or //~^</code> and //~|</code> when we're looking for multiple notes. With that, the test now passes: running 1 test test [compile-fail] compile-fail/asm-out-assign-imm.rs ... ok test result: ok. 1 passed; 0 failed; 0 ignored; 0 measured </code></pre> From there we work our way through the errors. Having these error tests failing gives us a chance to be sure that our fix is good. Here, when we fix compile-fail/liveness-assign-imm-local-in-loop.rs</code> we see that in a loop the two spans we're using might be the same span if the error happens in a loop. Let's fix that: let mut err = struct_span_err!( self.tcx.sess, span, E0384, "re-assignment of immutable variable `{}`", self.loan_path_to_string(lp)); err.span_label(span, &format!("re-assignment of immutable variable")); if span != assign.span { err.span_label(assign.span, &format!("first assignment to `{}`", self.loan_path_to_string(lp))); } err.emit(); </code></pre> While we're at it, let's run a quick tidy to make sure our code conforms to the compiler coding standard: python src/bootstrap/bootstrap.py --stage 1 --step check-tidy</code> Perfect. Ship it! After we've finished with updating the error and the corresponding unit tests, we can create a PR. When you file the PR, put the message "r? @sophiajt" at the bottom of your commit message. This will add me to do the code review. Extra credit</h1> A keen eye will notice that the file we've been working on did not show up in our failed tests. "Why is that?" you might ask. Unit tests that look at the compiler messages work in a few different ways. Some look at how a test looks (eg, tests in the src/test/ui directory). Others make sure the content is there, like the unit tests we just fixed. Because there is a lot of content to look for there are two kinds of "look for content" error test cases. The first kind will only look at the main message. You can see that this is the case because these test cases have no indicators to look for NOTE or HELP, only ERROR and WARN. Watch what happens if we add a single NOTE to our original test case: error: /Users/jturner/Source/rust/src/test/compile-fail/borrowck/borrowck-match-binding-is-assignment.rs:25: unexpected "note": 'first assignment to `x`' error: /Users/jturner/Source/rust/src/test/compile-fail/borrowck/borrowck-match-binding-is-assignment.rs:32: unexpected "note": 'first assignment to `x`' error: /Users/jturner/Source/rust/src/test/compile-fail/borrowck/borrowck-match-binding-is-assignment.rs:33: unexpected "note": 're-assignment of immutable variable' error: /Users/jturner/Source/rust/src/test/compile-fail/borrowck/borrowck-match-binding-is-assignment.rs:38: unexpected "note": 'first assignment to `x`' error: /Users/jturner/Source/rust/src/test/compile-fail/borrowck/borrowck-match-binding-is-assignment.rs:39: unexpected "note": 're-assignment of immutable variable' error: /Users/jturner/Source/rust/src/test/compile-fail/borrowck/borrowck-match-binding-is-assignment.rs:44: unexpected "note": 'first assignment to `x`' error: /Users/jturner/Source/rust/src/test/compile-fail/borrowck/borrowck-match-binding-is-assignment.rs:45: unexpected "note": 're-assignment of immutable variable' error: /Users/jturner/Source/rust/src/test/compile-fail/borrowck/borrowck-match-binding-is-assignment.rs:50: unexpected "note": 'first assignment to `x`' error: /Users/jturner/Source/rust/src/test/compile-fail/borrowck/borrowck-match-binding-is-assignment.rs:51: unexpected "note": 're-assignment of immutable variable' </code></pre> That's more like it! Now, we can continue fixing this unit test like we did the others. Tips</h2> As you convert errors you may need to change span_err</code> (or span_err!</code>) to struct_span_err</code> (or struct_span_err!</code>), add new .span_labels, or tweak the wording for the error message to feel right. I'm an experienced Rust compiler hacker!</h1> Perfect! There are a few hairy things we need to figure out, still, and they may require the keen eye of a compiler hacker. In the list of errors</a> there is a second list of issues for experienced compiler hackers. Additionally, there are areas that need a bit more work to make them feel like the new messages. Region errors</h2> Regions errors currently provide a decent amount of information, but unfortunately to understand it you currently need a pretty decent understanding of Rust already. Ideally, these messages would get a similar treatment to the borrowck errors mentioned above. Something like: error[E0495]: cannot infer an appropriate lifetime for lifetime parameter 'r in function call due to conflicting requirements --> src/test/compile-fail/regions-infer-call-3.rs:18:24 | 18 | let z = with(|y| { select(x, y) }); | ^^^^^^^^^^^^ | note: first, the lifetime cannot outlive the anonymous lifetime #1 defined on the block at 18:21... --> src/test/compile-fail/regions-infer-call-3.rs:18:22 | 18 | let z = with(|y| { select(x, y) }); | ^^^^^^^^^^^^^^^^ note: ...so that reference does not outlive borrowed content --> src/test/compile-fail/regions-infer-call-3.rs:18:34 | 18 | let z = with(|y| { select(x, y) }); | ^ note: but, the lifetime must be valid for the call at 18:12... --> src/test/compile-fail/regions-infer-call-3.rs:18:13 | 18 | let z = with(|y| { select(x, y) }); | ^^^^^^^^^^^^^^^^^^^^^^^^^^ note: ...so type `&isize` of expression is valid during the expression --> src/test/compile-fail/regions-infer-call-3.rs:18:13 | 18 | let z = with(|y| { select(x, y) }); | ^^^^^^^^^^^^^^^^^^^^^^^^^^ </code></pre> You can see that we say things like "for the call at 18:12" or "on the block at 18:21". The span information is available but we're not really using it well. Instead of saying this, we may want to calculate the actual span that relates to "the call at 18:12" or the end point of "the block at 18:21". Macro errors</h2> I've started a thread on improving macro errors</a> on the Rust internals list. The hope is to look at a better overall design for macro errors that fits better with the new error style. Thanks!</h1> Thanks for wanting to jump in and help! You can find me on irc.mozilla.org as sophiajt and on github.com as sophiajt. If you have any questions, please ask. You can also get lots of good help in #rust-internals and #rustc on irc.mozilla.org. Rhai 0.2 release 2016-03-26T00:00:00+00:00 I've just updated Rhai</a> to 0.2. This release focuses on improving the interaction between Rhai and Rust. The result is a cleaner, easier-to-use API. This new API does mean some API breakages, so you'll need to update your code to work with 0.2. With this release, Rhai also now has arrays as a built-in type. You can upgrade/install Rhai from cargo</a>. API improvements</h1> Rhai 0.2 has a lot of small improvements that give it a much more natural feel in Rust. The main three areas are function registration, getters/setter helpers, and simpler eval. Function registration</h2> Function registration now works through method calls on the engine, and we can infer the expected function type. The end result is generally much cleaner. [Before] &(add as fn(x: i32, y: i32)->i32).register(&mut engine, "add"); </code></pre> [Now] engine.register_fn("add", add); </code></pre> Getter/setter helpers</h2> To help cut down on repetition, Rhai 0.2 has a few helpers for working with getters and setters. [Before] &(TestStruct::get_x as fn(&mut TestStruct)->i32).register(&mut engine, "get$x"); &(TestStruct::set_x as fn(&mut TestStruct, i32)->()).register(&mut engine, "set$x"); </code></pre> [Now] engine.register_get_set("x", TestStruct::get_x, TestStruct::set_x); </code></pre> Simpler eval</h2> Eval will now take the expected return type as a type parameter, letting you cut out a few steps. We've also moved to using &str rather than String for eval inputs. [Before] if let Ok(result) = engine.eval("40 + 2".to_string()).unwrap().downcast::<i32>() {} </code></pre> [Now] if let Ok(result) = engine.eval::<i32>("40 + 2") {} </code></pre> New feature: arrays</h1> In addition to the API improvements, Rhai now has a built-in array type. var y = [1, 2, 3]; y[1] = 5; print(y[1]); </code></pre> Thanks!</h1> Thanks for the feedback on Rhai's first release. Also, big thanks to @nikomatsakis</a>, who gave me a few key pointers on how improve the API. Looking forward to hearing from you. Embedded scripting in Rust 2016-03-14T00:00:00+00:00 For the last few weeks, I've been working on an embedded scripting language for Rust, based loosely on ChaiScript</a> called Rhai</a>. What's an embedded scripting language? While the definition might depend on who you ask, for this post embedded scripting has a few distinct features: A language that derives all functionality from bindings to the hosting environment</li> As a result, it allows complete control over the API exposed to the script, letting you lock down the script to your application's needs</li> </ul> In short, it's like the little brother of a full scripting language. It's smaller, thinner, but still useful for a variety of apps. Because I'm using Rust as the host language, I also put a few other constraints on the project to make it work with a wider range of use cases. No use of 'unsafe' - this lets us script where security is a concern</li> No additional dependencies - keeps our design small</li> Easy-to-use - even being safe and lightweight, it still needs to be easy to get started</li> </ul> Introducing: Rhai</h1> Let's take a look at what Rhai looks like, then we'll break down how it works. Hello world</h2> As I mentioned earlier, one of the goals with Rhai was to have minimal setup to get going. This example is a full "hello world" of scripting that evaluates some script and gives us the result: extern crate rhai; use rhai::Engine; fn main() { let mut engine = Engine::new(); if let Ok(result) = engine.eval("40 + 2".to_string()) .unwrap().downcast::<i32>() { println!("Answer: {}", result); // prints 42 } } </code></pre> Looking a little closer. We pull in the Rhai crate and get the scripting engine into view: extern crate rhai; use rhai::Engine; </code></pre> Next, we create an instance of the engine: let mut engine = Engine::new(); </code></pre> Finally, we call into the engine. Perhaps the most interesting part here is the roundtrip. The 'eval' method takes a String</code>, and returns a Result<Box<Any>, EvalAltResult></code>. Because we're in a scripting context, everything we're working with is dynamically typed, so as values get passed around, we make liberal use of Box<Any></code>. Here, the expression "40 + 2" is going to evaluated to give us a Box'ed number. Of course, we can also have errors occur during script execution, so we need to return a Result. To go from a Result to the actual numeric value, we have to take a few steps to peel away the layers. I cheat and use 'unwrap' to get through the Result, then the downcast takes us from a Box<Any></code> to a Box<i32></code>. engine.eval("40 + 2".to_string()).unwrap().downcast::<i32>() </code></pre> That's it. We've done a full round trip of creating script and then making sense of its output. The language, at this point, has only basic functionality because we haven't registered anything the script can use. So let's fix that. Registering your functions</h2> To make your functions visible to the script, you'll need to register them with the script engine. Let's do the same example, but this time we'll create an add function in our native code and call it from script: extern crate rhai; use rhai::{Engine, FnRegister}; fn add(x: i32, y: i32) -> i32 { x + y } fn main() { let mut engine = Engine::new(); &(add as fn(x: i32, y: i32)->i32).register(&mut engine, "add"); if let Ok(result) = engine.eval("add(40, 2)".to_string()) .unwrap().downcast::<i32>() { println!("Answer: {}", result); // prints 42 } } </code></pre> The big change in this example is this line: &(add as fn(x: i32, y: i32)->i32).register(&mut engine, "add"); </code></pre> This uses the function registration capability taken from the FnRegister trait that we've added to our imports. It's a trait implemenented on functions which gives them a register method. We call the method and pass into it both our engine and the name we want the function to be called in script. With that, we're good to go and can call the function. Let's do two more examples so we can see how some of the other details play out. Function overloading</h2> Rhai also has supported for function overloading. This gives you a way to work with generic functions (like the example below) or to register multiple functions to the same name. During call resolution, the engine will call these functions and will fall through to the case that matches the types at runtime. use std::fmt::Display; extern crate rhai; use rhai::{Engine, FnRegister}; fn showit<T: Display>(x: &mut T) -> () { println!("{}", x) } fn main() { let mut engine = Engine::new(); &(showit as fn(x: &mut i32)->()).register(&mut engine, "print"); &(showit as fn(x: &mut bool)->()).register(&mut engine, "print"); &(showit as fn(x: &mut String)->()).register(&mut engine, "print"); } </code></pre> Working with custom types</h2> Finally, in addition to working with your functions, Rhai also supports working with custom types. In this example, we introduce a few more features. First, you can register functions that take a &mut</code> as their first argument. This allows them to be used as methods in the script. Next, we register a special getter function. This lets us read (or write, if we register a setter also) the member directly in script. You'll notice too that we make sure our object is clone-able. As we'll talk about in the next section, this ends up being a requirement when working with values in the script engine. We make sure to register our custom type with the engine to make cloning possible from inside the engine. extern crate rhai; use rhai::{Engine, FnRegister}; #[derive(Clone)] struct TestStruct { x: i32 } impl TestStruct { fn update(&mut self) { self.x += 1000; } fn get_x(&mut self) -> i32 { self.x } fn new() -> TestStruct { TestStruct { x: 1 } } } fn main() { let mut engine = Engine::new(); engine.register_type::<TestStruct>(); &(TestStruct::new as fn()->TestStruct).register(&mut engine, "new_ts"); &(TestStruct::update as fn(&mut TestStruct)->()) .register(&mut engine, "update"); &(TestStruct::get_x as fn(&mut TestStruct)->i32) .register(&mut engine, "get$x"); if let Ok(result) = engine.eval("var myts = new_ts(); myts.update(); myts.x".to_string()) .unwrap().downcast::<i32>() { println!("result: {}", result); // prints 1001 } } </code></pre> How it works</h1> Let's take a look at how the scripting engine works. There may be even better ways to do some of these steps, so I'm eager to hear your feedback. Without further ado, let's jump in. Function resolution</h2> The heart of the system is the function resolver. In Rhai, all function calls, even all operators like plus and minus, end up going through the resolver. Yet, it's a relatively simple piece of code. First, find all functions which match the name. Let's say 'foo'</li> For each 'foo' function, try to call the function with the argument(s) given</li> If it succeeds, return the resulting value</li> If it fails, continue trying the rest of the functions registered to 'foo'</li> If none succeed, error</li> </ul> Going through all the functions which match a name gives us the ability to do overloaded functions. It also lets us do operators, which themselves are commonly overloaded. Function registration</h2> The next piece to working with functions in the engine is registering them. There are two kinds of functions that are registered: inner functions, or those created inside the script itself, and external functions, or those created in the native code. Inner functions are relatively simple. They work by creating a new scope, adding the function arguments to this new scope, and then executing the function body. External functions, if you will, are where the magic happens. Here, we handle the boundary between the scripting world and the native world. Let's look at how to register a function with one pass-by-value argument fn(T)->U</code>: impl<T: Any+Clone, U: Any+Clone> FnRegister for fn(T)->U { fn register(self, engine: &mut Engine, name: &str) { let wrapped : Box<Fn(&mut Box<Any>)->Result<Box<Any>, EvalAltResult>> = Box::new( move |arg: &mut Box<Any>| { let inside = (arg).downcast_mut() as Option<&mut T>; match inside { Some(b) => Ok(Box::new(self(b.clone())) as Box<Any>), None => Err(EvalAltResult::ErrorFunctionArgMismatch) } } ); let ent = engine.fns.entry(name.to_string()).or_insert(Vec::new()); (ent).push(FnType::ExternalFn1(wrapped)); } } </code></pre> There's a lot going on, so let's break it down: impl<T: Any+Clone, U: Any+Clone> FnRegister for fn(T)->U </code></pre> As I mentioned earlier, all the types used in the scripting engine need to be clone-able. We enforce that here as well because we'll use the clone feature in the call. We are implementing the FnRegister trait on all functions that match this constraint. Next, we are going to create a wrapper function which will call the function given: let wrapped : Box<Fn(&mut Box<Any>)->Result<Box<Any>, EvalAltResult>> </code></pre> All functions of arity one have this internal wrapper function, regardless of if they are pass by value or reference. This lets us work through a vector of functions easily during function resolution, without having to have too much information out the specific type of the function. Next, we have the wrapper function itself: Box::new( move |arg: &mut Box<Any>| { let inside = (arg).downcast_mut() as Option<&mut T>; match inside { Some(b) => Ok(Box::new(self(b.clone())) as Box<Any>), None => Err(EvalAltResult::ErrorFunctionArgMismatch) } } ); </code></pre> The wrapper is a closure that moves its environment. In this case, it's moving the function we're wrapping. Once it has ownership, we can safely return the wrapper and complete registration. Inside the closure, we handle working with the argument. Because we know the type of the function during registration, we use this knowledge to downcast the argument and give it the proper type. If this succeeds, we know we have the correct function and can call it. If it fails, we know this isn't a match and later can try the next function in the overloads. To call the function, since we're pass by value, we need to take the extra cloning step to get to a value rather than a reference. This lets us respect the ownership model, albeit at the cost of the clone. That's the heart of how the resolution system works. Resolution works through each possible match, and registration gives us a wrapper function that can try each function against the types known at compile time. The rest of the story</h2> Many of the remaining features built on these two main components. As mentioned earlier, when you register special get and set functions for fields, the engine will use them when you work with the fields in script. The engine will register a set of basic arithmetic and comparison operators to get started with. These also use the same function resolution mechanism. For variables, the engine works by passing a scope stack around. For lookup, in order to follow the ownership rules (and maintain pass-by-value semantics), we clone variable values when they're used in an expression. During assignment, we use a mutable reference to their value in the scope stack so that it can be updated with the right-hand side of the assignment. What's next?</h1> Rhai is still very young and has a bit of growing to do before it's ready for primetime. Parsing and error reporting are barebones and don't have the kind of error recovery you'd expect from more industrial-strength scripting. There are also a number of convenience features that would be relatively easy to add, like vectors, iterators, for..in.., and so on. If you'd like to help out, jump over to the GitHub site</a>. Going down the rabbit hole with Rust traits 2016-02-09T00:00:00+00:00 One of the first things you might notice about Rust, if you come to it from other OOP-style languages, is that Rust separates methods from the data they work on. You create your struct</code>, then you impl</code> a few methods on it later. struct Rect { height: i32, width: i32 } impl Rect { fn area(&self) -> i32 { self.height * self.width } } fn main() { let r = Rect { height: 10, width: 5 }; println!("area: {}", r.area()); } </code></pre> Which, to me, seemed slightly awkward at first. But, as I dug in a little futher, I started asking questions like "Can I do that to existing types? Even basic types like the String type?" Indeed you can. In this post we'll see some of what's possible. Warning: we're going down a rabbit hole under the influence of the Bad Idea Bears</a>. I hope you have your snorkel. :) Warming up</h1> We've talked a little about the basics already. In Rust, impl</code> gives us a way to "add" methods onto types. But, when we try to do the same thing to String</code>, we hit a snag: impl String { fn my_len(&self) -> usize { self.len() } } fn main() { let name = "foo".to_string(); println!("my length: {}", name.my_len()); } </code></pre> When we try to compile this, the compiler gives us back an error: mylen.rs:1:1: 3:2 error: cannot define inherent `impl` for a type outside of the crate where the type is defined; define and implement a trait or new type instead [E0116] mylen.rs:1 impl String { mylen.rs:2 fn my_len(&self) -> usize { self.len() } mylen.rs:3 } </code></pre> So we can't impl</code> for a type outside of... oh wait. We can do it, so long as we give it a trait to work with. Try #2: trait MyLen { fn my_len(&self) -> usize; } impl MyLen for String { fn my_len(&self) -> usize { self.len() } } fn main() { let name = "foo".to_string(); println!("my length: {}", name.my_len()); } </code></pre> Woah, that worked! With that, we peek into the rabbit hole. Traits and extending strings</h1> To kick off our explorations, let's give ourselves an example to use. Let's say we want to read in the contents of some file, and we get the filename for that file from the commandline. use std::io::Error; fn read_all(fname: String) -> Result<String, Error> { use std::fs::File; use std::io::prelude::; let mut contents = String::new(); let mut f = try!(File::open(fname)); try!(f.read_to_string(&mut contents)); Ok(contents) } fn main() { use std::env::args; if let Some(fname) = args().nth(1) { if let Ok(contents) = read_all(fname) { println!("{}", contents); } } } </code></pre> Wouldn't it be nice to not mix and match helper functions and methods? Wouldn't it read a little nicer to havefname.read_all()</code>? Using the same pattern we used before, this is pretty easy to do. use std::io::Error; trait FileReader { fn read_all(&self) -> Result<String, Error>; } impl FileReader for str { fn read_all(&self) -> Result<String, Error> { use std::fs::File; use std::io::prelude::; let mut contents = String::new(); let mut f = try!(File::open(self)); try!(f.read_to_string(&mut contents)); Ok(contents) } } fn main() { use std::env::args; if let Some(fname) = args().nth(1) { if let Ok(contents) = fname.read_all() { println!("{}", contents); } } } </code></pre> That's not bad! Go get my nth argument, check if we're good, then read the contents. If we could stop ourselves there, we'd be mostly okay. Our example is still pretty readable, though we might get a few dirty looks from the senior devs in the lunch line. Traits and extending Option</h1> The Bad Idea Bears call us deeper! Can we play a little code "golf"</a> and squeeze out using Option</code> in our main? Sure! All we need to do is create another impl</code> for our FileReader</code> trait, this time for Option</code> (specifically those that contain Strings): use std::io::Error; trait FileReader { fn read_all(&self) -> Result<String, Error>; } impl FileReader for str { fn read_all(&self) -> Result<String, Error> { use std::fs::File; use std::io::prelude::; let mut contents = String::new(); let mut f = try!(File::open(self)); try!(f.read_to_string(&mut contents)); Ok(contents) } } impl FileReader for Option<String> { fn read_all(&self) -> Result<String, Error> { use std::io::ErrorKind; match self { Option::None => Err(Error::new(ErrorKind::NotFound, "No such file or directory")), Option::Some(ref s) => s.read_all() } } } fn main() { use std::env::args; if let Ok(contents) = args().nth(1).read_all() { println!("{}", contents); } } </code></pre> Yay! Our main is smaller, and we've still managed to maintain at least a little of our decency. I mean, we could have used unwrap</code>, and thrown care to the wind! Instead, we're still doing error-checking on our contents. Albeit, our error-checking is kinda smooshed into one place. What's that, Bad Idea Bear? We should go further? Okay! Traits and extending iterators</h1> We're doing a read_all()</code> on the Option, but what if we go right to the arguments themselves? If we poke around in the docs, we find out that args() returns its own type, Args</a>. As we read on we see that Args is just an Iterator</a>. That gives us a couple choices. We can implement a new trait on Args: trait FileReaderIter { fn read_nth(&mut self, pos: usize) -> Result<String, Error>; } impl FileReaderIter for Args { fn read_nth(&mut self, pos: usize) -> Result<String, Error> { self.nth(pos).read_all() } } </code></pre> Or, we can work with Iterator: trait FileReaderIter : Iterator<Item=String> + Sized { fn read_nth(&mut self, pos: usize) -> Result<String, Error> { self.nth(pos).read_all() } } impl FileReaderIter for Args {} </code></pre> Let's actually stop here for a minute and look at this one. We can't extend Iterator itself. This is a Good Thing(tm), because if we could, we would end up having to extend countless implementors of Iterator. Instead, we need to create a new trait that extends from Iterator. Next, we need to say what kind of iterator we can work on. We know ahead of time that this works on specifically iterators over Strings. We use the <Item=String></code> to lock that knowledge in place for the compiler. This is called an associated type. With the new trait and associated type, we're almost done. Lastly, we're doing something a little different this time around. We have a default implementation inside of the trait instead of just having a prototype. A default implementation saves you the trouble of having to repeat yourself with each type that implements the trait, if the function is general and trivial. The gotcha is that if you use a default implementation that uses self</code>, the compiler needs to know how big self</code> will be because it will pass it by-value. Hence, anything that implements FileReaderIter</code> must have a known size so the compiler knows how many bytes to copy in when self</code> is passed to the method. To ensure this, we add another trait to inherit from, Sized</code>. With that, we can say that Args trivially implements this trait. Phew, okay, that's it. What have we gotten for our effort? Both approaches give us this simple main: fn main() { use std::env::args; if let Ok(contents) = args().read_nth(1) { println!("{}", contents); } } </code></pre> At this point we're far enough down in the rabbit hole that there's only a pinpoint of light from the entrance. Let's work our way back out, shall we? (ps: if you want to follow Bad Idea Bear further, this little nugget is hidden in the Rust docs: both Result and Option implement iterator traits and can be used as iterators</a>) Climbing back out of the rabbit hole</h1> Cleverness and readability often trade off with each other. As we tried to get more clever and squeeze our main down, the end result became more code we had to maintain. We also increased the amount of 'magic' we used, whether it was treating multiple failure styles/cases as one case or the amount of additional capabilities we were giving types like String and Option. Rust's trait system is powerful and clever. It lets you express the algorithms that are core to your project in a way that can even extend the basic Rust API. Indeed, that's one reason why the Rust language and core libraries are intentionally kept pretty small and tidy: so you can make these kind of additions (using eg. modules from creates.io</a>) and the standard library doesn't get in your way. But, as they say, with great power comes great opportunities to follow Bad Idea Bears down rabbit holes. Rust quickie - matching Strings 2016-02-08T00:00:00+00:00 In case you find yourself trying to match a String</code> (perhaps as part of an Option</code> or Result</code>), here's a little trick. As a concrete example, let's say you're working with commandline args and want to do different things if it's there and equal to a special value (like "-"), if it's any other filename, or if it's not there. match args().nth(1) { Some("-") => println!("Input is stdin"), Some(ref x) => println!("Open file: {}", x), None => println!("Open default file /foo/bar") } </code></pre> When you try to compile this, you get: error: mismatched types: expected `collections::string::String`, found `&'static str` (expected struct `collections::string::String`, found &-ptr) [E0308] </code></pre> Bummer! At least on the surface, it looks like we don't have an easy way to use a string constant like "-" in our pattern match when what we're matching against a String. Luckily, Rust pattern matching has a way to help us. We can approximate the above by using match guards</a>. Match guards allow us to put an additional if</code> expression on the pattern. match args().nth(1) { Some(ref x) if x == "-" => println!("Input is stdin"), Some(ref x) => println!("Open file: {}", x), None => println!("Open default file /foo/bar") } </code></pre> Match guards give you an extra bit of "juice" to express your logic. Feature Bias, or Rethinking the Blub Paradox 2016-01-23T00:00:00+00:00 I got a lot of great feedback on my previous post "Rust and the Blub Paradox"</a>, which inspired me to write a follow-up. In its original formulation, the Blub Paradox</a> is something of a tool for people to feel smug about their favorite language. I heard a lot of feedback that it feels condescending. I hadn't realized that in my mind the Blub Paradox had morphed from its original form into something different. In this post, I'd like to break down the Blub Paradox into to what I see as its fundamental observation: that we, as programmers, can have a form of unconscious bias. I'll call this Feature Bias so that we don't confuse it with Paul's original idea. There are no universally better languages</h1> There is no way to universally sort programming languages on a continuum that is monotonically increasing in awesomeness. It's impossible. Not only are languages too complex to have any kind of universal ranking, they are tools and as tools the value is left to the tool user. It's like asking "is pencil better than a paint brush?" It all depends on what you're doing and what you're more comfortable with. The types of languages that Paul criticizes in Beating the Averages aren't universally worse than his favorite languages. Taking one example, everyone loves to put Java down, but have you ever watched Notch (the creator of Minecraft) code in it</a>? He seems perfectly content to create with his tool of choice, and he's certainly capable of doing so. Feature Bias happens locally per feature</h1> Rather than trying to sort languages universally, a useful version of the Blub Paradox is a local phenomenon. It happens on a feature by feature level. If a programmer accustomed to working with a feature sees a language that lacks the feature, they can also see the use cases where programmers in that other language might benefit from its use. If a programmer who doesn't use a feature looks at a language with this feature, until they've spent time to understand it, they may see the feature as unnecessary. Feature Bias happens to most programmers, even the best</h1> As a language PM at Microsoft sitting on a few programming language committees, I saw Feature Bias happen to people from junior developers to famous language designers with decades of industry experience. The first time a developer sees a new feature, regardless of skill level, there's a chance they may experience the bias. True story: the C# feature LINQ had to 'bake' for at least a year with the design committee before enough members saw its value for it to become part of the language. All the while the champions behind it were trying to educate and get other committee members to give it a chance. Features don't have universal value</h1> Once the developer puts in the time to understand the feature, they still may not like it. That's not the paradox/bias anymore. They've put the time in to understand the trade-offs, and they don't feel the trade-offs are worth it. That's a totally acceptable response. The root of the bias</h1> To summarize, just like other unconscious biases, this bias is pointing out something that's common for people to experience and they may not realize they're experiencing it. A lot of programmers I know, including myself on a good day, use an understanding of the bias as a way to train ourselves to take time with each feature and work through the knee-jerk "oh this is crap!" reaction. I definitely still fall victim to it. Just as with other biases, you have to keep working at it. In seeing the responses from people, it's unfortunate that the original Blub Paradox has been used as a club to beat people with. It makes it that much harder to see where it can be helpful. In my experience, it can shine a light on an area that helps us become better programmers and teachers. Rust and the Blub Paradox 2016-01-22T00:00:00+00:00 A few weeks ago, I read an analysis of Rust, D, and Go</a> by Andrei Alexandrescu. Andrei, a respected member of the C++ community</a> and a core developer of the D programming language</a>, took a stab at Rust at the end of his writeup with what seems like a pretty astute observation: "Reading any amount of Rust code evokes the joke "friends don't let friends skip leg day" and the comic imagery of men with hulky torsos resting on skinny legs. Rust puts safe, precise memory management front and center of everything. Unfortunately, that's seldom the problem domain, which means a large fraction of the thinking and coding are dedicated to essentially a clerical job" Having met Andrei and seen a few of his talks, I know that he likes to poke fun. Still, let's take the bait. Is his observation funny because it's a funny image or is it funny because there's an underlying truth to what he's saying? EDIT: thanks to all the responses to this post, I've written a follow-up on where the Blub Paradox is wrong and where it's helpful</a>. EDIT #2: Wow, some people also seem to be taking this article as an attack on Andrei. Not at all! Like I said, I've chatted with him in person and he's very much the type that likes to say funny things, whether about himself or the wide array of programming languages. I just took one of his quotes about Rust as a starting point and had fun with it and looked at what might be weird about Rust coming from other languages. Blub paradox</h1> Whenever I think about how useful certain language features are, I almost always come back to Paul Graham's essay "Beating the Averages"</a>. In it, he observes an interesting phenomenon among programmers he calls "the Blub Paradox". To recap for those who haven't heard of it, the paradox goes something like this: let's say there's a programmer using a language called Blub. Blub falls somewhere in the middle of the programming language pack in terms of capability. It's not the lowest, most minimal language nor is it the most powerful. When the Blub programmer looks "down" the spectrum of languages, they can easily see that those languages are less expressive than their preferred Blub language. Unfortunately, when they look "up" the spectrum of languages, they can't tell they're looking up. As Paul puts it: "What he sees are merely weird languages. He probably considers them about equivalent in power to Blub, but with all this other hairy stuff thrown in as well. Blub is good enough for him, because he thinks in Blub." When I read that years ago, I remember thinking "wow, that's pretty astute." Little did I know that years later it would become firmly ingrained as part of how I thought about teaching programming as a programming language PM. As a programming language PM at Microsoft, I worked on TypeScript</a>, a typed version of JavaScript. Without fail, if I was speaking to an audience of predominantly JavaScript developers, I would get many frowns as I tried to enumerate the advantages of adding a bit of type-checking to JavaScript. Even if it was optional. Even if I showed half a dozen powerful advantages. As Paul notes, it just looked "weird". To a JavaScript developer, TypeScript would often just seem like a language that was equally expressive with "hairy" stuff thrown in. As I talked to more language PMs, and I met more and more people at conferences, I learned that Paul's observation was not only accurate, it was also surprisingly universal. Most programmers will struggle when looking at a new language they don't use. Often, they'll have allergic reactions to features that are foreign. It takes time to work with the features enough to understand why they aren't just unnecessary ornamenation gumming up the works. In short, the Blub Paradox is something that as programmers we actively need to assume we're falling into and then work to get ourselves out of. Let's do just that. Let's look at a few stand-out Rust features and assume these features are weird and unnecessary. From there, we'll see if we can work to un-Blub ourselves. Weird feature #1: Rust-style polymorphism</h1> Let's make a Rust program that uses a bit of polymorphism to print out two different structs. I'll show you the code first, then break it down. use std::fmt; struct Foo { x: i32 } impl fmt::Display for Foo { fn fmt(&self, f: &mut fmt::Formatter) -> fmt::Result { write!(f, "(x: {})", self.x) } } struct Bar { x: i32, y: i32 } impl fmt::Display for Bar { fn fmt(&self, f: &mut fmt::Formatter) -> fmt::Result { write!(f, "(x: {}, y: {})", self.x, self.y) } } fn print_me<T: fmt::Display>(obj : T) { println!("Value: {}", obj) } fn main() { let foo = Foo {x: 7}; let bar = Bar {x: 5, y: 10}; print_me(foo); print_me(bar); } </code></pre> What an eye-full! There's polymorphism there, but it doesn't look anything like OO. It's generic, and not only generic, but it has constraints. And what's this impl</code> stuff? Let's break it into parts. I'm creating two structs to hold our values. I go a step further and make sure they implement something called fmt::Display</code>. In C++, you might overload the << operator for ostream. The end result is similar in Rust. After I have this in place, I can call print functions and pass these structs directly. With that, we're half way through the example. Next, we have our print_me</code> function. This function is generic, and will accept anything so long as what is passed in can do fmt::Display</code>. Luckily, we've already said how both structs can do just that. The rest is straight-forward. We create a few structs and pass them to print_me</code>. Phew... that seemed like a bit of work. And really, that's how polymorphism starts in Rust. It's all with generics. Now, let's switch to C++ for a minute. Someone using C++, especially a beginner, might not go the generic route at first and might instead opt to use OO-style polymorphism: #include <iostream> class Foo { public: int x; virtual void print(); }; class Bar: public Foo { public: int y; virtual void print(); }; void Foo::print() { std::cout << "x: " << this->x << '\n'; } void Bar::print() { std::cout << "x: " << this->x << " y: " << this->y << '\n'; } void print(Foo foo) { foo.print(); } void print2(Foo &foo) { foo.print(); } void print3(Foo foo) { foo->print(); } int main() { Bar bar; bar.x = 5; bar.y = 10; print(bar); print2(bar); print3(&bar); } </code></pre> Pretty easy, right? Okay, here's a quick quiz for you. What does the C++ code print? If you guessed wrong, don't worry. You're in good company. If you guessed right, congrats! Now, think a minute about how much do you have to understand about C++ to know the answer. From what I see, you need to understand the stack, how objects are copied, when they are copied, pointers, references, v-tables, and virtual dispatch. Just to do a few simple lines of OO. When I first started learning C++, I had a pretty steep hill to climb. Luckily, my cousin</a> is an expert C++ programmer and took me under-wing to show me the ropes. Still, I made tons of beginner mistakes like the one above. Why? In part, the hill is steep because of the cognitive load of learning C++. Some of that cognitive load is inherent in programming a computer at a system's level. You need to understand the stack. You need to understand how pointers work. What C++ adds is another layer of complexity around knowing when a value isn't fully copied and when virtual dispatch is and isn't used -- all without any warning to the user if they do something that is "probably a bad idea(tm)". This isn't to knock on C++. Much of Rust is based on keeping the C++ philosophy of low-level efficient abstractions intact. In fact, you can even write code that's very similar to the Rust example</a>. What Rust does do in this case is to separate inheritance from polymorphism and push you towards genericity from the start. In doing so, it gets you thinking generically from day #1. It might seem weird to separate inheritance from polymorphism if you're used to them being used together. That separation gives us one of our first Blub moments: what's the advantage of separating inheritance and polymorphism? In fact, does Rust even have inheritance? Believe it or not, at least as of Rust 1.5, Rust does not allow inheritance of structs. Instead, we can grow the functionality of structs from outside them using a feature called 'traits'. Traits allow you a way to add methods, require methods, and retrofit datastructures for existing systems. They're also allowed to inherit, so one trait can extend another. If you poke around in the Rust, you'll also notice something else. We don't have the same issues we had with C++. There's no worrying about what gets lost when the function is called or how virtual dispatch works. In Rust, we're uniform regardless of the type. Because of this, a whole class of beginner errors vanish. Weird feature #2: what, no exceptions?</h1> While we're on the subject of things Rust doesn't have, its next weird feature is that it lacks exceptions. Isn't that like going backwards? How do we talk about errors that only occur sometimes? How do we bubble out errors so we can handle them in one place? Next you'll be saying I have to learn monads. Actually... no, just kidding, you don't need to learn monads. Rust's exception-less programming is a lot more straight-forward. Here's an example of what this looks like in practice. First, examples of what the functions look like: impl SystemTime { /// Returns the system time corresponding to "now". pub fn now() -> SystemTime; /// Returns an `Err` if `earlier` is later pub fn duration_from_earlier(&self, earlier: SystemTime) -> Result<Duration, SystemTimeError>; } </code></pre> Notice the now</code> function returns a SystemTime and doesn't have any error cases, while duration_from_earlier</code> does have a Result that can be either a Duration or the error SystemTimeError. At a glance, you know a function's success and possible failure cases. But all those failure cases are bound to stack up. Who wants to see them all over the code? Sure, it's good to do error checking, but the point of exceptions is to be able to handle them not only locally but also to let them bubble up and handle them in one place. Rust lets you do that, too. fn load_header(file: &mut File) -> Result<Header, io::Error> { Ok(Header { header_block: try!(file.read_u32()) }) } fn load_metadata(file: &mut File) -> Result<Metadata, io::Error> { Ok(Metadata { metadata_block: try!(file.read_u32()) }) } fn load_audio(file: &mut File) -> Result<Audio, io::Error> { let header = try!(load_header(file)); let metadata = try!(load_metadata(file)); Ok(Audio { header: header, metadata: metadata }) } </code></pre> While it may not look like it at first, the above example uses the "bubbling" style of exceptions. The trick is in a Rust macro called try!</code>. What it does is pretty simple. It will call a function for you. If that call returns with a success value, it will hand that to you. If it instead returns an error, try!</code> will return from the containing function with that error. The end result means that if if load_header</code> ever has a problem when calling file.read_u32()</code>, that Error will be returned instead. If it is, the same thing happens in load_audio</code>, meaning that it, too, will return the same Error. And so on, until the calling function finally handles the error case. I wrote a post about try!</code></a>, if you'd like to learn more. Weird feature #3: the borrow-checker</h1> You know, it's funny. Many people mention the borrow-checker as the first thing about Rust. Often, they even rank it as the thing that sets Rust apart. To Andrei, this is the "hulky torso" part of Rust. To me, though, the borrow-checker is just another pass in the compiler. Just like type-checking, it helps catch a level of bugs before they happen at runtime. That's it. It might seem hefty at first, but I'd argue that's just because it's different and learning to work with it grows a new muscle as a programmer, not unlike working with a new type system. What kinds of bugs does the borrow-checker catch, you ask? Use-after-free</h2> Ah yes, the classic problem of freeing memory and then trying to use it later. Quite often, this is what makes programs crash with the dreaded "null pointer exception". There are a good handful of C++ best practices to help avoid this (using RAII, preferring references or smart pointers over raw pointers, documenting the ownership in your API docs, etc), which is likely what Andrei referred to as "large fraction of the thinking and coding are dedicated to essentially a clerical job". A team of highly trained C++ programmers would see avoiding use-after-free as largely clerical because that's about what it amounts to -- so long as you obey all the best practices, never cheat, and you always only grow the team with more experts. Invalid iterator</h2> Ever modify a container you're iterating over in C++ and then have a random crash later on? I have. If you managed to add or remove an item to the container that was enough to force memory reallocation, your iterator has become invalid. I don't often make this error, but I still do from time to time. Data races</h2> In Rust, data is either shared or mutable. If it's mutable, it can't be shared, so you don't risk mutating from two threads in a race. If it's shared, you can't mutate, so you can read to your heart's content from multiple threads. If you're coming from C++, or any other language with a set of good parallel libraries, this is going to feel overly strict to say the least. Luckily, this isn't the whole story, but instead the general basis that gives you a simple set of rules to build more abstractions on. The rest of the story is still young and forming. There are a growing number of libraries focused on concurrency</a> that you can follow if you're interested to learn more. Tracking ownership</h2> This may seem a bit redundant, but it's actually something that C++ struggles with. I alluded earlier to "documenting the ownership in your API docs". The problem is that this information is in a comment rather than the code itself. Here's a scenario: you're in C++ and you need to call a library that you didn't write. Let's say it's a C-based library and takes a raw pointer. Will it delete what you passed in? Will it take ownership of it and put it into one of its data structures? Maybe you're calling into a scripting engine like Ruby. Then who owns it? Rather than reading through docs, Rust lets you verify that your assumptions match reality when it tracks how you are using the API with the borrow-checker. And more</h2> Borrow checking helps other classes of errors. For example, it helps improve local reasoning when you're writing a function because you can safely assume that any mutable values coming in are disjoint and won't interact with each other when you change them. This incidentally also opens the door for additional optimizations because the compiler can ensure that values can never be both aliased and mutable, something that is difficult to do in C-based languages with alias analysis. There's more good info about the borrow checker and classes of errors it prevents in "You can't spell trust without Rust"</a> if you're interested. Weird feature #4: rules are meant to be broken</h1> I'd argue that one of Rust's strongest suits is that it's pragmatic. Most of Rust's stringent checks can be circumvented using features like unsafe</code>, mem::transmute</code>, and more. That's right, don't like the borrow-checker for your use case? No problem, turn it off. Rust is made to be able to express anything you can do in a C-like systems language. The advantage of Rust is that it's much easier to start with safe defaults that can be pushed out of the way when needed. It's much more difficult to build safety on top of unsafe language features. While Rust lets you pick and choose, it biases towards not letting you shoot your foot off. Did Rust miss leg day?</h1> Speaking of feet, did Rust miss leg day after all? Is it lopsided? Has it been focused on the wrong things? Rust is getting stronger, and luckily it has enough safeguards in place to know how to go into the squat without pulling its back. This part can't be understated. The philosophy behind Rust is one of having good fundamentals. As long as the good fundamentals are there, the language can grow on that foundation. But you don't have to take my word on it. Like any fix for the Blub Paradox, it comes down to programmers picking up Rust and seeing if the kinds of capabilities Rust gives them allows them to express themselves efficiently and in a way that avoids classes of bugs that distract from the problem at hand. It's natural to think that some features are ornamental when you're first coming to them. Time and exposure are the real test. Special thanks to Yehuda Katz and Jason Turner for reviewing this post and sending me their feedback Travel log (Maui) 2015-12-15T00:00:00+00:00 When I left Seattle, I jokingly told my friend "the first country I'm travelling to is Hawaii". "But it is isn't a different country," she replied. It is. Oh it is. Looking at Maui from the Kula Lodge Having travelled around the United States, I can't say there are many places even similar to Hawaii. Or, more precisely, Maui, the island I spent most of my time on this past week. To start with many of the views look absolutely, totally fake. Yes, the above picture was my view at lunch one day. Only after staying in Maui for a week did I start to get used to the fact that driving around the interior of Maui, at any moment you could turn the corner and see another breath-taking view. I don't know if I could get used to it really. Half my shots of the beach or looking out over the mountain look like they're from a postcard. Kamaole Beach III Sure, I guess someone might over time get used to the amazing mixture of volcanic rock, perfect beaches, and mountain views rounded out with gorgeous weather. Add to that easy access to a wide variety of fresh produce and fish, and you have yourself a quite literal slice of paradise. It became quickly obvious why Hawaii has its reputation. It is, quite simply, nothing short of amazing. To give it an even more foreign feel, I was greeted by a number of foods and plants I'd never heard of before. Fruit that tastes like cinnamon (complete with a gritty texture), starfruit, and even the plentyful macadamia nuts used in new ways (like in the crust on Mahi Mahi, for example). Speaking of macadamia nuts... Like I said, a variety of foods. The plants, too, came in all shapes and sizes. One of the most striking is the Rainbow Eucalyptus. Rainbow Eucalyptus at Garden of Eden The above picture is from the Garden of Eden</a> on the Road to Hana. Both the Garden and the Road to Hana are well worth exploring. The Road is a windy affair that takes you around through some of the various micro-climates as you traverse the island. My friend and I explored it one day, stopping at the Garden of Eden, a nature reserve that contained many cool varieties of plants, including the Rainbow Eucalyptus above. In addition, many varieties of bamboo, bananas, flowers of various sorts, and starfruit flourish. But you don't really have to go very far in any direction to get the island experience. Maui, as one may expect being a holiday spot for many, is a great reminder to take things a little slower. Stop and look at the flowers, enjoy the views, soak in the sun, and rest up for whatever adventure might come along tomorrow. There's also a general friendly, caring spirit that people generally strive for called the Aloha Spirit, sharing its name with the famous greeting. As a sense of goodwill, it carries through the island's inhabitants and even into governance</a>. Just as important as soaking in the sun is to soak in some of the Aloha while you're there and take it with you. I'm looking forward to returning to Hawaii and seeing what else it has to offer. Like many places, it feels like I just scratched the surface. What I found was a warm, welcoming place, both in temperature and in vibe. Highly recommended checking it out. It's well-worth the flight. Just don't rush too much, or you'll miss some of the best parts. Mahalo for reading! HandmadeCon 2015 2015-12-05T00:00:00+00:00 HandmadeCon 2015, where even the welcome banner is handmade Today, I attended HandmadeCon 2015</a>. HandmadeCon is the first convention for the Handmade Hero</a> project by Casey Muratori, a game industry gun-for-hire who most recently worked on The Witness</a>, the follow-up game for Sophia Blow, the creator of Braid</a>. Key takeaways</h1> There was a ton of material in the interviews. Enough so that I suspect if two people wrote down their takeaways, they might have wildly different lists. Still, here are the takeaways I had: Quick iteration is king. Working towards faster turnaround time is going to deeply impact overall quality.</li> Lock-in fundamental game choices as early as possible. Examples: game controls for platforming, game speed for open-world. These impact how you design levels, and any change may cause you to have to redo levels, puzzles, assets, etc.</li> To work in the game industry you need to show both self-motivated passion and curiosity. The rest can be learned.</li> </ul> Format</h1> Casey (right) interviewing Sophia Blow Casey has hand-picked a list of game developers with deep technical experience for the conference. With each developer, he interviewed them about their strengths and would tease out the key points from the discussion, often drilling in to learn more. I hadn't heard of some of the people he'd picked, but I quickly realized he'd hand-selected each for their specific expertise in gaming, whether controls, code layout, networking, design, or ingenuity. Tommy Refenes</h2> Super Meat Boy</h2> Key quote(s)</h3> "Best way to get the best work out of people is to give people a way to give you the best work"</li> </ul> Assets pipeline</h3> Original asset pipeline used JSFL to output texture files with indexing from the Adobe Flash tools</li> Now he just works with the SWF directly. He (gasp) read the SWF manuals.</li> </ul> Movement</h3> Spent three months perfecting controls using a square that jumped around the screen</li> Movement was built on a set of hand-coded equations whose parameters were tweaked until everything felt right. He offhandedly called these "garbage numbers/equations"</li> Rather than being real-world, everything was based on how it played Eg) 200ms of grace time where meat boy sticks to the wall, allowing you to press the other way and then hit the jump button, or vice versa, to initiate a wall jump</li> </ul> </li> After the initial controls tuning, they were locked in and did not change for the rest of the development</li> This allowed level design to build on the controls</li> Explored things like analog, but ended up going digital. Reasons: never want a "half jump" or a slow acceleration</li> </ul> Testing</h3> Fans are the best testers</li> PC is crazy complex, and you're bound to bake in some hardware configuration assumptions</li> He's working on crowd-sourced alpha/beta testing platform that he can ship builds out to specific testers</li> </ul> Mike Acton</h2> Insomniac [creators of Ratchet and Clank]</h2> Key quote(s)</h3> "Adding stuff to the problem does not make the problem simpler"</li> "Everytime I hear 'I can just optimize at the end' I want to slap someone" Probably a chain of failures lead to this.</li> "Capture and visualize" your data. Actually look at the data. eg) build tools to look at the spacial queries that developers are running.</li> </ul> Reaction to Tommy</h3> Mike started off with a few reactions to Tommy's interview.</li> Tommy's input style isn't garbage, it is real and it fits the input of his movements and output of "Tommy smiles"</li> Doing things that are "real world" may actually just be the wrong answer and move you the wrong direction</li> He whole-heartedly agreed about locking down controls so you can lock down how levels are made</li> On a related note: ovement speed, esp in an open-world game, greatly affect how the engine works because of streaming off disk, etc</li> Read the CPU manuals</li> BTW, docs are seriously important. Doc your own internal formats, too. This is your key input.</li> Also, focus on your own problem and your own game. Too many indie devs mimicking other frameworks/tools.</li> </ul> Hiring a game developer</h3> Main corollary for game developer success: curiosity</li> Demonstrate what you've done with your own willpower</li> Learn how to practice, and take practice seriously. Anything else is an excuse. Your career is one of the most important things in your life, up there with family.</li> Practice with the goal of getting better/faster and solving the problem. Throw it away rather than keeping it. If it's practice, it's not a project. You're honing a skill rather than building a thing.</li> </ul> Pat Wyatt</h2> Guild Wars, Diablo, StarCraft and more</h2> Key quote(s)</h3> "Make your iteration speed as fast as you can, because this will determine the quality of your game"</li> </ul> Early network stacks</h3> Synchronized network models where client and server both had game state and were kept in lock-step</li> Warcraft: RTS are not so twitchy, don't need to see effect simultaneously</li> Eg) 4 network moves per second, and a move is when you've heard from everyone</li> Starcraft similar to Warcraft</li> </ul> Diablo on Battle.Net</h3> Move from singleplayer to real-time multiplayer</li> First idea: Synchronized at first, to help prevent cheating</li> Ideally, network model should be separate from UI data, since one is updated by network, and one by the client, and they may be updated at different times</li> Problem with synchonized client/server: if you don't do it perfectly, you diverge and it's hard to debug</li> Diablo written too quickly to do a synchronized version</li> Solution: "Asynchronous loosely-coupled game"</li> Each level has a 'master', and the master is the consensus driver for world state for that level</li> Replayable log of state that can be given to a new master if the first one drops out</li> Game host declares who is master for a given level based on who first entered that level</li> State change happens in two steps: "I am attempting to pick up the sword" then master plays through all attempts and break ties</li> DON'T DO A GAME LIKE THIS. People will cheat. Griefers will take advantage</li> Ask yourself: "How will people hack this?" "How will people abuse others?"</li> </ul> </li> </ul> Guild Wars</h3> All client/server, all TCP</li> Network functionality split across many servers</li> File server: handles versions and creates deltas based on what client has available, compresses, and sends update to client for assets in the manifest</li> Asset ids automatically assigned by perforce full path</li> File servers were put near users because this helps throughput. Related: tcp window and waiting for ack for each packet</li> Lobby server: authentication and presence. Arena, instance, etc all controlled through lobby server because you don't have a persistent connection. Helps with p2p messages, guild msgs etc, encryption</li> Matchmaking server: will match up servers with capacity with needs of user</li> Using tons of servers allows you scale up and distribute across machines</li> Network code is shared across all servers. Every server is actually a generic modular server. It just hotloads the specific module. eg) lobby server just loads the lobby module</li> Need RPC protocol that supports versioning. These days, things like protocol buffers</a> would work fine.</li> </ul> UDP vs TCP</h3> Mobile - UDP works great, since you're already going to be losing packets</li> MMO - TCP is great because you don't want to lose packets</li> </ul> Network game states</h3> Failure is always legal. Eg) can't load model file, you don't crash. Instead you get a white box</li> Server is authoritative and may pull players back to acceptable state. Also, every client sees their version of the server truth, like the arrow that "bends in space" because target moved</li> </ul> Best practices</h3> Always simulate network jitter and latency, minimum 200ms, so you are always creating robust networking code</li> You will always be hacked Deeply understand your interaction model</li> Rate limiting helps prevent abuse</li> Hiding IP addresses so they can't be targetted for attacks like DDOS</li> </ul> </li> </ul> Hiring a game developer</h3> Demonstrate passion</li> Read "cracking the coding interview"</li> </ul> Sophia Blow</h2> Braid, The Witness</h2> Iteration is king. More you optimize the less you can modify.</li> Programming for fungibility to allow for faster iteration.</li> Program for a decoupled system, as much as possible. This includes game mechanics, types of puzzles, etc</li> Doing programming as a game designer, you can more closely point the best way forward in the game design as you know exactly what's possible</li> Programming as a designer also helps you explore more of the design space than you might otherwise have. Eg) in Braid the enemies unaffected by your time travel.</li> Good design can often be a search space with a seed value and things that seem like time wasted, but that's part of the journey.</li> </ul> Ron Gilbert</h2> Secret of Monkey Island, creator of SCUMM</h2> For Maniac Mansion, C64 was hooked to a Unix/Vax machine with hotloading to C64. This is how they could swap out and do fast iterations</li> To learn how to do an interpreter, he banked out BASIC ROM to RAM and then manipulated it. Cool hack.</li> For pathfinding, used dijkstra's algorithm with rectangle shapes. This evolved over time.</li> Later games used seamlessly stiched-together soundtracks that changed as the player progressed.</li> Cutscenes, which came from Lucasfilm games, were originally hand-scripted. They had to "cheat" a little so the player couldn't interrupt the cutscene with their own movement. Eg) Making the character the player was following appear far enough ahead of the player so that the player couldn't run in front and interrupt animation</li> </ul> Travel log (Blacksburg) 2015-12-03T00:00:00+00:00 This is the first of posts detailing my current trip, which will take me around southwest Virginia, Seattle, Hawaii, Australia, New Zealand, Hong Kong, and Japan. My first stop was Blacksburg, home of Virginia Tech. I lived in Blacksburg from 1996 - 2009, so it was both familiar and a little alien, as it hasn't stood still since I've been gone. Still, a great time to reconnect with friends that I haven't seen in years. Virginia Tech</h1> Torgersen Hall at Virginia Tech It was nice to take a stroll around campus and see all the familiar buildings. Even new building felt familiar with the use of Hokie Stone, a distinctive limestone taken from the local quarries. The students were just coming back from Thanksgiving break, so when I first got here there were very few people around, giving it a ghost town feeling. Not in a bad way, though. I found it peaceful and quiet. War Memorial Chapel When you're walking around the Tech campus, you can't help but notice the Burruss Hall building. It's a giant building that is both familiar and imposing. If you've seen any brochures about Virginia Tech, it's likely you've already seen it. Staring up at Burruss Hall If you step back from Burruss Hall and look down, you'll notice a ring of stones. This is the memorial for those killed in the shootings on April 16, 2007</a>. Each stone has a name of someone who was killed that day. Rather than feeling overtly sad, the memorial feels more like a place to recognize and pay respect to the lost while at the same time giving the living space to move on and go about their lives. Still, the losses are felt, and the lost remembered. Blacksburg</h1> Look out at Blackburg from the Virginia Tech library Virginia Tech is surrounded by the town of Blacksburg. It's a fairly small affair, nestled into the hills of Virginia. Downtown Blacksburg is only a few streets wide, with a few dozen restaurants and shops packed together before the streets stretch into the neighboring residential areas. Because of its relatively small size, the student population inevitably develops a fondness for certain restaurants. Some of these became institutions that would go on to serve each new batch of students. One such institution is the Bollo's cafe and bakery. Bollo's, a Blacksburg institution (with photogenic dog) It was good to see that it's still around. As an undergrad student at Virginia Tech, I spent many a night here laughing with friends, going on dates, playing games, and drinking the strongest coffee I'd ever had. Sorry Seattle, you weren't my first latte. Speaking of Seattle, which is currently being swallowed by the efficient apartment complex, Blacksburg still has lots of great houses in prime locations. The house I'm staying in is rich with character, created sometime in the early 1900s. Its heating has been modernized, but it still has the history intact. In the basement is a shower for the field workers to wash off. It was great to see my Blacksburg friends again. It felt like taking a slow bath in the familiar before rising to walk off to the unknown. A perfect way to start the trip. I can't wait to see what adventures lie ahead. Building a simple JIT in Rust 2015-12-01T00:00:00+00:00 The other day I threw together a simple Just-In-Time compiler (or JIT, for short), and I thought it'd be fun to show the steps I did. With this, it should be possible to create a page of executable memory, write some machine code into it, and then treat it like a function call from Rust. Let's get started! You can grab my JIT project</a>, if you want to follow along. I've only tested this in OS X, but it should adaptable for other platforms. Updated 12/2/2015: Thanks to some reader comments, I've updated the code below to not use malloc and instead use posix_memalign directly as the allocation step, which avoids leaking memory. Creating executable memory</h1> For our JIT, we'll need to first allocate the memory that will hold our JIT'ted code. The most important thing here is to create a block of memory that's executable so that we can later jump into it, run code, and then return back out. To do this, we need a few functions from the standard C library, which we can access through the libc</code> external module. // added to Cargo.toml [dependencies] libc = "0.2.2" // main.rs extern crate libc; </code></pre> Once we've pulled in libc, we need to allocated memory. Not just any memory, we need to allocate aligned memory. Some operating systems, like OS X, require that executable code memory starts at a particular alignment. For example, that the address is exactly a multiple of 0x1000. const PAGE_SIZE: usize = 4096; // ... unsafe { let mut page : mut libc::c_void = mem::uninitialized(); libc::posix_memalign(&mut page, PAGE_SIZE, size); } </code></pre> Once allocated, we now have memory we can safely jump to. Well, almost. Before we can run code in our newly-allocated memory, we need to enable executing code in this area of memory. unsafe { libc::mprotect(page, size, libc::PROT_EXEC | libc::PROT_READ | libc::PROT_WRITE); } </code></pre> Now we're ready. We have something we can write into and then jump into. Since running JIT code is basically "no man's land" without any safeguards, it's easy to get yourself in trouble. One step that I add is to also fill the memory block with the RET</code> instruction, which will let us return from our function even if we happen to accidentally run other memory in the block. extern { fn memset(s: mut libc::c_void, c: libc::uint32_t, n: libc::size_t) -> mut libc::c_void; } // ... unsafe { memset(page, 0xc3, size); // prepopulate with 'RET' calls (0xc3) } </code></pre> We've allocated our JIT memory, aligned it, set it as executable, and then filled it with the RET</code> instruction. We're ready to write some code into it. Note: You'll notice we don't explicitly call libc::free()</code> after we allocate. For simplicity, we treat the JIT'ed functions we create as long-lived, but a full-blown JIT would likely use better memory hygiene. Getting ready to write our first program</h1> To write our first JIT program, let's first make a way to more easily work with our memory. In its current state, it's a raw C void* pointer, but this is a bit unweildy as we don't have a lot of support to work with that in Rust. To make something that feels more natural, we'll create a new struct type that will hold our pointer and allow us to create indexing functions for easier access. This will let us access the memory like m[0] = 0x10</code>. Let's create the struct we'll use: use std::mem; struct JitMemory { contents : mut u8 } fn alloc() -> JitMemory { let contents: mut u8; unsafe { //note: allocate 'page' as before //then, transmute contents = mem::transmute(page); } JitMemory { contents: contents } } </code></pre> We've got the JitMemory struct which will hold the memory we've allocated, and we use the mem::transmute</code> call to convert our raw void pointer to a raw u8 pointer, which will be easier to work with in our next step. Now that we have the struct, let's create some indexing functions. use std::ops::{Index, IndexMut}; impl Index<usize> for JitMemory { type Output = u8; fn index(&self, _index: usize) -> &u8 { unsafe {&self.contents.offset(_index as isize) } } } impl IndexMut<usize> for JitMemory { fn index_mut(&mut self, _index: usize) -> &mut u8 { unsafe {&mut self.contents.offset(_index as isize) } } } </code></pre> With these in place, we can now more easily write instructions into memory. Before we do, let's put everything we'd done so far into a constructor for our struct: impl JitMemory { fn new(num_pages: usize) -> JitMemory { let contents : mut u8; unsafe { let size = num_pages PAGE_SIZE; let mut _contents : mut libc::c_void = mem::uninitialized(); // avoid uninitalized warning libc::posix_memalign(&mut _contents, PAGE_SIZE, size); libc::mprotect(_contents, size, libc::PROT_EXEC | libc::PROT_READ | libc::PROT_WRITE); memset(_contents, 0xc3, size); // for now, prepopulate with 'RET' contents = mem::transmute(_contents); } JitMemory { contents: contents } } } </code></pre> Writing our first JIT program</h1> With our new constructor and indexing functions in place, we can write our first JIT program. The simplest "hello world" that I use when doing JIT is a function that takes no parameters and returns a simple value. We can do this with two assembly instructions: MOV RAX, 0x3 ; move our return value (0x3) into RAX, ; the register in x64 used for return values RET ; return from the function call </code></pre> Great, now we just need to write this program into our JIT memory. But wait, we don't have an assembler :) Not to fear, there are plenty of assemblers to get us started. There are even online assemblers</a> you can use. Let's plug the first line MOV RAX, 0x3</code> into our online assembler. The top line of the result is the raw hex. This is what we want. These are the actual bits we'll be writing into memory for our function: 48C7C003000000 With these bytes, and the RET instruction we already filled our memory with, we now have our full function. We can use our indexing functions to write this out: let mut jit : JitMemory = JitMemory::new(1); // allocate a page of memory jit[0] = 0x48; // mov RAX, 0x3 jit[1] = 0xc7; jit[2] = 0xc0; jit[3] = 0x03; jit[4] = 0x00; jit[5] = 0x00; jit[6] = 0x00; </code></pre> Turning our memory into a function</h1> We have an executable block of memory with the code for our function filled in. The last step is to turn this into a Rust function we can call. We do this by doing another mem::transmute: fn run_jit() -> (fn() -> i64) { let mut jit : JitMemory = JitMemory::new(1); jit[0] = 0x48; // mov RAX, 0x3 jit[1] = 0xc7; jit[2] = 0xc0; jit[3] = 0x03; jit[4] = 0x00; jit[5] = 0x00; jit[6] = 0x00; unsafe { mem::transmute(jit.contents) } } </code></pre> And with that, after a few handfuls of unsafe calls and transmutes, we have our function. The only thing left is to call it: fn main() { let fun = run_jit(); println!("{}", fun()); } </code></pre> Debugging</h1> When working with a JIT, it's inevitable at some point something will go wrong, and we'll need to dive into the debugger. Luckily, Rust works with LLDB out of the box. Here are some helpful LLDB commands to get you started: Start the JIT in LLDB: lldb target/debug/rustyjit</code> Set a breakpoint for the line that contains the call into our JIT code: > br set -line 64</code> Run to our breakpoint: > run</code> Now that we're about to run our JIT code, there are a few things we can do. The first one is to verify the address of our code: > p fun</code> Unfortunately, that's not the most helpful thing in the world. What we often want is look at the memory behind the function: > mem read fun</code> This gives us back something like this: 0x100804000: 48 c7 c0 03 00 00 00 c3 c3 c3 c3 c3 c3 c3 c3 c3 H??....????????? 0x100804010: c3 c3 c3 c3 c3 c3 c3 c3 c3 c3 c3 c3 c3 c3 c3 c3 ???????????????? </code></pre> That's better. Now we can see if the bytes we expect to be there are in fact there. If we wanted, we could update memory using mem write</code> if something was out of place. We can also use the built in disassembler to read back out what the asm instructions are for our function: > dis -A x64 -s fun</code> Finally, we can continue with the program: > cont</code> Lots ahead</h1> Now that we have a tiny JIT and a way to debug it, the sky is the limit. Turning source code into machine code is the heart of any compiler, and with these few additional steps that compiler could be made to output code that we can run directly. Porting a C# NES emulator to Rust 2015-11-20T00:00:00+00:00 Many years ago, I created a simple NES emulator in C#</a> using SDL and Mono. For fun, I wanted to see what it would be like to port that emulator to Rust. A couple weeks later, after poking on it on and off between packing for a move, I was able to get it working</a>. What I did</h1> Loading a binary file</h2> Loading the types of files commonly used in emulators requires us to work with binary files. To do so, we need a way of loading the raw data both as just plain bytes and also as numbers. Rust, by default, comes with a fairly slim API</a> for dealing with binary data. No matter, Rust is also very flexible and allows you to build on additional parts to the API to suit your needs. I started there, and built out a few functions that would let me read in the numbers from the file both using big-endian and little-endian. pub trait BitReader { fn read_u32_be(&mut self) -> Result<u32, io::Error>; fn read_u32_le(&mut self) -> Result<u32, io::Error>; } impl BitReader for File { fn read_u32_be(&mut self) -> Result<u32, io::Error> { let mut buffer = [0; 4]; try!(self.read(&mut buffer)); Ok(buffer[3] as u32 + ((buffer[2] as u32) << 8) + ((buffer[1] as u32) << 16) + ((buffer[0] as u32) << 24)) } fn read_u32_le(&mut self) -> Result<u32, io::Error> { let mut buffer = [0; 4]; try!(self.read(&mut buffer)); Ok(buffer[0] as u32 + ((buffer[1] as u32) << 8) + ((buffer[2] as u32) << 16) + ((buffer[3] as u32) << 24)) } } </code></pre> Once I had the trait, I could import it where I wanted to use these extra read capabilities on File. With these extra methods in place, I could plow through the binary file loading with a mix of the File read functionality and the BitReader trait. I did notice I used the odd concoction buffer.iter().cloned().collect()</code> to convert from an array to a Vec of the same contained type: for _ in 0..(num_prg_pages4) { let mut buffer = [0; 4096]; try!(f.read(&mut buffer)); prg_rom.push(buffer.iter().cloned().collect()); } </code></pre> There may be a more Rust-native way of doing it. Personally, I think it'd be nice to be able to read directly into a Vec, so you could elide that step. Side-by-side porting</h2> One of the first things I did after I had file loading in place was to start porting over the CPU emulation. Because I wanted the code to be fairy clean, I opted to not do a literal port. Instead, I opened two separate editor windows: one for the original C# source, and one for the new Rust source. Tedious, but I hoped that a port focused on what would fit better with Rust would be less error-prone. There's another reason not to do copy/pasting of code when porting: too many little things needed to be changed to the Rust style. For example, Rust-friendly names often differ from the C# equivalent (eg the ReadChrRom method would need to be read_chr_rom to match Rust naming conventions). Like-wise, little things that help readability in Rust, like the "no parens around if guards" means that copy/pasted code would need to be touched up. All told, a minor frustration, but it gave ample time to make the code look a bit cleaner in the transfer so likely a net win. Feeling more comfortable with Rust's use of expressions, I could write code in a more dense way. For example, here is some code from the ROR or Rotate Right opcode in C#: if ((valueholder & 0x1) == 0x1) bitholder = 1; else bitholder = 0; valueholder = (byte)(valueholder >> 1); if (carry_flag == 1) valueholder = (byte)(valueholder | 0x80); carry_flag = bitholder; if ((valueholder & 0xff) == 0x0) zero_flag = 1; else zero_flag = 0; if ((valueholder & 0x80) == 0x80) sign_flag = 1; else sign_flag = 0; </code></pre> and here's what it became in Rust: let bit = (value & 0x1) == 0x1; value = (value >> 1) & 0x7f; value += if self.carry {0x80} else {0}; self.carry = bit; self.zero = value == 0; self.sign = (value & 0x80) == 0x80; </code></pre> Some of these changes could have been done in the original C#, but I didn't notice them at the time. Still, I think the result Rust is still pretty readable, even though it's more compact. While we're talking about bit rotation, one of the annoyances I experienced working on bit manipulation code is that Rust is more vigilant about overflow checking. In hindsight, this is actually a good thing, but coming from languages like C# and JavaScript that are less vigilant, it was a bit of a headache to adjust to. I ended up doing some of my own bit masks so I didn't have to worry about it, but I noticed that they've added a number of "wrapping" operations in Rust 1.4</a> that allow you to work around the overflow checks. Hitting the first ownership error</h2> A few days into the port, I hit my first ownership error. Here's a quick rundown of the NES, so you can see why that might be so: CPU: The CPU :)</li> PPU: The NES's graphics unit</li> MMU: Not technically not called the MMU, but let's call it an MMU for our purposes</li> Cart: The program to run</li> </ul> Now that we have the parts, here's how they talk to each other: CPU asks the MMU for an instruction, and the CPU may also write back to memory</li> PPU reads memory from the Cart, and can also transfer memory from MMU to graphics memory</li> MMU needs to read information from the Cart and can also read and write from the PPU</li> </ul> All the subsystems, in essence, need to be able to talk to each other at some level. Which is exactly how I set up the C# project. Singleton objects with references to each other. When I ported over to TypeScript a few years ago, similar idea. But in Rust, ownership needs to be clear. If you call a method, you can't also pass something you own to that method, like so: foo.x.y(foo.x.z) </code></pre> The net result is that I went back to the drawing board and thought long and hard about what was actually necessary for each component. After a couple of refactorings, I think I landed on a solution that's both cleaner and clearer. Every object has one owner, and I don't do any ref cells or ref counting. Deriving debug</h2> There are a few tricks that I loved in Haskell that I wished other languages would pick up. One of them is the ability to automatically be able to print out objects for you. In Rust, you can do this using the 'derive' command: #[derive(Debug)] struct Foo { x: u32 } //... let foo = Foo { x: 10 } println!("{:?}", foo); </code></pre> By deriving Debug, you have the ability to output a debug format of that structure, which is output using {:?}. This is very handy to help bootstrap a project, when you quickly want to look at the contents of what is working so far. After a while, I found I ended up removing these helpers and putting in my own, customized implementations. SDL</h2> Using SDL was dead simple. I used the SDL2 Rust bindings</a>. After a few tries to follow the directions to install SDL2 carefully, I had it working in both Windows and OS X. After it's installed, the bindings "just work". I'd never used SDL2 before, but after looking through the examples directory</a>, especially the texture sample, I was good to go. What I didn't do</h1> Chase crashing bugs</h2> After wrestling a few ownership errors, I largely spent my time on correctness. I ended up having a few hard-to-find issues where I flipped the boolean logic on a check or discovering I forgot to implement a piece of the machine that was required. After many hours pouring over the code side by side, or looking at the output log, I finally managed to get the outputs similar. Just as in the C# and JavaScript versions of the emulator, I didn't worry about any crashes. A few times I had an arithmetic overflow or error accessing a vector out of bounds, but those were easy to fix. Optimize</h2> After I got the emulator working, I let it rip. The result was something unplayably fast. Not bad for a first try. In hindsight, this isn't too surprising. The CPU gets to work directly with the MMU. Ditto with the MMU and the PPU. The systems can call functions in each other without any abstraction, VM, or GC. By default, the core structs sit on the stack. Using Rust had nudged me into doing what would have made sense for a decent C-based approach, without the risks. Of course, it's not like Rust prevents you from adding in a hefty abstraction layer, but the defaults seem to push you in the right direction. Unit test</h2> For this to feel like a real Rust project, though, it needs some tests. I asked Patrick Walton</a>, the author of probably the first NES emulator for Rust</a>, about testing emulators with Rust unit tests, but he didn't use them for his project either. Still, I suspect there's a way of creating fairly testable, and still performant, parts of the emulator. For example, opcodes could each have their own unit test if there was a way, possibly with traits, to decouple them from working so closely with the MMU. Add audio support</h2> In Patrick's video about his emulator</a>, he mentions that he took it on to show that Rust can do real-time processing. For an emulator, that means that the audio is emulated properly and sounds correct. I haven't added any audio support, but it would be a fun project at some point. Precisely emulate the machine</h2> The never-ending rabbit hole for any emulation project is to see just how precisely you can emulate the hardware. There's plenty of work here. What I've learned</h1> I'm getting a little more comfortable with Rust. Thinking through an ownership refactoring can still take a while, but it's starting to feel like the hurdle you take on so that you can worry less about safety and performance. I still have a ways to go before it becomes second nature. Cargo shines again, and the SDL bindings were easy to put in place and worked as expected. No complaints there. Feels like I'm still scratching the surface. There are likely some slick spots to use traits and macros to cut down on the redundancy and coupling, repectively. But I'm happy with the progress. Steam Machines enter the market 2015-11-11T00:00:00+00:00 This week marks the release of the first official Steam Machines</a>. For the last five months, I've been a proud owner of the Alienware Alpha</a>, one of the three Steam Machine types now available. Knowing what could be possible, I've been curiously awaiting the official launch. The plan</h1> Valve's motivation for creating the Steam Machine seems pretty straight forward: move to the TV and compete more directly with game consoles. Currently, that space is shared by the big three: Sony, Microsoft, and Nintendo. Why not make that the big four instead, with Valve in the mix? To get there, Valve has started with three initial devices. The Steam Controller is a gamepad with two round touchpads and a set of traditional gamepad buttons. It can be configured to work as a normal gamepad for games with gamepad support. Alternatively, the touchpads can be used just as touchpads to play games that work with a mouse instead of gamepad. The Steam Link is something of a gaming bridge between the PC and your TV, allowing you to stream games from your PC over to your TV. At around $50, it's pretty inexpensive. Alienware Steam Machine Lastly, the Steam Machine. The Steam Machine itself isn't actually a thing. Rather, it's a PC specification that hardware vendors can build to. Out of the gate, three hardware vendors have already produced a Steam Machine: Alienware (pictured above), Zotac, and Syber. Steam Machines also come with a new operating system called SteamOS</a>, a form of Linux that's been optimized for gaming. The right plan?</h1> Now that we know the plan, is that plan the right plan? Specifically, does the Steam Machine have a home? It's clear to me that a device like the Steam Machine has value in the market. It's smaller than a gaming console, more flexible, and just as powerful. I've spent countless hours gaming, reading news, watching movies, and even programming on mine. I've even stopped using my Playstation 4, opting instead to do all my gaming on the Alpha. But as I look at the Steam Machine release this week, I have some concerns. Concern 1: No flagship Steam Machine</h2> The console market is notorious for razor-thin margins on release hardware. Traditionally, consoles are often sold as loss leaders in their early years. Meaning, they are sold below the manufacturing costs</a>. The thought here is the licensing profits from the games bought by initial adopters will help offset the loss. Once the console becomes embedded in the market, manufacturing improves, and over many, many small iterations, the console itself becomes cheap enough to make a profit. Consoles are always, with rare exception, first-party. This means that the company who built the software stack that the games run on is also the company who owns and sells the hardware. Nintendo, Microsoft, Sony, and even previous game vendors like Atari and Sega, all made first-party consoles. Rather than being first-party, Valve has opted for an approach that's closer to non-console markets like phones: third-party hardware. Early software plays like Microsoft with MS-DOS and Windows, and more recent ones like Google's Android platform are some examples here. The gist is that the standard platform works against a standard hardware spec. It doesn't really matter who the vendor of the hardware is, so long as it matches the spec. That might look good on paper for Valve. Since they're new into the hardware game, why not let the vendors who know what they're doing do it? It's still a risky play, and, I'd argue, one that's potentially riskier than partnering with one vendor to create a Steam-branded first-party machine. Some reasons why: Third-party vendors can't afford to sell loss leader machines. This means the machines will necessarily be lower-powered than they could be. This narrows the performance per $ gap between this batch of Steam Machines and established consoles like Playstation 4 and Xbox One. Which, in turn, hits customers in the wallet. While the Steam Machines are cool pieces of technology, they are also spendy pieces of technology. For example, the Zotac machine is $900 at the time of writing. At that price, I could buy 3 WiiUs -or- an Xbox One and a Playstation 4, with money left over for games. Third-party purchase experience is going to be whatever that vendor makes available. For example, this morning when looking at the new Alienware machine I got website errors on the Dell.com site. Of course, some of this will get better over time, but it's difficult to see how both a higher price tag and a worse purchasing experience is going to help the Steam Machine brand. As an aside, you might think that being third-party would help you create the Steam Machine of your dreams, but as of the time of this writing, you can't even configure an Alienware machine to use SSD. For a gaming machine. No SSD. What? If you're like me, that's the first thing you'll want to upgrade. I asked the Alienware rep over Twitter about it and got: Again, likely something that will be fixed in time, but not the best initial experience and one that will turn-off some gamers from the Steam Machine idea. Lastly, Valve's Steam brand is already better than the third-party vendor brands. Steam is the major name in PC gaming today. Period. Who is Zotac or Syber? I honestly have never heard of them. Why play second fiddle to brands that aren't as strong as your own? Concern 2: SteamOS</h2> The motivations here also seem straightforward: Valve gets to create a platform that a) they control and can tweak to their hearts content, b) supports a standard set of APIs that other vendors can leverage and is possibly less of a moving target than the competing APIs, and c) is free. With c), Valve helps ensure that vendors can cut costs and deliver less-expensive Steam Machines. But the whole play here is very confusing. Remember that a Steam Machine is a whole PC. Unlike a game console like Playstation 4, these guys can be a full Windows 10 machine if you install Windows on them. And Windows already runs Steam. So, in fact, a Steam Machine with SteamOS is actually a less powerful experience than taking SteamOS off and putting Windows in its place. As for the advantages, again it seems like Valve is trying to take a page from the Android playbook. Unfortunately, the current market is decidedly different than the market Android found itself in. With the release of the iPhone, and its rise in the smartphone market, you had a gap created with a lot of phone vendors wanting an iPhone-like experience without an easy way to get it. By creating a comparable work-alike, many vendors could sign up and build out the Android ecosystem. Is SteamOS entering a market of hardware vendors looking to enter the game console market? Having only one major PC brand, Dell, signing on to ship hardware with SteamOS is an indicator this may not be the case. Likewise, are game developers scrambling to get onto SteamOS? Guessing by the size of the game lists, SteamOS has about a fifth the number of titles available for Steam on Windows. That, by itself, is actually pretty impressive. But, consoles really live and die on the titles themselves. Let's do a quick rundown of my favorite kind of game, RPGs: Witcher III: no</li> Final Fantasy XIV - A Realm Reborn: no</li> Dark Souls II: no</li> </ul> In fact, I struggled to find triple-A RPGs from the last few years that run on SteamOS. What's telling is that all three are available on Windows as well on consoles, where they use gamepad support. In theory, this should be a solid group of games to show off your powerful new platform. In fact, Fallout 4, which was also released this week, does not have SteamOS support. Likely to be one of the biggest games of the year, and it won't run on Valve's newly-shipping machines. I suspect that the tech-savvy buyers will do exactly what I did when I got my machine: take out the hard drive, put in an SSD, and install Windows. Concern 3: Partnerships, or lack thereof</h2> Concerns 1 & 2 above could be summarized as "Steam Machine lacks partnerships". The more high profile hardware companies competing to create the best Steam Machine, the better the end result. The more high profile game companies making games for the Steam Machine, the more SteamOS feels like an established platform like Playstation 4 and Xbox One. Partnerships are a catch-22. It's difficult to get them without first showing that your platform is a big enough draw to support the investment it takes to port a game or build a machine. Without the partnerships, the platform never grows to that size. Road ahead</h1> It's too early to say whether or not Valve will be successful. This release of the Steam Machine is muddied by a confusing story and a lack of partnerships, but there is still time to improve the experience. When Steam first came out, it had few games and many thought it was a weak idea. It, of course, grew to the giant it is today. Like building a game distribution mechanism like Steam, a game console is also a long play. It takes years and lots of cash to build up a brand in the console market. Microsoft famously rumored to have spent north of a billion dollars to establish the first Xbox</a>. From outward appearances, Valve is trying to compete with the console on a fraction of that budget. Will Valve and its partners have the patience to build their own niche? Time will tell. Learning to 'try!' things in Rust 2015-11-04T00:00:00+00:00 I started learning Rust in earnest a few weeks ago. Coming from C++, a fair share of the idioms felt right at home. There was clear memory management and an eye towards lightweight abstractions. But when I started looking around for how to do exceptions, I was surprised to find that Rust had none. In its place were types like this: fn foo_maker() -> Result<Foo, io::Error> { ...} </code></pre> I recognized this style, having come from playing with languages like Haskell, but despite doing some functional programming before I wasn't entirely sure how to approach it. At first blush, the Result value seems pretty straightforward, it either returns something of Foo or an io::Error. We know which one it returns by pattern matching out the value: let result = foo_maker(); match result { Ok(foo) -> foo, Err(err) -> println!("Error: {:?}", err) // Ooops, we hit an error, tell the user } </code></pre> Great, you might think, that's cute. Well, at least at first. Real code tends to have a few reads in a row: let num_chans = read_u16_le(&mut f); let samples_per_sec = read_u32_le(&mut f); let avg_bytes_per_sec = read_u32_le(&mut f); let block_align = read_u16_le(&mut f); let bits_per_sample = read_u16_le(&mut f); </code></pre> Since we're reading from a file, any of these reads could hit an error. Perhaps the hard drive dies, or we prematurely hit the end of the file. Because of that, each function returns a Result that we later have to pattern match to get the value out. This means we have to pattern match every result. Yuck. If you poke around on the internets, you'll find a sneaky way around this using .unwrap()</code>. The .unwrap()</code> call does just what it name implies: it unwraps the Result and gives us the value, assuming the result is Ok. What if it errored with an Err? Well, then the application panics. let num_chans = read_u16_le(&mut f).unwrap(); let samples_per_sec = read_u32_le(&mut f).unwrap(); let avg_bytes_per_sec = read_u32_le(&mut f).unwrap(); let block_align = read_u16_le(&mut f).unwrap(); let bits_per_sample = read_u16_le(&mut f).unwrap(); </code></pre> If you felt a cold shiver run down your spine looking at that, good on you :). Now, let's talk about how we actually fix this code. Do or do not. There is a try!</h1> Looking a bit further in Rust examples, I learned about another way to call functions using the try!</code> macro. Maybe, I thought, try!</code> would save us. So I tried it out: use std::io; use std::fs::File; fn main() { let mut f = try!(File::open("foo.txt")); } </code></pre> ...and BOOM <std macros>:5:8: 6:42 error: mismatched types: expected `()`, found `core::result::Result<_, _>` (expected (), found enum `core::result::Result`) [E0308] <std macros>:5 return $ crate:: result:: Result:: Err ( <std macros>:6 $ crate:: convert:: From:: from ( err ) ) } } ) read_a_file.rs:5:17: 5:44 note: in this expansion of try! (defined in <std macros>) <std macros>:5:8: 6:42 help: run `rustc --explain E0308` to see a detailed explanation error: aborting due to previous error </code></pre> Okay, seriously? I feel like I jumped in the Haskell pool after drinking a C++ template error cocktail. What is that? Luckily after asking around, I was finally able to piece together what was going wrong. The hint lies in these lines: expected `()`, found `core::result::Result<_, _>` </code></pre> Ah, some type mismatch is happening. Our main function doesn't return anything, which is the ()</code> in the 'expected' above, but try!</code> only works inside functions that return a Result</code>. This might be the first time in my life I've ever seen a function in a C-style language that was callable only inside functions of a particular type. Shouldn't be too hard to fix, just put it in a function that returns Result, right? use std::io; use std::fs::File; fn read_file() -> Result<(), io::Error> { let mut f = try!(File::open("foo.txt")); Ok(()) } fn main() { read_file(); } </code></pre> Yup, that did it. Then I promptly closed my editor and walked away. I mean, this thing just felt alien. Unnatural. Gross. What I didn't realize was that a few weeks later I'd grok the genius in this approach. Genius #1 - Unlike try/catch, try! is precise</h2> Let's look at a real-life example from a project I'm working on: fn load_data(fname: &String) -> Result<Data, io::Error> { let mut f = try!(File::open(fname)); let num_pages = try!(f.read_u8()) * 4; let mut pages : Vec<Vec<u8>> = Vec::new(); for _ in 0..num_pages { let mut buffer = [0; 4096]; try!(f.read(&mut buffer)); pages.push(buffer.iter().cloned().collect()); } // ... } </code></pre> It may take your eyes a minute to adjust to the new syntax, but knowing what we know, we can look at the code above and know exactly which functions might give us errors. The try!()</code> call even works on calls that are part of an expression, like try!(f.read_u8()) * 4</code>. Compare this to traditional try/catch blocks and how they tend to get put around whole sections of code when in reality only parts of the code blocks may throw exceptions. The end result is code that's very clear and doesn't muddy exceptional code and value-based code together. Genius #2 - Feels like exceptions</h2> With this little bit of syntax, we get something that is value-based, but still feels like exceptions. Let's say your task is to write code that will load a variety of audio file formats. In each audio format loader, you have various subsections that get loaded (eg, a header, a metadata chunk, a chunk of audio data, etc). It'd be great if you could create specialized functions for each. If there are any errors, you just want to bubble that back out to the original caller loading the file. The resulting code may look something like: fn load_header(file: &mut File) -> Result<Header, io::Error> { Ok(Header { header_block: try!(file.read_u32()) }) } fn load_metadata(file: &mut File) -> Result<Metadata, io::Error> { Ok(Metadata { metadata_block: try!(file.read_u32()) }) } fn load_audio(file: &mut File) -> Result<Audio, io::Error> { let header = try!(load_header(file)); let metadata = try!(load_metadata(file)); Ok(Audio { header: header, metadata: metadata }) } </code></pre> We can refactor our file loading into helper functions, and we continue to use the same familiar try!</code> pattern. How does this work? The trick is that all the Results agree that io::Error</code> is the error case. If one of my helper functions hits a bad read and has to return an error, that error naturally bubbles all the way out without calling any additional code. The end result is that we have precise syntax that is transparent about errors at every layer, stays pretty lightweight, and gives us the familiar "bubble out" feel of languages with exceptions. Trade-offs</h1> Of course, there are trade-offs. We end up repeating try!()</code> in places we might have just wrapped in one big try/catch block. For example, the fix to our earlier use case might look like this: let num_chans = try!(read_u16_le(&mut f)); let samples_per_sec = try!(read_u32_le(&mut f)); let avg_bytes_per_sec = try!(read_u32_le(&mut f)); let block_align = try!(read_u16_le(&mut f)); let bits_per_sample = try!(read_u16_le(&mut f)); </code></pre> Personally, I'm happy making the trade-off for the added precision. We also saw another trade-off earlier. If you use try!()</code> in a function that doesn't return a Result, you get an incomprehensible error message. But once you know what's going on, you can use it with confidence. Telling the truth in code, function signatures and transparency 2015-10-28T00:00:00+00:00 When people talk about reasons to use a type system, there are always a couple usual suspects: error checking, documentation, and if you're an IDE person like me, tooling/editor tricks that use type information, like auto-complete. But there's another aspect to the types of function signatures that's related to documentation but is perhaps stands on its own: transparency. What is transparency?</h1> For this post, transparency helps us answer a few questions: What does this function require?</li> What does this function return?</li> What is this function going to do with what I give it? Can it modify it?</li> Can it delete it?</li> </ul> </li> Can this function fail? If so, how?</li> Does this function do anything besides use its value? ie) does this function "change the world" in some way?</li> </ul> </li> </ul> Why is transparency important?</h1> A way most of us can relate to transparency is in the relationships you have in your life. When you're not transparent with another person -- keeping all of your thoughts to yourself, never saying when you're angry, etc -- how does the other person ever know how to react to you? Transparency in code works similarly. When a function is transparent, you know what you need to do when the function is called, and you know how to react when the function is called. Let's look at a few examples. In each example, we're going to talk about a function that takes in an employee name and writes that name to a file. Examples Python</h2> def writeToFile(name): # write out to the file </code></pre> We'll start with the simplest form of transparency: minimal transparency. Looking at what we have it's tempting to say we don't know anything at all, but that's not quite true. We do know the function name and that it takes a single parameter. Things we don't know: What kind of thing does it take?</li> Can it modify the thing we give it?</li> Can it delete what we give it?</li> Does it return anything?</li> Does it throw any exceptions?</li> Does it have any permanent effect on our system eg) send files over the network, add to a database, launch the missiles?</li> </ul> </li> </ul> Without this knowledge, we resort to aggressively testing, commenting, and using naming and coding conventions to get the idea across. C</h2> int writeToFile(char name) { / write out to the file / } </code></pre> Moving into C, we start to see a bit more information popping out. We know that it takes in a char</code> for the name. Since this also isn't const</code>, we know that it's allowed to change the contents of the string we pass in. We also know it returns an int</code>, likely for error codes. We are starting to see into what the function can do, but there's still quite a bit we don't know: Can this function change our system?</li> Can the name</code> value be deleted?</li> Can the function raise a signal (a C-flavored way of exceptions/events)?</li> And what about those error codes? How do we know the int</code> is for error codes, and not returning the length of data written?</li> If it is for error codes, how do we know which ones it supports?</li> </ul> </li> </ul> We've improved some, but we still lean heavily on conventions, code comments, and testing to ensure that this function behaves correctly. As an aside: one interesting point of C is that it doesn't have exceptions, like many languages, so you're almost encouraged to have a more value-based programming style, though it lacks many common features that help fill out the value-based story (pattern matching, algebraic data types, etc). Rust</h2> fn writeToFile (name: &str) -> Result<()> { // write out to the file } </code></pre> Above is a simple Rust version. If you've never seen Rust before, let's quickly break down it says: The fn</code> keyword creates our function</li> Next, we give the function a name: writeToFile</li> Our function gets one argument called 'name' which has the type &str</code> &str</code> is a type in two parts: str</code> is the simple string type</li> &</code> meaning that this value is being borrowed and is owned by someone else (aka you're not allowed to delete it)</li> </ul> </li> </ul> </li> Finally, we return something of type Result<()></code> Result<()></code> is the Rust way of saying I can return either: a) successfully, without a value (the () above)</li> b) an error</li> </ul> </li> </ul> </li> </ul> The Rust example tells a bit more of the story than the C version. In Rust, everything is immutable by default, so we know from looking at the signature that what we pass in can't be changed. It also won't be deleted, because what is passed is 'borrowed' rather than us giving ownership over to this function. Rust is similar to C in that we don't have exceptions and instead use return values to denote success or failure (read more in Rust Error Handling). Rust encodes the success or failure in the return value, requiring the caller to code defensively around possible errors. This has the effect of encouraging code to be more robust by handling both success and failure where they can occur. An improvement over where we were in C, but we still have a lingering question: does this function have any permanent effect on the system? To be able to answer that question, we go one example deeper down the rabbit hole. Haskell</h2> writeToFile :: String -> IO () writeToFile name = do </code></pre> Since this might be unfamiliar let's take a second to breakdown what we're seeing: We start by declaring the type (which isn't terribly dissimilar from a forward declaration in C)</li> The type has two parts: We take one argument that has a type String</code></li> We return an IO ()</code> [see below]</li> </ul> </li> After the type, we write out the actual implementation of the function</li> Just as in Rust, Haskell is immutable by default. When we say we pass in a String, we're also implying that this string can't be changed or deleted.</li> </ul> Also similar to Rust, in Haskell there are no exceptions, so you must handle all possible values when you call the function. By saying IO ()</code>, this might at first seem similar to the Result<()></code>. In some sense, it is, in that IO carries along any possible errors. It also goes a bit further. In essence, it says: "I'm going to go change the world, and when I come back you need to carry that truth on." In Haskell, if you call a function that performs I/O like our writeToFile function, you must also carry on that fact in your own return type. You have to be fully transparent all the way through the call stack of who is changing the world, even indirectly. This lets you clearly document who is modifying the world and who isn't. Dark side of transparency</h1> But, it can't be all roses, right? Let's look at what I see as the dark side of transparency: encapsulation (or its lack thereof). As good software engineers, we know an important piece of building a system is having good encapsulation. Any module or object worth its weight exposes only necessary functionality to the world and hides the rest. This encourages loose coupling, so that components don't depend on each others' implementation details. In a way, transparency in the type runs counter to information hiding in that it encourages more coupling in the code. Some of this is natural. For example, in the C code, we have to remember what the return values are. If we add a new one, we may have to change the code handling the call as well. Likewise, in Python, if we change the types of parameters we allow, we may have to go update the callers. On the other hand, knowing that a function may fail may be sufficient. In the Rust case, code that handles the call may need to be updated if you change how the function fails, even if it only cares about if it fails. Likewise, in Haskell you may want to drop some probe into your system to catch the result of a function while you're debugging. Such changes can have a ripple effect in your codebase as this probe may change a series of function calls to reflect that I/O has been performed. (Not withstanding workarounds with fun names like unsafePerformIO</code>) All a balance</h1> Types are there to help. Leaning on them not enough means encoding more information in your tests and code comments. Leaning on them too much may mean tightly coupling parts of your system. What the right balance is may change from project to project, or developer to developer. Just as in our relationships with other people, it makes sense to not tell everyone we meet everything about us. It's a balance. Finding the right amount of transparency that encourages flexible, resilient systems is an art. In the last few weeks, in playing with Rust more I've grown to appreciate its pragmatic set of trade-offs. Yes, you don't get to know if the system was changed, but if something can impact the rest of the program, you're encouraged to handle it. Not that there aren't workarounds or ways to easily opt-out, but just as in life, it's all a balance. Lessons from the first 12 Euler problems in Rust 2015-10-09T00:00:00+00:00 !! Spoiler Alert !! I'm going to be talking about solutions to select problems from the first 12 Euler problems. If you haven't solved these for yourself yet, I highly recommend taking the time to solve them on your own first. Background</h1> Before I went to Microsoft for a stint in JavaScript</a>, I did my fair share of systems programming in C++. I've built compilers, head-tracking software, signal processing software, and emulators and even did some of my first programming language projects in it and for it</a>. So when word got out four years ago about a new systems language called Rust</a>, I was intrigued. I mean, it was new and it was called rust. Obviously there was some cheekiness involved :) While I've dabbled with it before, this week was the first time I really gave it some dedicated time to learn it. TL;DR: hey, this is kinda neat! In this post I talk through solving some of the Euler problems in Rust, both de-rusting my programming skills and learning what this Rust thing was all about. If you're interested, you can grab the source code to my solutions</a>. Problem 1</h1> "sum of all multiples of 3 or 5 below 1000" fn main() { let mut answer = 0; for x in 0..1000 { if x % 3 == 0 { answer += x; } else if x % 5 == 0 { answer += x; } } println!("{}", answer); } </code></pre> My solution to problem 1 is pretty straightforward, but you can already start to see some of the characteristic earmarks of Rust. On line 2, there's a let keyword which creates a variable. After that is the mut keyword for saying that the variable we're creating is mutable, as variables are immutable by default. Line 4 shows off using an iterator. Rust's iterators are lazy, meaning they generate values only as needed. This gives them some cool properties, which we'll see later, but it has the property of being relatively cheap to iterate over them, since we don't generate a full range before iteration starts. Line 5 reminds us that we're not in C. The if doesn't require parens around its condition. Indeed, if you put them there, the compiler warns you that you don't need them. Line 13 shows how to print out a value. Note the '!' after println. If you're like me, you're bound to leave this off once or twice and get an incomprehensible error message. Similar to C and many other languages, you pass println! a format string. Here, we just use "{}". The {} part means to use a default output of the argument we pass in, which in our case just writes out the number. Problem 2</h1> "sum of even fibonacci below 4,000,000" struct Fibonacci { curr: u64, next: u64, } // Implement 'Iterator' for 'Fibonacci' impl Iterator for Fibonacci { type Item = u64; // The 'Iterator' trait only requires the 'next' method to be defined. The // return type is 'Option<T>', 'None' is returned when the 'Iterator' is // over, otherwise the next value is returned wrapped in 'Some' fn next(&mut self) -> Option<u64> { let new_next = self.curr + self.next; self.curr = self.next; self.next = new_next; // 'Some' is always returned, this is an infinite value generator Some(self.curr) } } // Returns a fibonacci sequence generator fn fibonacci() -> Fibonacci { Fibonacci { curr: 1, next: 1 } } fn main() { let mut sum:u64 = 0; for i in fibonacci().take_while(|&x| x < 4000000).filter(|&x| x % 2 == 0) { sum += i; } println!("sum: {}", sum); } </code></pre> Lest you think that I somehow jumped from beginner level to advanced in a few minutes, rest assured that I used the tried and true method of the copy/paste</a>. In trying to find out how to make an iterator so I could iterate over a fibonacci sequence, I found the source to do just that. Ah, the internet. There's a fair bit going on here, so let's break it down by line. Lines 1-4: Create a simple struct to hold the current value to be returned as well as the next value. Fibonacci needs at least these two values to continue to calculate each iteration. Line 7: Now Rust is showing off the fancy. What is going on here? In Rust, a type, like our Fibonacci struct from lines 1-4, can implement a trait. What's a trait? Think of it as a small contract with the compiler. If you can show how your type satisfies all the requirements of the contract, then wherever that trait is required, the compiler knows how to use your type. In this example, we want to use our type in the for..in</code> language feature. In some languages, this wouldn't be possible. In Rust, if we can satisfy the Iterator trait, the compiler has enough information to allow that type to be used in the for..in</code> feature. Line 8: Ah the problem with copy/paste. As I started writing this blog post, I had no idea what that line was doing. Docs to the rescue. It turns out that this line is saying what the 'associated type' is. With some traits, there's going to be both a) the type that is implementing the trait (here: our Fibonacci struct) and b) an additional type that needs to be known to get the rest of the story. For us, since we're creating an iterator over the fibonacci sequence, we need to let Rust know what the type of each "turn of the crank" on the iterator will output. Each fibonacci number will be a 64-bit unsigned integer, so we use the Rust short-hand type u64</code>. Lines 12-20: Here's where we describe how to turn the crank for each step of our iterator. We calculate what the next turn will output and then return the current value. This allows us to keep one in reserve ready to be used to calculate the next value, and so on. You can see on line 19 that instead of just returning a u64</code> directly, we instead say that it's optionally a u64</code>. While in our case we iterate forever, using Option</code> here gives us a way to shut off the valve and finish the iterator if we ever wanted to. Notice, too, on line 19 the value is said last and the return statement is elided. This is just shorthand for returning a value from the function. Lines 24-26: Create a simple function to give us our starting value to begin our fibonacci sequence. You can see a couple Rust-isms here, too. As before, a value by itself is an implied return value. It also shows how to create an instance of the struct without a constructor. Line 30: We're finally in the main! I've forgotten what problem we were solving by now. Ah right, "find the sum of even fibonacci numbers less than 4,000,000". Let's look at the solution. Since there's a lot going on in this fluent-style approach, let's break it up into steps: fibonacci()</code> - we call our function and get out the default value to start iterating with. Since we already told Rust how to use values of this type as iterators, it gets all the capabilities that iterators have. .take_while(|&x| x < 4000000)</code> - that didn't take long, we're already jumping in and using the functionality of the iterator to check each turn of the crank, and if we exceed 4,000,000 stop. The |&x| x < 4000000</code> is a simple function that takes in a reference to each element we bind to x, and then we check the value we're handed in the body of our function. If you're coming from C++, this reference style might look a little strange. Aren't you risking someone updating the value? Turns out Rust's references are immutable by default</a>. The reference here also shows us the performance characteristics of .take_while</code>. By this I mean, when we look at the .take_while</code> call, we see it will only pass references, so know ahead of time if we're iterating over big structures, we're not paying the price of copying each one with each step of the iterator. Speaking of performance, it may look like .take_while is going to be exceedingly expensive. Is it consuming all of our fibonacci sequence until we have all the items we need to get to 4,000,000? Luckily, it isn't. Instead, .take_while</code> itself returns an iterator that you can continue your fluent calls on. I think of it like a series of machines in a factory: -> [ ] -> [ ] -> [ ] -> </code></pre> If I start the conveyor up and I pull from the right hand, I'm moving whatever was on the far left all the way through the system. In our case, we're starting up the fibonacci iterator and getting that machine running. It spits out a single fibonacci-shaped number for us. This number chugs along and enters the next machine, which is our machine that checks if it's too big. If it's not, the conveyor chugs along and out drops a single fibonacci-shaped number of the acceptable size. As these machines work in lock-step, we're getting just what we need when we need it rather than trying to run ahead and doing a bunch of work in advance. This allows you compose a lot of steps together and still stay efficient. .filter(|&x| x % 2 == 0)</code> - Just like .take_while</code> above, this one checks each number as it passes through. This time, it only lets the number through if it's divisible by 2, and hence even. Now we have all the steps in place, each time we turn the crank of the whole iterator, only fibonacci numbers that are less than 4,000,000 and even will pop out. Line 31: We've already done all the hard work. Now we just have to sum those numbers together. With that, we're done. While it was quite a mouthful, we can see the compositional nature of Rust, a bit of the type system, and how iterators work in more detail. Problem 3</h1> "highest prime factor of 600851475143" fn is_prime(num:u64) -> bool { for i in 2..(num / 2 + 1) { if num % i == 0 { return false; } } return true; } struct Prime { curr: u64, } impl Iterator for Prime { type Item = u64; fn next(&mut self) -> Option<u64> { let mut new_next = self.curr + 1; while !is_prime(new_next) { new_next += 1; } self.curr = new_next; Some(self.curr) } } // Returns the primes fn primes() -> Prime { Prime { curr: 1 } } fn main() { let mut num:u64 = 600851475143; let mut highest_prime_factor = 0; for i in primes() { if num % i == 0 { highest_prime_factor = i; } while (num % i == 0) && (num >= 2) { num /= i; } if num == 1 { break; } } println!("num: {}", highest_prime_factor); } </code></pre> Very similar to Problem #2 above, I create an implementation of Iterator, so I can use it later. This time, rather than creating a stream of fibonacci values, I create a stream of prime numbers. Just like we had a struct for Fibonacci in Problem #2, I create one here for Prime numbers. Since we don't need the previous one to calculate the next one, I only keep the current value around and then start with that number + 1 when trying to find the next prime. Once we have our iterator, we take each prime, and then try to divide out all its multiples from our give number. What's left should be our highest prime factor. Easy peasy. PS: I know it's probably already too late, but I should have warned you to avert your eyes from my inefficient prime number skills :) Problem 4</h1> "largest palindrome product of two 3-digit numbers" fn is_palindrome(num: u64) -> bool { let s = num.to_string(); for (i1, i2) in s.chars().zip(s.chars().rev()) { if i1 != i2 { return false; } } return true; } fn main() { let mut largest = 0; for i in 100..999 { for j in 100..999 { if is_palindrome(i * j) && ((i * j) > largest) { largest = i * j; println!("Largest: {}, {} x {}", largest, i, j); } } } } </code></pre> Problem #4 also uses some iterators to find the palindrome product (line 13 and line 14). If it finds a palindrome product, and that new product is larger than what it saw before, it replaces it with the new one. There's one fun trick in this solution on line 3. We need to check if the number is a palindrome. While you can do this numerically</a>, I couldn't help but use some more iterators. This time, after converting the number to a string, we create an iterator over the characters in the string (using .chars()</code>). We want to see if the number is a palindrome, so one way to test this is that if we reverse the string and if still get the same value, we know we've got one. That's exactly what we do. Here, I use .zip(s.chars().rev())</code> to combine the iterator that goes forward over the characters with one that goes in reverse. The .zip()</code> call brings both iterators together and iterates over both iterators for us, returning a pair (or tuple) of values. Think of it running both machines side by side, and with each turn of the crank one value pops out of each and zip returns both values tied together with a bow. Now that we have a single pair, we immediately use it. At the beginning of line 3, you can see the for (i1, i2)</code> in part of the line. Just as before for..in</code> will run over the iterator. We know that the iterator is going to be outputting pairs of values. We could have just grabbed the pair as a single value and used it later, but here I go one step further by taking the pair the iterator gives me and pulling it apart so I can use it. This pulling apart, called destructuring, is a handy way of working with structured data when you're interested in its constituent parts rather than the structure itself. In this case, this lets me more easily compare the left side to the right side, so I destructure the pair into two values: i1</code> and i2</code>. Problems 5 through 7</h1> Problems #5</a>, #6</a>, and #7</a> are all straightforward and don't show off anything new in the language. Let's jump ahead to the next interesting one. Problem 8</h1> "largest product of 13 adjacent digits" fn main() { let input = "73167176531330624919225119674426574742355349194934\ 96983520312774506326239578318016984801869478851843\ 85861560789112949495459501737958331952853208805511\ 12540698747158523863050715693290963295227443043557\ 66896648950445244523161731856403098711121722383113\ 62229893423380308135336276614282806444486645238749\ 30358907296290491560440772390713810515859307960866\ 70172427121883998797908792274921901699720888093776\ 65727333001053367881220235421809751254540594752243\ 52584907711670556013604839586446706324415722155397\ 53697817977846174064955149290862569321978468622482\ 83972241375657056057490261407972968652414535100474\ 82166370484403199890008895243450658541227588666881\ 16427171479924442928230863465674813919123162824586\ 17866458359124566529476545682848912883142607690042\ 24219022671055626321111109370544217506941658960408\ 07198403850962455444362981230987879927244284909188\ 84580156166097919133875499200524063689912560717606\ 05886116467109405077541002256983155200055935729725\ 71636269561882670428252483600823257530420752963450"; let mut largest = 0; let input_bytes = input.as_bytes(); let mut largest_string: &[u8] = &input_bytes[0..1]; let span_width = 13; for i in 0..(input_bytes.len() - span_width + 1) { let mut sum = 1u64; for j in 0..(span_width) { sum = (input_bytes[i + j] - 48) as u64; } if sum > largest { largest = sum; largest_string = &input_bytes[i..(i+span_width)]; } } println!("Largest: {} is {:?}", largest, std::str::from_utf8(largest_string)); } </code></pre> In Problem #8, we see an example of working with strings, this time using a new feature called a 'slice'. We start with the block from line 3 to line 22 that gives us this giant input string. Notice that I use the backslash '' to continue the line to the next line. Rust is smart enough to let us indent without introducing new whitespace into the string. Now that we have our string, on line 25 we turn it into a array of bytes we can search through. Notice we didn't use .chars()</code> this time. Recall that .chars()</code> gave us an iterator. Rather than iterating over one value at a time here, we want the flexibility to look across spans in our string. To do that, we use .as_bytes()</code> to get an array (technically, a byte slice). Line 26 introduces the slice we'll use. A slice is a window into values in memory. Since we want to look at a span of values, we create a holder for this slice. We'll later set it to spans of our 13 adjacent digits. PS: you'll notice I set it to a dummy default value, which I later throw away. This is to get around the compiler warning about uninitialized values. The rest of the search follows fairly naturally. We do an iterator over the indices in our input, stopping short of the length of the span. We use that index to calculate the product at that span (line 32 to line 34). Once we find a match, we save it off, using a slice to save off the winning span (line 37). Finally, we use another format string {:?}</code>, which can print out our winning span for us using the debug formatter. The {:?}</code> format is a handy builtin which can handle a wider range of types with a default formatter. Problem 9</h1> Problem #9</a> is another solution that's simple and doesn't show off any new features. Moving on. Problem 10</h1> "sum of all primes below 2 million" fn main() { let mut sum = 0; const SIZE: usize = 2000000; let mut slots: [bool; SIZE] = [true; SIZE]; slots[0] = false; slots[1] = false; // We calculate the primes using a simple stride and marking off multiples for stride in 2..(SIZE/2) { let mut pos = stride; while pos < (SIZE - stride) { pos += stride; slots[pos] = false; } } for (idx, pr) in slots.into_iter().enumerate() { if pr { sum += idx; } } println!("Sum: {}", sum); } </code></pre> Reading the problem description, you might have guessed that I would use the iterator solution from earlier. The astute reader probably already noticed that doing so would be. very. slow. Especially, as we look at primes above a million, where each step is itself taking hundreds of thousands, if not millions, of calculations. While I could let it run and heat up my apartment, it's better to do a more direct approach. We trade space for time. On line 4, I create my first fixed size array that will hold whether or not the number in that position is prime. Once we have the variable, it's initialized using another cool shorthand: [true; SIZE]</code>. This creates an array of boolean true values of the size given on the right of the ';', here the SIZE constant I defined. For example, you can create a 500-element array of zeros using [0; 500]</code>. Neat. Once we have our slots, we loop over them using any possible multiple and check that off the list by setting that position to false. Once we're done, we have an array that tells us where the primes are. The only trick remaining is how to get the numbers back out again. To do this, we use the iterator on line 17. Similar to our fluent iterators before, this time we turn our array of slots into an iterator, and then call .enumerate()</code>. The .enumerate()</code> call is a special kind of zip that gives us the index at that point in the iteration paired with the actual value. Just as before, we destructure the index and the boolean that indicates if it's a prime number separately. On line 18, we check if the number is prime, and if so we add it to our sum. You'll notice the pr</code> call here. Just as in C, this lets me dereference and get at the value at that location in memory. I could have instead written for (idx, &pr)</code> in and just gotten to the value that way, using destructuring instead of dereferencing. With that change, the line would have been if pr { ... }</code>, and I could get at the value pr</code> without needing to dereference. Problem 11</h1> "find the largest product in a grid" let input = "\ 08 02 22 97 38 15 00 40 00 75 04 05 07 78 52 12 50 77 91 08 \ 49 49 99 40 17 81 18 57 60 87 17 40 98 43 69 48 04 56 62 00 \ 81 49 31 73 55 79 14 29 93 71 40 67 53 88 30 03 49 13 36 65 \ 52 70 95 23 04 60 11 42 69 24 68 56 01 32 56 71 37 02 36 91 \ 22 31 16 71 51 67 63 89 41 92 36 54 22 40 40 28 66 33 13 80 \ 24 47 32 60 99 03 45 02 44 75 33 53 78 36 84 20 35 17 12 50 \ 32 98 81 28 64 23 67 10 26 38 40 67 59 54 70 66 18 38 64 70 \ 67 26 20 68 02 62 12 20 95 63 94 39 63 08 40 91 66 49 94 21 \ 24 55 58 05 66 73 99 26 97 17 78 78 96 83 14 88 34 89 63 72 \ 21 36 23 09 75 00 76 44 20 45 35 14 00 61 33 97 34 31 33 95 \ 78 17 53 28 22 75 31 67 15 94 03 80 04 62 16 14 09 53 56 92 \ 16 39 05 42 96 35 31 47 55 58 88 24 00 17 54 24 36 29 85 57 \ 86 56 00 48 35 71 89 07 05 44 44 37 44 60 21 58 51 54 17 58 \ 19 80 81 68 05 94 47 69 28 73 92 13 86 52 17 77 04 89 55 40 \ 04 52 08 83 97 35 99 16 07 97 57 32 16 26 26 79 33 27 98 66 \ 88 36 68 87 57 62 20 72 03 46 33 67 46 55 12 32 63 93 53 69 \ 04 42 16 73 38 25 39 11 24 94 72 18 08 46 29 32 40 62 76 36 \ 20 69 36 41 72 30 23 88 34 62 99 69 82 67 59 85 74 04 36 16 \ 20 73 35 29 78 31 90 01 74 31 49 71 48 86 81 16 23 57 05 54 \ 01 70 54 71 83 51 54 69 16 92 33 48 61 43 52 01 89 19 67 48"; let input_split = input.split_whitespace(); let input_as_num: Vec<i32> = input_split.map(|x| match i32::from_str_radix(x, 10) { Ok(v) => v, Err(u) => { println!("Garbage in input: {}", u); 0 } } ).collect(); </code></pre> I'm only showing the interesting part of the solution since it's a bit long, but you can read the whole solution</a>. Here we do a bit more string manipulation using some new methods. The .split_whitespace()</code> method on line 23 lets us turn our input string into an iterator of strings, splitting at any whitespace. Once we have this iterator, line 24 takes this iterator and maps a function over it. This function attempts to convert each substring to a signed 32-bit integer i32</code> using the .from_str_radix(x, 10)</code> call. This call returns a Result</code>, a way of handling either a success or failure condition of an action that might fail. As is the case with converting from strings to numbers, if you happen to pass in something that can't be converted to a number, there needs to be a way to signal back out that the conversion failed. Some languages do this through exceptions, which break out of normal execution and give you an error you can handle. Rust uses a simpler, more functional approach and treats errors as just any other value. With this method, the type system encourages us to be vigilant and always have code available to handle errors. To see which of success/failure is returned, we use the match keyword. Just as in other languages with pattern matching, the match keyword lets us ask which of the possible values is in the Result: Ok</code> or Err</code>. Pattern matching lets us destructure and find the success value or error message. Finally, on line 29, we use the .collect()</code> method to run through the iterator and create a vector of i32</code>. Problem 12</h1> "triangle number with over 500 divisors" fn main() { let mut num = 0u64; // Cache the primes we'll be using let first_primes: Vec<u64> = primes().take(1000).collect(); for i in 1..1000000u64 { num += i; let mut num_div = 1u64; let mut num_tmp = num; for &x in &first_primes { let prime = x; if (num % prime) == 0 { let mut exponent = 1; while (num_tmp % prime) == 0 { exponent += 1; num_tmp /= prime; } num_div = exponent; } if num_tmp == 1 { break; } } if num_div > 500 { println!("num: {}", num); return; } } } </code></pre> Again we'll trim down to the interesting parts, since we've seen the prime iterator before, but you can read the full file</a>. This looks pretty familiar. On line 5, we run our primes iterator, take enough to use later, and turn that into a Vec<u64></code> we can use a bunch of times. The other interesting line is line 13. If you've played with vectors in Rust before, you know that you can iterate over them, like this: {% highlight rust %} fn main() { let x: Vec = (1..10).collect(); for i in x { println!("{}", i); } </code></pre> } {% endhighlight %} But if we try to do this in our Euler problem, we get a new error: error: use of moved value: `first_primes` </code></pre> What is a moved value? It turns out that a moved value is part of Rust's ownership system. Just like the first time a programmer sees a pointer in C or a monad in Haskell, ownership in Rust is one of those things that's so uniquely Rust that it takes getting used to. Once you begin to understand it, you'll be able to see how the type system is trying to help you describe your intentions a bit more clearly. If you're scratching your head, here's an example: imagine malloc'ing an array in C and then passing that malloc'ed array into a function. Who is responsible for cleaning up that memory, the function you called or the function that created the array? In C, you don't know by looking at the code, you only know as the writer of the code what your intention was. Maybe the function is called 'cleanup' at which case, yes, you do want it to free that memory. Or maybe the function is called 'inspect' and you decidedly don't want it to free the memory. This is why Rust makes this very explicit. By looking at the code, you can follow who is responsible for what. The set of checks that do this make up Rust's ownership system. There are a lot of resources</a> to understand ownership,which do a good job of explaining ownership in detail. For us, ownership is like trying to answer questions like "who is responsible for this thing?" and "when does this thing get removed?" Ownership can be passed from variable to variable, and its strictly checked by the compiler. When you first encounter the above error, if you haven't already read about ownership, you might be in for a few hours of reading to get caught up. But we have some hints here that we can get started with to help figure out what's happening. In our small example, we could iterate over a vector and all was well. When did things start to go wrong in the Euler problem? On line 13, we want to loop over a vector, and when we look out to line 7, we see this is actually an inner loop being used by an outer loop. That means that if the inner loop takes control of our vector, the next time around the outer loop we've lost the ownership. We don't want the inner loop to take ownership and then lose the ability to use the vector again in the next iteration of the loop, so we have to 'borrow' ownership. As the name implies, by borrowing, we're only taking temporary ownership for a time, and we have to follow all the rules stated when we borrow. With that in mind, let's look at line 13 again: for &x in &first_primes </code></pre> We borrow first_primes</code> instead of taking permanent ownership using the &</code> operator. This operator says "I would like to borrow first_primes, and I can not edit the contents". The compiler will then be able to check that we're using what we've borrowed correctly. You might be curious borrowing and also being able to mutate. To do that, we could've used the &mut</code> operator assuming the original vector was mutable. Finally, we use the same destructure trick we did earlier to get at the values of x</code>. PS: In this example, since all we were doing was iterating, we could have used the .iter()</code> rather than borrowing, though that won't always be the case in other examples that need more direct interaction with the vector. Cargo</h1> After I got enough of these problems together that I got tired of building them by hand, I moved over to using Cargo. Cargo is Rust's build/package tool that is quite handy. [package] name = "eulerinrust" version = "0.1.0" [[bin]] name = "ex1" path = "src/ex1.rs" </code></pre> An example of Cargo.toml Cargo lets you describe dependencies that your project needs, which it fetches from crates.io. You can also control whether you're doing a release or debug build, and quite a bit more. I'll no doubt start directly with Cargo for any future Rust projects. Summary</h1> These are just tiny examples created while playing around in a fairly simple set of problems, and yet there's actually quite a lot of Rust's richness shining through. Rust has a powerful iterator story that feels composable and clear. Rust's error checking is keeping us vigilant about handling error cases, matching types, and remembering who is responsible for values in the system. Overall, it feels like a very well thought-out language that holds together well. Off to new adventures 2015-10-05T00:00:00+00:00 A few weeks ago I posted about moving teams inside of Microsoft. As sometimes happens when you make a change, you realize that you actually need to make a bigger change. A few weeks into the new team, and I could tell that it was time for me to move onto new adventures. Microsoft has been a great place for me to "grow up" as a program manager. When I started as an intern, I was mentored by some of the most capable PMs I've ever met. They took a keen interest in me, and I benefited greatly from their years of experience and patience. From a little bird of a PM, I grew to being able to lead the great TypeScript team. I got to meet countless numbers of people and make some good friends along the way. It was a wild adventure to be on the team, and I'm excited to see what they come up with next. But as little birds grow, there comes a time to leave the nest and to take more risks. There are many adventures to be had, and to have them, you have to ride out the storms in search of the blue skies. My first step will be to take some time to travel the world and see friends I haven't seen in years. My goal: recharge, explore, and take stock. After that, who knows. Part of the adventure is not seeing the way ahead and going anyway. Programming languages are a failure 2013-02-17T00:00:00+00:00 After spending years working in programming languages, helping out with some big-name projects and trying to generally make programmers' lives easier, I've decided something. Programming languages are a failure. Okay okay, before you write this off as a troll for attention, hear me out for a minute longer. Calling a programming language a 'language' is false advertising. Think of what comes to mind when I say 'language'. A language, as in human-to-human language, has some pretty interesting properties: It can be used to express intent</li> It can transmit information even if one (or even both) of the speakers are non-native</li> </ul> The act of programming is inherently not like speaking. Imagine having a conversation with someone in the same way you program. Say what you want the other person to understand</li> Say it again, this time with slightly different wording</li> Say it again, with yet more slightly different wording</li> (Rinse, repeat. Time passes...)</li> Test the other person on what you've said to be very sure they understood</li> Move on to the next thing you want the other person to understand</li> </ul> Of course, the reason we have to do this in programming is because we want what we say to be precise, so the computer exactly what to do, right? Human language is filled with all that icky imprecision, which is why we use a programming language. The special kind of language that precisely describes what we mean to the computer, so long as we're patient enough to explain ourselves just so. What we've actually created is something akin to legalese for programming rather than legal documents, a sort of programese. It's a one-way conversation with the computer in a style that removes as much room as possible for misinterpretation. The dirty secret is that programming languages, like many legal documents, never really finish going through refinement, even after years and years of refinement. Why not embrace the ambiguity? Better yet, why not make programming a two-way communication that more closely resembles people talking? Instead of this: > The bird sits on the pole. Error: I don't understand 'bird'. I don't understand 'pole'. > The bird, an animal, sits on the telephone pole. Error: The animal 'bird' can not "sit". > The bird, an animal, stands on the telephone pole. Okay. </code></pre> Why not have this: > The bird sits on the pole. Question: By "sits" do you mean "stands"? > Yes Understood. </code></pre> Let the programming system ask you when it truly doesn't understand. The difference is, where it can infer an understanding based on context and background knowledge, it does. This opens up further possibilities. What the programming system could be doing for you behind the scenes is fill out what it understands, changing the text much like Telescopic Text. This would replace "The bird sits on the pole" after you type it in to be a telescopic text you could click on "[The bird] [sits on] [the pole]"</code>. If you clicked on [The bird]</code>, the text would telescope in one level, giving you: "[The bird, an animal][sits on][the pole]"</code>. Imagine how much easier Promising new programming languages 2011-09-22T00:00:00+00:00 It seems like these days there are lots of programming languages springing up. A combination of the web, parallel programming, and just outright curiosity seems to be fueling creativity towards new mixtures of classic features and experimental new ones. Here are a few new(-ish) projects I'm keeping my eye on: Rust</a> (Mozilla Research) - Concurrent language that only allows sharing of mutable state if it's handed off between threads. Uses typestates to enforce this ownership. Expected to be used in Project Servo, Mozilla's experimental parallel browser engine. Recent talk slides look promising. Still experimental (haven't reached 0.1 yet). Kotlin</a> (JetBrains) - Impressively clean/lightweight syntax language for the JVM in the spirit of a Scala light. Perhaps may be to Scala what Firefox was to Netscape, but it's not yet release so it's hard to say for sure. Still, their presentation at the JVM Language Summit is worth a watch. ooc</a> - You have to like the infectious audacity of this project. Hobbyist putting together a language, then proceeding to write one compiler for it in itself and then another. Dart</a> (Google) - Mentioning it here because I'm curious, but the hype has already overshadowed it. We're still weeks away from knowing anything about it. Still, Gilad Bracha is like a "curiously strong mint", so we at least know it won't be flavorless. These are just the ones on my radar. There are certainly others (Ceylon, Gosu, Mirah - and those are just the ones on the JVM), but they haven't quite made it to my twitter follow list. Will parallelism go implicit? 2011-03-08T00:00:00+00:00 I've been spending quite a bit of time recently with Implicit Parallel Programming in pH</a>, which luckily the school library had a copy of. It's quite a good textbook to introduce programmers to functional programming, perhaps even one of the better ones (next to the original Introduction to Functional Programming</a> by Bird, Wadler that I mentioned in an earlier blog post). What turned me on to it, though, wasn't the treatment of functional programming - it was its far more audacious titular goal. Which leads to the obvious question. What is implicit parallelism? If you bring up the term around PL geeks, you might get a few questioning looks and some comments that hint at automatic loop parallelization. Even more confusingly, this second kind of parallelism is sometimes referred to as automatic parallelism. It refers to taking sequential codes and reasoning about where parallelism could occur without changing any results. Implicit parallelism is a more holistic approach - what if the only sequentiality in your program comes from the data dependencies? Then the rest, it stands to reason, can be done in parallel. I mentioned this as an audacious effort. The first and foremost reason, to my engineering mind, is how in the world will you make this efficient? I can just imagine the compiler identifying tens, perhaps even hundreds, of tiny tasks that can be done in parallel at any one timeslice. How do we schedule these tasks in such a way we don't pay a high overhead for synchronization and task switching? I haven't dug into the implementations of the languages yet to see how they address this issue, but as I was brainstorming this morning it hit me. Implicit parallelism isn't so different from implicit (or automatic) memory management. When garbage collectors first came out, they weren't the lean mean things they are today. That's part of the advantage of having decades to polish an approach, and now fewer and fewer applications programmers use languages that don't have gc support. It saves more headaches than it causes, and that's generally a winning combination. Not always, of course. The HPC community has regularly shunned gc in their languages, where high performance codes are hand-tuned Fortran or C with manual memory management. Still, I would argue that HPC is boutique, a cottage industry focused on maximal efficiency. Most everyday programmers don't need these levels of efficiency, and I imagine, will instead opt for automatic support for parallelism. This saves them the headache of separately tuning for the same application to run comfortably on either 16 or 128 cores. From the consumer's point of view, this is also a win. Buying that new 256 core machine actually will make these apps run faster. Some final thoughts Perhaps it's time to focus on introducing more forms of automatic and implicit parallelism into our programming languages and runtime systems. Our early attempts, taking from the experiments of the past could kick it off, and once industrial interest has turned to it, I'd expect steady progress in efficiency (just like we've seen in gc). Is there some practical limit to the amount of parallelism in code? There's a tension between Amdahl's law</a>, that maximum speedup through parallelism is limited by the amount of sequential code, and Gustafson's law</a>, which says that the increased computing power lets us tackle larger problems. To continue to gain we may have to change how we approach problems, or even reframe the problems themselves. Thoughts on actor-based languages 2010-09-07T00:00:00+00:00 (or "Minnow, a post-mortem") I've been fortunate to work on both Minnow</a>(a shared-nothing actor-based language) and Chapel</a>(concurrent global-view language). I wanted to give a few thoughts comparing the two approaches. First, when we talk about concurrency or parallelism, without going into the pedantic definitions of each, we should answer the question "why not stay serial?" There are a lot of reasons to do so, if we can. Serial code is going to be easier to write, easier to debug, and easier to reason about. Generally, going parallel is just going to complicate our lives. I see two main reasons why going parallel would be required: Reliability - Does the system have to tolerate failures of its parts?</li> Performance - Is serial too slow for your application?</li> </ul> In Erlang</a>, reliability is king. Its shared-nothing actor-model style allows actors to fail and be restarted, which allows for more advanced features like hot code swapping. For areas where performance is king, C and Fortran still reign. In high-performance scientific computing, it's common to remove abstractions and program as "close to the metal" as possible. Though this doesn't preclude reliability, that reliability must either be done by the programmer or by the underlying architecture. It is, as you can imagine, very fast. When I created Minnow, I set out to create an actor-based language that could perform as fast as C but potentially gain the benefit of reliability from the actor abstraction. Focusing solely on single node, shared-memory machine, I was able to optimize: Message sends, down to a few atomic operations</li> Number of actors possible, using light-weight threads and continuations</li> Run-time overhead, even in the presence of fairness counters for cooperative multi-tasking, by minimizing extraneous work in loops and function calls</li> </ul> In short, Minnow was quite fast. Was it fast enough? As it turns out, the answer is no. Though Minnow avoiding copying where possible, it did not allow the user to see a large matrix and operate on pieces of it. Doing so would go against the notion of shared-nothing, we might even say that shared-nothing runs counter to the goal of performance. Algorithms where we could bifurcate data into chunks, for example, wouldn't suffer, but in general we can no longer use contiguous memory in a natural way. The modern batch of concurrent languages (Chapel, X10, Co-Array Fortran, and others) return to looking at memory in a contiguous way. In fact, using partitioned global address space</a> (or PGAS), the notion of contiguous memory is extended to encompass the memories of all the nodes in a cluster. The question of where reliability should live, and if we should push it into the language and sacrifice performance, appears to be solved issue in the high-performance community. It remains to be seen if the larger programmer community will embrace this approach. Digging into scheme/racket 2010-09-05T00:00:00+00:00 I spent some of last night and this morning digging into Racket, the modern version of the venerable Scheme language. Every time I return to a Lisp variant, I find the parentheses less annoying that I did the last time. With proper indenting and some syntax highlighting, is there all that much difference? (define factorial (lambda (n) (if (= n 0) 1 (* n (factorial (- n 1)))))) </code></pre> Arguably the trailing parentheses are an eye chard. With the download of Racket you also get DrRacket, which is an updated version of DrScheme, a learning tool aimed at programmers of different experience levels (and to my understanding different ages as well) I've just begun to experiment with DrRacket to see what it can do. To get familiar with the Scheme style, I've been working through Teach Yourself Scheme in Fixnum Days</a>.

Sophia Turner's Blog

I wrote a book!

Bottom Surgery at WIH

Where I'm at now</h2>

Two years on HRT

I stopped believing in America

Trans Lesbian Book List

Sharing Trans Femme Lit

Following new paths ahead

The search for easier safe systems programming

Exploring related work</h3> Over the years, there have been a number of memory management techniques tried</a>, including many that lie outside of the ones commonly found in languages today. We'd like to explore these more deeply to see which, if any, may help June.</p>

Worst part of being trans

One year on HRT

Mind changes</h2>

Voice changes</h2> Haha, if only HRT helped with this. It sadly does not. I have been practicing regularly for the last 6 months, which has helped a bunch. But wow, it's work work work.</p>

Books read in 2023

April Daniels</h2> Now we get into the trans and sapphic books. April was the first trans author I read. She wrote two books about a trans superhero. After I read those, I was hooked and kept looking for more.</p>

Sakaomi Yuzaki</h2>

Meredith Russo</h2>

Julie Anne Peters</h2>

Vyria Durav</h2>

Molly J. Bragg</h2>

Tiffany E. Taylor</h2>

Akwaeke Emezi</h2>

Alison Rumfitt</h2>

Silvia Shaw</h2>

Issy Waldrom and Mackenzie Sutton</h2>

Ela Bambust</h2>

Rosemary Kirsten</h2>

Maria Ying</h2>

Andrew Joseph White</h2>

Kai Cheng Thom</h2>

Mel E. Lemon</h2>

Zoe Storm</h2>

Avery Arden</h2>

SD Simper</h2>

May Peterson</h2>

Elena Abbott/Lily Hardt</h2>

That's wrap</h2> And that's it for my list. If any of you have reading recommendations that fall along with this list, let me know! I got turned on to a lot of authors that I'll be watching in the coming year(s).</p>

Becoming Sophia

The case for Nushell

The lay of the land</h2> Let's take a look at some of the offerings out there that people are using everyday.</p>

Show us the money</h2> I make some pretty bold statements in the above. There is a heritage of technology that got us to this point. While that heritage is important, it's not without its drawbacks. Are there better ways of doing it?</p>

Exploring related work</h3>
Over the years, there have been a number of memory management techniques tried</a>, including many that lie outside of the ones commonly found in languages today. We'd like to explore these more deeply to see which, if any, may help June.</p>

Voice changes</h2>
Haha, if only HRT helped with this. It sadly does not. I have been practicing regularly for the last 6 months, which has helped a bunch. But wow, it's work work work.</p>

April Daniels</h2>
Now we get into the trans and sapphic books. April was the first trans author I read. She wrote two books about a trans superhero. After I read those, I was hooked and kept looking for more.</p>

That's wrap</h2>
And that's it for my list. If any of you have reading recommendations that fall along with this list, let me know! I got turned on to a lot of authors that I'll be watching in the coming year(s).</p>

The lay of the land</h2>
Let's take a look at some of the offerings out there that people are using everyday.</p>

Show us the money</h2>
I make some pretty bold statements in the above. There is a heritage of technology that got us to this point. While that heritage is important, it's not without its drawbacks. Are there better ways of doing it?</p>