Thanks to ITA Software (powering Kayak and Orbitz), Google dedicates resources to open-source Common Lisp development. More specifically, to SBCL:
> Doug Katzman talked about his work at Google getting SBCL to work with Unix better. For those of you who don’t know, he’s done a lot of work on SBCL over the past couple of years, not only adding a lot of new features to the GC and making it play better with applications which have alien parts to them, but also has done a tremendous amount of cleanup on the internals and has helped SBCL become even more Sanely Bootstrappable. That’s a topic for another time, and I hope Doug or Christophe will have the time to write up about the recent improvements to the process, since it really is quite interesting.
> Anyway, what Doug talked about was his work on making SBCL more amenable to external debugging tools, such as gdb and external profilers. It seems like they interface with aliens a lot from Lisp at Google, so it’s nice to have backtraces from alien tools understand Lisp. It turns out a lot of prerequisite work was needed to make SBCL play nice like this, including implementing a non-moving GC runtime, so that Lisp objects and especially Lisp code (which are normally dynamic space objects and move around just like everything else) can’t evade the aliens and will always have known locations.
That's interesting. Where is Lisp used at Google? I vaguely recall reading long ago that Google had some kind of policy against using Lisp or Smalltalk for projects, but I can't find the article I am thinking of [1].
[1] I'm not 100% sure, but it was probably some kind of article about "programming languages used by Google in production." I recall it talking about how C and Python were approved, as well as Java (~"because there are a lot of good programmers who write Java"). Lisp and Smalltalk were explicitly banned, mainly because engineers so frequently proposed using them because of their appeal, despite the supportability issues w.r.t. skillsets.
That's close, but I think it's reflection of the same policy, not the same article. The article I remember talked about more languages that just Lisp, Smalltalk, and Java. It was overall down on Java, but had positive things to say about some Java programmers.
Pharo would be the practical one, Squeak the less industrial one. Lots of stuff has been implemented with both, enough that you could run a consultancy and build applications for people.
You'd need to be a bit of a Smalltalk nerd though, and be prepared to do most of the work yourself since there aren't that many smalltalkers to recruit from. You'd also be in competition for developers with companies building software for insurance or ERP, and many smalltalkers spent relatively much time in academia so that could affect salary expectations.
Mediagenix's "What's On" ( https://www.mediagenix.tv/whats-on/ ) is written in Small-talk. The main reason is the company's ancestry had strong academic roots, but it worked for them. It's not Open Source, but since there aren't that many successful commercial Small-talk products in the wild I thought I'd mention.
Absolute mountains. Consider: Ruby is Smalltalk with alternative syntax and core library.
I'm only being a little bit flippant here; the only core language feature in ST that Ruby is missing is thisContext, and most of the things that end developers do with thisContext (continuations, exceptions, and the debugger) are natively supported by the standard library.
There is something nostalgic about these old Google documents. It reminds me of when search ads were on the right side.
Google is famous for only supporting a few general languages. Initially it was just C++, Python, Java, and Javascript (browser only!), although a few more eventually trickled in. The language deviants kept writing "official" style guides and adding support to random infrastructure and libraries, hoping that one day the next major project might get written in OCaml. New Googlers often had a confusing moment after learning the restrictive language policy when they compiled their first program and blaze would print that it was generating protocol buffer bindings for Haskell (this would eventually be suppressed).
Haskell is another oddball. It was for a hardware project that used the to-be-auctioned analog TV RF spectrum for a new WAN standard.
It was canceled when it became clear the spectrum wasn´t going to be available. The Haskell rules then stuck around for another ~7 years before I deleted them as part of the cleanup for the Bazel release. They then promptly reemerged as one of the first language rules to be expressed in Bazel's newfangled Starlark language (then still called Skylark).
> Google is famous for only supporting a few general languages. Initially it was just C++, Python, Java, and Javascript
I think most of the big companies are like this. It tracks my experience at least.
At the scale that these companies operate at, it's a good thing. Even if you restrict language choice, you tend to end up with half a dozen or more languages in common use at your company, each of which you have to support with developer environments, CI/CD configurations, security & supply chain code vetting, etc.
One of the largest Borg jobs? Really? This goes hard against my expectations, both based on my understanding of the software involved in flight search, and in terms of my knowledge of things that run on borg and how big they get. Feel free to reach out internally, I'd love to chat about it!
Remarkable to me that Google would spend so much on software engineers and then hobble them with blub languages. If they decided to push a niche but superior language then people would learn it (as they did with Go).
Google's main line of business back in the day involved crawling and indexing as much of the entire web as possible. When you're operating at that kind of scale, back in the 2000s-early 2010s your choices were C++, and pretty much nothing else. Garbage-collected languages introduced too much overhead. Keeping the number of languages small, and keeping them purpose-focused (C++ for the heavy stuff, Java/Python for web backends, JavaScript for UIs) helped unify coding standards and conventions within and across teams, reining in the chaos that would have ensued had Google allowed developers or teams to choose languages willy-nilly.
These sorts of language restrictions are not a knock against the skill of Google engineers. A good programmer can be good in any language. Even C++. Even Java. Think of the midwit meme, with the simpleton at the left going "just write everything in C++ or Java", the midwit in the middle going "NOOOOOO! Lambda calculus/monads/Hindley-Milner type systems!" and the genius/Jedi monk at the right going "just write everything in C++ or Java".
The appeal of Lisp when writing production code is to lazy, mediocre programmers who think Lisp's sexy features will extend the reach of what they can create. Which is true, to a certain extent, but a) not as much as they may think; and b) when you consider that Lisp programs are bloated and slow compared to the corresponding C++ programs, the juice just isn't worth the squeeze.
Lisp peaked ~60 years ago. There are very sound technical reasons why it is obscure today.
I think this is the perfect opinion to have for a middle manager at a large FAANG with seemingly infinite developer supply and little existential incentive to innovate.
In practice, C++ and Python codebases have insane amounts of churn. They're constantly moving targets and require a lot of overhead to keep in shape, from just about every perspective: performance, security, maintainability, implementation compatibility, etc. It's tolerated though because it's standard practice, and we developers get a paycheck anyway.
Languages like OCaml, Common Lisp, Haskell, and other niche but obviously practical languages continue to be indispensable to certain cohorts of companies and developers. While I do agree that Lisp specifically peaked in popularity decades ago, it still exists in the commercial domain with resounding staying power, even in environments where it could in theory be replaced by C++ or Python.
It's also abundantly clear there's a hunger for new (or at least replacement) languages for the stalwarts. Rust, Julia, Mojo, Swift, and OCaml are a few such languages that get attention in their commercial investment and development, often with the specific purpose of replacing Python or C++.
As a personal anecdote, I find professional Common Lisp programmers generally to be quite sensitive to matters of bloat. Some programmers go as far to chisel away needless cons cells (or allocation in general), and constantly be in touch with the compiled assembly—something frequently ignored by professional C++ programmers of comparable experience.
I would love to hear the "very sound technical reasons" explaining Common Lisp's obscurity. Given the vast and diverse biome of programming languages in use today, including and especially in professional settings, I'm quite interested in understanding what technical reasons uniquely relegate Lisp to obscurity.
My take (not OP) is that it's not so much the technical features of Lisp itself, but the fact that other languages evolved competing features. Over the years, they have slowly chipped away at Lisp's advantages. 50 years ago, Lisp was uniquely powerful with no rival. Today, Lisp is yet-another powerful language to choose from.
I disagree: Common Lisp has many remaining powerful features AND a large set of them, when a language maybe caught up with one. The interactive debugger is unmatched, image-based development nearly doesn't exist elsewhere, the condition system… Example: fix a bug without quitting the debugger, re-compile the fixed function with one keystroke (instant feedback), go back to the debugger, choose a stack frame to resume execution from, see the execution pass. You didn't re-run everything from zero.
SBCL has pretty good type inference at a keytstroke (not complete, but useful, for a Haskell on top of CL see Coalton), it's fast, different implementations, stability…
> As a personal anecdote, I find professional Common Lisp programmers generally to be quite sensitive to matters of bloat. Some programmers go as far to chisel away needless cons cells (or allocation in general), and constantly be in touch with the compiled assembly—something frequently ignored by professional C++ programmers of comparable experience.
C++ and Rust programmers usually don't have to go down into the assembly. The language itself provides fine-grained control over when and how allocations take place, and how memory is laid out. It's a doddle to establish efficient allocation patterns and lay out memory to take advantage of locality of reference so you don't blow up your data cache. Lisp, not so much; in addition to the pointer-heavy semantics of Lisp objects you have the inherent nondeterminism of a GC. So not surprising that when a Lisp program stutters or hangs, the programmer has to go down to assembly level to determine the memory usage pattern that's causing the slowdown. With C++ or Rust, the memory usage patterns are (usually) obvious from the code itself.
> I would love to hear the "very sound technical reasons" explaining Common Lisp's obscurity.
* Lisp is dynamically typed. Yes, I know Common Lisp allows for type annotations, but the type system is just barely enough to enable speed hacks in the compiler; it is neither sound nor robust when compared to, say, fully parametric type systems (now commonplace in blub languages like C# and TypeScript). The lesson of the past couple decades or so is that strong, static typing is an unmitigated win, especially when building large systems; to choose dynamic typing in 2024 is to forgo early checking of all sorts of bugs.
* GC. You will incur a performance and memory consumption penalty using a GC'd language, an in this day and age with RAII and ARC, those are penalties you need not incur in order to safely, conveniently use memory within deterministic space and time constraints. With languages like Rust we can get the same kinds of advantages with static lifetimes that we do with static types. There is no need to defer object lifetime management to runtime; and to do so just wastes electricity and contributes to climate change.
* The syntax. Programmers are not just annoyed by all the parens. Today, they're used to being able to get a list of all the methods valid on a given object by typing '.' after the object's name. You need a certain kind of syntax to do that, plus static typing, and Lisp eschews both.
* Lispers wax nostalgic for the Lisp machines, which were pretty groundbreaking in their day... but as the sibling to my comment points out, the tooling for other languages has caught up if not surpassed the best Lisp development environments since. Microsoft has had edit and continue in its IDEs since decades ago; I believe Eclipse got something similar later. Modern refactoring tools. Project Lombok. There has been a lot of research done into how to improve the developer experience in practical languages. Meanwhile, Lispers are still recommending frickin' Emacs to beginners.
* Plus, the library support for practical languages like Java, JavaScript, Python, or C# -- or even C++ -- is just so much better than what's available for Common Lisp.
> You will incur a performance and memory consumption penalty using a GC'd language, an in this day and age with RAII and ARC, those are penalties you need not incur in order to safely, conveniently use memory within deterministic space and time constraints.
A distinguishing characteristic of languages like Lisp or Elixir is the metaprogramming. C++ and Java have nothing like their macro systems, and when an application grows and turns complex it's a very nice tool to keep developer facing code neat and readable. Usually it also keeps it more robust under change, there are fewer lines where typos or logic bugs can creep in when you write less boilerplate and use well designed code generation instead.
Then why are virtually all browser engines, high-performance games, and video codecs written in C++? Why is every massive-scale web application written in C++? The only one that comes close that's written in Lisp is QPX, and even then they had to write a lot of C++ because Lisp's I/O facilities just weren't up to the task of moving the huge amounts of data they had to move.
> C++ and Java have nothing like their macro systems
C++ has template metaprogramming, which covers like 90% of the cases where you'd actually need Lisp macros, and outputs code that's statically typechecked with deterministic memory and runtime constraints.
> Then why are virtually all browser engines, high-performance games, and video codecs written in C++?
Path dependence. There are only two lineages of widely used browsers today: Chromium/Webkit (coming from Konqueror, initial release 2000) and Firefox/Gecko (coming from Netscape, initial release 1994). Nobody's rewriting them.
The same goes for video game engines and other high-performance software.
The C++ bits in QPX were, as far as I know, relatively tiny elements for setting up memory mapped file, done this way because there wasn't good native interface in CMUCL.
The contents of that data file was generated by lisp batch job, too.
Kinda. Language-specific expertise should not be downplayed.
If you've spent a decade honing your craft writing C and assembly for exotic embedded platforms, you'll be a valuable asset for certain projects, but your experience won't translate to Scala or Haskell.
I think I meant more like "every language can have good programmers writing top quality code in it, even (especially) the blubs", not that a given good programmer will be able to immediately translate their expertise to a vastly different language or environment.
Consider an expert in C programming for embedded platforms, and an expert in Ada (or Forth) programming for embedded platforms. The surrounding expertise is similar, but the language-specific expertise shouldn't be dismissed.
Yes, and it's proven an awful suggestion; I'm surprised Google was promulgating it. Specs from the 1980s are filled with terrible design choices. Heck, equalp and equal are recommendations of the standard as the only two "structure-equality" options for hash tables -- and both of them are broken.
I disagree that it has "proven an awful suggestion." Proven how? Through what experience? I've found the one-semicolon comment placed at a suitable column (as recommended by the spec) leads to very clear and easy to read code annotations, even without syntax highlighting. Emacs does a fine job managing the formatting of these one-semicolon comments (e.g., setting at the right column, justifying/reflowing the comment over multiple lines, making sure the comment doesn't overflow your preferred source code width), and with syntax highlighting, they're even more patently visually obvious.
You seem to think that the choice is acceptable because "it works fine in Emacs". Language design decisions should not be based on the choice of editor. Indeed Google's own examples are not syntax-highlighted. And offsets won't do the job either: Google also recommends that lines not be longer than 100 characters, which makes it rather hard to guarantee that inline comments be offset by very much.
A single-semicolon comment is intended to be very terse and to-the-point. Longer inline comments are supposed to be double-semicolon dedicated to their own lines.
(let* ((head (cons nil nil))
(tail head))
;; This is a longer expository comment on the following
;; segment of code. Here, we efficiently append to a list by
;; keeping track of the tail node at all times.
(dotimes (i 10) ; RPLACD+SETF is an O(1) append
(rplacd tail (cons i nil))
(setf tail (cdr tail)))
(cdr head)) ; discard dummy cons
I don't think this style of commenting is acceptable because of my editor, but just that an editor like Emacs keeps it very easy to edit, organize, and reflow them. I have no shame in saying that I use my editor as an integral part to my programming workflow. :)
Look at the XSLT stylesheet this XML uses - it is doing parsing and indenting of Python in XSLT! I know XSLT is Turing-complete, but that is still highly unusual these days. Who knew that would even work in contemporary browsers...
Cant say I particularly like the way the page is generated under the hood.. but then I guess I have never been fond of XML (with the exception of forced HTML for websites)
Anyway, after digging around the page, I wonder if this covers it? Maybe you can add .xsl to it, etc.
https://github.com/ndw/org-to-xml
> Doug Katzman talked about his work at Google getting SBCL to work with Unix better. For those of you who don’t know, he’s done a lot of work on SBCL over the past couple of years, not only adding a lot of new features to the GC and making it play better with applications which have alien parts to them, but also has done a tremendous amount of cleanup on the internals and has helped SBCL become even more Sanely Bootstrappable. That’s a topic for another time, and I hope Doug or Christophe will have the time to write up about the recent improvements to the process, since it really is quite interesting.
> Anyway, what Doug talked about was his work on making SBCL more amenable to external debugging tools, such as gdb and external profilers. It seems like they interface with aliens a lot from Lisp at Google, so it’s nice to have backtraces from alien tools understand Lisp. It turns out a lot of prerequisite work was needed to make SBCL play nice like this, including implementing a non-moving GC runtime, so that Lisp objects and especially Lisp code (which are normally dynamic space objects and move around just like everything else) can’t evade the aliens and will always have known locations.
https://mstmetent.blogspot.com/2020/01/sbcl20-in-vienna-last...
https://lisp-journey.gitlab.io/blog/yes-google-develops-comm...
The ASDF system definition facility, at the heart of CL projects, also comes from Google developers.
While we're at it, some more companies using CL today: https://github.com/azzamsa/awesome-lisp-companies/