Metaclasses: Thoughts on generative C++

I’ve been working on an experimental new C++ language feature tentatively called “metaclasses” that aims to make C++ programming both more powerful and simpler. You can find out about it here:

  • Current proposal paper: P0707R1. I hope the first ten pages give a readable motivation and overview. (The best two pages to start with are 9 and 10, which probably means I need to reorder the paper…)
  • Initial intro talk video: ACCU 2017 (YouTube). This is the initial public presentation three months ago. Thank you to Roger Orr, Russel Winder, Julie Archer, and the other ACCU organizers for inviting me and for holding back the video until we could have the ISO C++ summer meeting about a week ago, so it could go live along with a report (herein) on the results of this feature’s first presentation to the ISO C++ committee. And special thanks to Ina and Arvid, the two audience volunteers who graciously agreed to come on-stage to participate in a live mini UX study. There’s a lot of subtle information in their nuanced reactions to the code examples; pay special attention when their responses are different or as their responses evolve.
  • “Incomplete and experimental” prototype compiler. The Clang-based prototype by Andrew Sutton is available as an online live compiler at cppx.godbolt.org, and as source at github.com/asutton/clang. It’s incomplete but can compile a number of the examples in the paper (see the paper for example code links). Thanks to Matt Godbolt for hosting it on godbolt.org!

Please see the paper and video to answer “what are metaclasses and why should I care?” If you’re the “show me code first, English later” kind of person, try the live compiler and these quick examples: interface, base_class, value (regular type), plain_struct (these links are also in the paper).

The rest of this post aims not to duplicate any information above, but to provide some context about the broader journey, and what I and others are attempting to accomplish.

A journey: Toward more powerful and simpler C++ programming

Phase 1: By using the existing language better

About five years ago, I started working on long-term effort toward making using C++ simpler and safer.

In the first phase, a small group of us—centered on Bjarne Stroustrup, Gabriel Dos Reis, Neil MacIntosh and Andrew Pardoe—pushed to see how far we could get with “C++ as it is” plus just a few well-chosen library-only extensions, with a particular goal of improving type and memory safety. Bjarne, Neil, and I first publicly reported on this effort in the two CppCon 2015 plenary sessions “Writing Good C++14” and “Writing Good C++14… By Default.” The results of that work so far have manifested as the C++ Core Guidelines and its support library GSL that adds a limited number of library types (e.g., span, now being standardized); and I led the Lifetime design in particular (available in the Guidelines /docs folder) which Neil and I and others continue to work on formalizing with the aim of sharing a “draft” static analysis spec later this year.

One of the goals of this phase was to answer the question: “How much progress can we make toward simplifying the existing C++ language with only a few key library extensions?” The answer as I see it turned out to be: “Some solid progress, but probably not a major simplification.” And so that answer led to phase two…

Phase 2: By evolving the language

Two years ago, I started to focus specifically on exploring ways that we might evolve the C++ language itself to make C++ programming both more powerful and simpler. The only way to accomplish both of those goals at the same time is by adding abstractions that let programmers directly express their intent—to elevate comments and documentation to testable code, and elevate coding patterns and idioms into compiler-checkable declarations. The work came up with several potential candidate features where judiciously adding some power to the language could simplify code dramatically.

Of those potential candidate features, metaclasses is the first major piece I picked to propose for ISO C++. [*] We presented it for the first time at the summer ISO C++ meeting earlier this month, and it received a warm reception. There was (rare) unanimous support for pursuing this kind of capability, but also some concern about how best to expose it and specific design change feedback the committee wants us to apply to improve the proposal. [**] We’ll work to include in a revision for the November standards meeting as we start the multi-year process of vetting and refining the proposal. So this is good progress, but note that it (only) means encouragement to continue the experiment and see where it leads; it’s far too early to talk about potential ship vehicles.

So do expect change: The proposal is still evolving, and it in turn assumes and builds on the static reflection proposal (P0578 et al.) and the compile-time programming proposal (P0633), both of which are actively evolving in their own right. Incidentally, one of the contributions of Andrew Sutton’s prototype metaclasses compiler is that it is implementing those other proposals too(!), since the metaclasses feature needs them. The aim is to keep the latest compiler and the latest P0707 paper in sync with each other and with those related proposals, but there will doubtless be occasional drift in between syncs.

What’s next

I’ll talk about metaclasses more in my upcoming CppCon 2017 talk this September, and Andrew Sutton will also be giving two CppCon talks about metaclasses—one about implementing them in Clang, and one about using them for a real project.

This is just the beginning, and we’ll see whether it all pans out and leads somewhere, but I hope you enjoy this exploration and I look forward to talking with many of you about it at CppCon this September.

 

———

Notes

[*] I actually brought a smaller piece from this same work to the committee at the previous meeting, the winter meeting in Kona: P0515 (consistent comparisons), which proposes adding the three-way <=> comparison operator. P0515 is only about a minor feature, and not one of the most important things that can help improve C++, so normally I wouldn’t have picked that piece to contribute first; but the committee was already continuing to actively discuss comparisons, so I cherry-picked it from my design work and contributed it since I had the design in my pocket anyway. Happily the committee liked what they saw and both EWG and LEWG accepted it, and it is now progressing well and on track to hopefully be voted into draft C++20 in the next meeting or two. Thanks to Jens Maurer and Walter Brown for the heavy lifting of writing the core language and library standardese wording, respectively, for that P0515 proposal.

[**] The committee’s design feedback was primarily about how to wrap up the transformation code: Instead of putting it inside a new “meta” class-like abstraction, how about wrapping the same code inside a compile-time function-like abstraction that takes an input meta::type parameter and returns a generated meta::type return value? This doesn’t affect the proposal’s basic engine, just the shape of its steering wheel—for example, we could change the first line of each example metaclass definition from the class-like syntax

$class interface {
    constexpr {
        // … basically same code …
    }
};

to the decorator-function-like syntax

meta::type interface(const meta::type source) {
    // … basically same code …
};

where the latter has the advantage that it’s easy to see that we’re reading one type and generating another type. Interestingly, I think this dovetails with the mini UX study in the video where most of the difficulty the UX participants seemed to encounter was in understanding the $class syntax, not the metaclass bodies and not later using the metaclasses to author new types.

But we’ll explore this and other options and validate/invalidate it with more experiments… and feel free to express your thoughts in the comments if you like one of these styles better, or perhaps another variation.

78 thoughts on “Metaclasses: Thoughts on generative C++

  1. I have to say, wow, that paper is one of the more interesting proposals I’ve seen recently. Please next time put the abstract in your blog post. Some handheld devices do not handle PDFs well. You gave us one word “metaclasses” and asked that we read ten pages out of a paper. Not trying to be critical. But there is so much happening in the C++ world these days that it’s hard to keep up. I usually would not open a paper unless I have a good idea what it’s about beforehand.

  2. This looks really awesome and promising! It looks like having metaclasses + concepts in C++ working together will give C++ the full typeclass feature from Haskell. Based on my limited understanding of Haskell typeclasses, a typeclass allows for 1) interface to define a common set of behaviors, and 2) the ability to provide default implementation of some methods using other methods. The first is fulfilled by concepts in C++. interfaces in Java, etc, but the second looks to be fulfilled by metaclasses. So if my understanding of the paper is correct, then we define an Eq metaclass and an instance of it:

    Eq Foo {
    bool operator==(const Foo &other) {
    // ….
    }
    }

    When compiling, the Eq metaclass will use reflection to see that operator== is defined but operator!= is not, and thus automatically define operator!= to return “! operator==”. (This is the behavior of Eq in Haskell). Is this a correct usage of metaclass?

  3. I am getting an “error: interfaces may not copy or move” from the example compiler, when doing this:

    Shape(const Shape&) = delete;
  4. This is truly amazing work. It will need some time getting used to it, as the change is quite fundamental, but it shows a lot of promises.

    If it is integrated in the standard, we will really be able to say “it feels like a new language”. My first impression is that it renders a lot of the current features obsolete. For example, templates. Do they still make sense with that powerful metaclass? I could write, for example, a metaclass vector that i would then declare simply as “vector { MyType; };”, reading the type to use for the vector in that way. Of course it feels like a misuse a bit, but it could also be more powerful that the template in the sense that i could have a vector with my own custom functions added to it. Am i correct? Have you thought about that?

    Which makes me think, can i implement the metaclass by using functions i know will be provided? For example, in the case of vector can i write:

    vector LimitedVector
    {
    int myType; // Type stored by this implementation of the metaclass

    void emplace_back(int value) { // “overload” the method of the metaclass
    if (size() < 10) // size() is expected to be provided by the metaclass
    emplace_back(value): // somehow call the emplace_back from the metaclass?
    }
    }
    };

  5. I agree this feature will rock!

    I only wondering if we could express object hierarchies using this feature, especially for GUI where hierarchies of widgets is essential part of the code.
    Something similar to QML or xml like WinForms should be expressible by metaclasses in some way too.

    Is this possible using C++ syntax only as input to metaclasses?

  6. Also, on my opinion, property declaration looks a bit strange with redundant curly braces. I understand that you are trying to stay within C++ syntax here.

    And to be complete, $interface could be inherited from other interfaces only. So we need possibility to check if a class implements some metaclass(es).

  7. Something’s not quite right about the plain_struct example. It would seem, based on comments, that I should be able to stick a public, non-virtual function in group_o_things and I can’t. Attempts to guess a fix based on the compiler output did not work.

    The problem appears to get worse with a static function. It seems to be included in the collection returned by “functions” but doesn’t have the members used in the code.

    I also would expect f.is_public() || !f.is_virtual() to be && rather than || to express “must be public and non-virtual”. I would expect this test to succeed with a non-public non-virtual or a public virtual.

  8. (Sorry everyone for the NSFW comment, that’s not welcome here, comment deleted.)

    @Thomas: Good point, I probably should have included the abstract in the post.

    @bm: Yes, you could do that. See also P0515 (https://wg21.link/p0515).

    @The Sim: Interesting case I didn’t cover in my code, in part because with metaclasses there is no reason to explicitly delete a generated function (see the paper and search for “=delete”). But this is just code, so instead of posting the answer I invite anyone to take a crack at it: How would you change the “interface” metaclass to not flag an error on “Shape(const Shape&) = delete;”? Hint: See the meta header here for the traits, including “deleted”:

  9. @Herb I tried changing the compiler.require clause to: (!f.is_copy() && !f.is_move()) || f.is_deleted(), but i hit a static_assert on line 690 of meta:
    static_assert(is_defined());

    I’d guess this is a bug?

    Are metaclasses going to be first class in the typesystem (can we pass them around to template metaclassfunctions or whatever)? If i want to create a class that’s both final and ordered, is there a syntax for that?

    ex:

    both foo : some_interface {
    int x,y;
    void some_function() override {}
    };

  10. gonna try that code block again

    both foo : some_interface {
        int x,y;
        void some_function() override {}
    };
    
  11. That is what I would like. However, this language is becoming such a huge mess of so many keywords and ways of writing stuff that I would just be happy if there was a new language that would break compatibility with C, and older C++ and a new language C+++ would exist with only these new good features.

  12. Since I see only one existing non-fully-enthusiastic comment, I’ll add my 2c. We don’t need yet another sub-language tacked onto the existing one. Just like Microsoft Windows is heading towards obscurity and financial ruin for Microsoft without fixing the obvious up-front extreme issues (like the scroll-away-when-you-click in Windows Explorer), but instead introducing a never ending stream of new very limited things and sabotaging the existing stuff, so it seems, is C++. I would like modules: they’re logically *missing*. I would like a way to write `const` and non-`const` versions of a member functions, that’s logically *missing*. I would like support for covariant features like a clone functions, and maybe that could also address the `const`+ non-`const` issue: that’s logically *missing*. These are things to fix in the language. Adding more on top, however, without fixing the glaring issues, that’s the Microsoft Windows way.

  13. Herb, nice of you to delete my comment, you can pretend that everybody loves your awful design that way…
    You can even claim at the ISO meetings that everybody you talked to loved it, like you pretended that your UX group in the video liked it.

  14. I could not enforce the “interfaces must only inherit from other interfaces” rule. Is there any way to find out the meta type of a type?

    I tried implement a lighter version: verify if the base classes are abstract and publicly inherited. It seems like I broke the compiler, because now it’s complaining that the virtual functions overriden in Circle are not virtual.

  15. It think there is a mistake in the fourth code example.
    Instead of

    compiler.require(f.is_public() || !f.is_virtual(),
                                 "a plain_struct function must be public and nonvirtual");

    it should be

    compiler.require(f.is_public() && !f.is_virtual(),
                                 "a plain_struct function must be public and nonvirtual");

    In short || should be &&.

  16. This looks like an upolished version of Template Haskell / Rust Macros.

    In both case, you can directly inject AST stubs where the meta calls are.

    You might want to look at it.

  17. @Bender: I approved your new comment above, thank you for not including NSFW material this time. I don’t mind contrary opinions if they’re reasonably professional, and not just trolling/abusive/NSFW.

    Note that at the ISO meeting people did unanimously like the capability, but as I said in the endnote above some had concerns about how the compile-time logic was wrapped in an abstraction, and the group directed to explore further whether $class was the right abstraction to wrap the compile-time code in; the UX participants in the video found it difficult too. Also, a vocal minority in the committee strongly want a syntax that does not include the $ character (not just for $class, but also for $expr reflection) because they have large code bases that use $ in source code that is not C++ code but is processed to emit C++; removing $ is an easy change at any time and we’ll just follow what the committee decides for reflection syntax (in fact, the alternative syntax I showed in the endnote above removes the need to write $). So further work is needed on those items, but fortunately none of it affects the core model.

  18. @Tyler: Ah, I think that static_assert (and the others in the file) are bugs. I’ll open an issue… Thanks!

    Also, yes there will be a way (but not yet in this compiler) to apply several metaclasses, including in the body of the class. I’m still attached to providing a convenient “replace ‘class’ up front” syntax for applying a single, or dominant, metaclass, for much the same reasons that Bjarne is attached to providing a convenient “replace a parameter type name with a concept name” abbreviated syntax to constraint function parameters with concepts — it makes the common case clean/natural/readable. But I’m not saying I’m wed to that a priori, it would just take a lot to convince me it’s a bad idea.

    @Alf: I hear you (though I’m not sure what you mean by the const/nonconst example since we can do that). It is a real problem that we are adding individual special-purpose language extensions; one benefit of a general mechanism like this is that it would replace the need for some (not all) special-purpose language extensions we’re actively working on now. And I appreciate focusing on “what’s *missing*” — that was part of my motivation too, because some of the very vendor-specific extensions you rightly complain about are repeatedly reinvented because key things are missing in the language — does the final section 4 of the paper perhaps sway you on that just a little?

    @Nicolas: I think you’re right! Thanks.

  19. @Adrien: Note that a metaclass is to specify a group of types having common characteristics, and simplify writing such a type, using generative code to automate/enforce the boring/required parts. Templates are about parameterizing a type.

    I don’t think vector itself wants to be a metaclass, but you could write a “container” metaclass with common container-related code/requirements to help write vector itself (and other container types) more simply and robustly as

    template<class T> sequence_container vector { /*...*/ };
    template<class T> sequence_container array  { /*...*/ };
    template<class T> sequence_container deque  { /*...*/ };
    // ...
    

    Does that make sense?

  20. @Alex, this is an interesting suggestion:

    > property declaration looks a bit strange with redundant curly braces. I understand that you are trying to stay within C++ syntax here

    First, my usual response has been that properties are an abstraction/scope, so that’s why they have braces in other languages.

    Interestingly, though, it’s your comment this time that made me think more about how properties in other languages (e.g., C#) do omit the {} if you use the defaults, and actually there is no reason I can’t do that in this proposal… in the enclosing type’s metaclass, the logic can just choose to treat a nested property class forward declaration the same as if it was a nested property class with an empty definition. (Note that in this example in the paper, the enclosing type’s metaclass just throws away the nested property class anyway and ‘inlines’ it into the enclosing type, so nothing is lost by allowing “property<int> x;” to mean the same as “property<int> x { };” here.)

  21. Removing the need for side compilers is the single greatest thing that could be done for C++. This proposal, and the proposals on which it depends, seems to remove that need nicely.

  22. I just watched the first 20 minutes of the video and “participated”. Programming experience is 15+ years. My reactions to the three slides can be summarized as follows:

    I’m really of two minds about this. Metaclasses seem to be a very powerful way to describe the expected structure of classes, resulting in better compile time checking and better, more meaningful error messages. It also reduces the amount of code that has to be written – but that is not a true benefit in each case. It can reduce boilerplate, but it also makes a lot of things very implicit, which is a very dangerous route imho.

    If you are a good programmer, this new feature is certainly incredible powerful and useful, but for weak programmers the implicit nature of the resulting code is really a problem. You increase the amount of things you have to keep in mind and have to know when reading/writing a piece of code that uses new specifiers introduced by metaclasses. The good programmer will look up the metaclass definition, and will be able to parse it (maybe). The weak programmer will just assume things and go by his/her gut feeling. This tendency is imho the key problem of “modern” programming languages like Python et al. It makes these languages attractive to weak programmers, because they “appear” to be simpler to use. But this appearance is just based on assumptions that are for the most part not backed up by actual knowledge of what is going on. As a consequence, you get pretty horrible code that is full of assumptions made by the programmer that seem ok, but are actually wrong.

    As much as I am intrigued by the power of metaclasses, I am also very skeptical about introducing a language feature that increases the potential amount of assumptions made by programmers.

  23. Very nice. I have use case in mind:

    In VTK (a 3D visualisation toolkit), objects have a last modification timestamp, that the setters update whenever the new value differs from the old one. As of now, this is implemented with macros:

    int SomeField;
    public:
    vtkSetMacro(SomeField, int)
    

    where vtkSetMacro generates a function like

    void SetSomeField(int x) { if (SomeField == x) return; Modified(); SomeField = x; }
    

    With metaclasses, we could have a trivial wrapper

    template<typename T> Settable<T> struct { T value; }
    

    and a meta class that generates a setter for each field that is a Settable.

  24. Regarding the use of the $ sign. Maybe a good alternative is the € symbol – though I am not sure if the symbol is readily available on non-EU keyboards. The € sign looks a bit like a C (class) that was cut open on the operating table, which one could use as a metaphor for reflection ;-)

  25. The other “obvious” candidate would be the § sign. As a law or regulation is something akin to a specification – which metaclasses essentially are.

  26. I’ve found the metaclasses (and surrounding compile time programming) talks very interesting. I think this feature will likely be overused tremendously to start with and eventually we will settle down as a community to a few common uses (apart from cutting edge library authors who will think if amazing uses).
    I would like to see if you could use metaclasses to write compile time html or similar technologies in a very natural way that could then be converted into C++ types.

  27. Very interesting way to extend the ‘language’. It could encourage better implementations, but the flip side is you would’ve given people a bigger shotgun to shoot themselves in the foot.

    Would you expect there to be some ‘standard’ meta classes defined for things like interface, value, raii_value etc.

    My main concern is the difficulty in managing user defined abstractions which materially can affect code written elsewhere. e.g. updating the definition of meta::type flintstone would propagate to all class barney, class wilma etc.

    Probably no worse than base class dependencies, but another place for bugs, subtle interface issues and surprises to hide.

    That said, seems to elegantly extend in some areas of C++ weakness and may reduce reliance on the pre-processor.

  28. There has been quite a lot of C++ proposal where I have been wondering where I could apply them.
    Concepts, ok it will help but not sure how often. Modules yes I can see how that would help too.
    But this one is the first one where I could see how to solve problems that have been creeping in our applications and framework over the years by making the code cleaner, a lot simpler AND safer.

    Just one request, can you make a prototype version in Visual Studio 2017 ;)

  29. I appreciate that it is probably too early to tell, but what chance do you think that some form of metaclasses will be in C++20?

  30. $ operator syntax overlaps sigil from PERL for strings. As this can improvise WRL like COM dialects radically, when using /std:latest flag VC++ UWP developers must get this ASAP. This must be technical specification at least, as some dependencies will hold up until C++ 20.

  31. To be more constructive. I would be happier if you didnt introduce new syntax. Why cant we have a special code block costexpr and inside regular c++. The reflection can be stored in some well known variable with name access with regular methods. Injecting members not with an arrow but rather a method… AddFunc… etc. Java has simmiliar mechanism when generating code based on annotations.

  32. I know that C++ is sometimes considered an “expert’s language”, and that this feature renforces that consideration, but it also opens the door for expert senior programmers to give themselves and their juniors a construct with rules that they not only define, bht can change if something seems illogical later, and also have code, not English, for any user of their metaclass to check to see if an assumption is true.

  33. Also in java you can chain these code generations … therefore i wouldnt put it into a new block ut rather have it ordinarry code that gets chained…

  34. Great work! There is definetly no reason not to push as hard as possible to make this go into c++20
    Is there a timeplan when this might become an TS?

  35. @Martin: Yes, that’s the direction we’re exploring. The current paper doesn’t mention it but the current design and compiler already allows constexpr { } compile-time code blocks in regular classes too.

    @Jonathan: Good news, it is actually not too early to tell — there is zero chance of this being in C++20. :) Besides that this needs further development and trying out at increasing scales in more domains, the door for C++20 features closes in two years and this depends on two other features (reflection and compile-time programming) of which only one (reflection) is even yet in progress toward becoming a draft TS itself.

    @Paul: Yes, since it’s “just code” you can put it in namespaces, libraries, etc. just like any code — including putting common ones in namespace std.

    And yes, reducing reliance on the preprocessor (and template metaprogramming) is a primary goal. TMP is “meta” too but templates are “accidentally” meta :) whereas the point here is to design and leverage first-class compile-time programming.

  36. Question: throughout the example implementations, there are variables defined to do calculations during the constexpr code evaluation (compute the number of objects, etc.) Some of these variables, although not all, are then referenced in the code *outside* the class-level constexpr block (for example, in the bitfield example). Is it your intention that these will be visible to code outside the constexpr block, but not injected into the protoclass (as they would have been if they were declared outside the constexpr block)?

  37. Amazing. I hope it will get to c++23, it would be so nice to use with concepts. And I’ll finally be able to compute semi–complex stuff without workarounds in those

    constexpr

    blocks. <3

  38. This almost seems like it could be used instead of concepts. At first I was thinking of it as concepts specify requirements on a type, and a metaclass modifies a type to make it adhere to a concept. Was thinking it would probably be common to have concepts and a matching metaclass in pairs. How is the name collision between concepts and meta classes? Does it cause problems to have a concept named ForwardIterator and a metaclass called ForwardIterator as well? Since metaclasses have requirement clauses, could a metclass implicitly create a concept as well so you don’t have to create a separate matching one? Then you can get the terse concept syntax with your metaclass for free. void foo(observer_ptr arg);

  39. About properties. While metaclasses can generate ordinary methods from property declarations, what about caller side? Does user of the class have to use generated methods instead of property name to get or set property? If so, why bother to implement properties at all?

  40. A question regarding the basic_enum to_string example:
    I’m assuming that a flag_enum instance (class?) would match the function – since a flag_enum can have values that aren’t part of the enum itself, does this mean that a flag_enum isn’t a basic_enum? This could be fixed easily enough with a more constrained to_string function, but it feels kind of weird that the more general function doesn’t work. Of course the to_string of the original could be (inefficiently) rewritten to work but that, again, doesn’t feel right.
    Am I missing something basic or reasoning about this incorrectly?

    That aside, this a really exciting feature (and CRTP wouldn’t be needed anymore!)

  41. I missed the code that defines property. In particular, I would be interested in the part that creates

     T get_xxx() { return value; }
    

    Is that code available somewhere? It requires constructing names out of thin air. Does the prototype by Andrew Sutton already support that?

    Afaict, this could mean that we could finally get variadic composition.

  42. While I like the direction this is going, I think that that there is a bit of a mix-up between concepts and transformations in here. This can be seen in the definition of .is which assumes that the result of a metaclass transformation is a kind of concept and a second application of the same transformation is a No-op. But that is not the case for examples like the one that adds a new overload of each function with an additional int parameter.

    The function syntax for metaclasses embraces that more. And why stop there? If the transformation is a function, then .as can be seen as a constexpr function call and maybe should have the function call syntax with types as parameters? It could then also have additional type or non-type parameters (like flags or similar) and transformations could call other transformations without special syntax that has an implicit call order.
    And finally maybe concepts should be the same syntax, functions returning bool for concepts and functions returning meta::type for transformations.
    If metaclass transformations result in a type that conforms to a concept, the function could return the name of that concept instead of meta::type (although the question is when the concept check should be done, some kind of cast maybe?).

  43. Commenting on “C++ Core Guidelines” where it says: “Note Capitalize the names of your user-defined types to distinguish them from standards-library types.” – You guys are ‘hypocrites’!!! You recommend something that you DON’T do! Why are standards-library types lower-case? Because nobody likes hitting the shift key. No One’s gonna follow that. They’ll just do everything lower-case. I’m commenting on this because I was following the Google Style Guide for C++ but I DON’T like it because of all the shift key usage. So, I’m refactoring my code to be all lower case so it’s easier to type. Also, I switched around the underscore key so I DON’T have to hit shift to get the underscore. THAT IS LAME

  44. @kpfleming: The original intention when that code was written before constexpr blocks was that they be visible, yes. However now those variables should be injected into the class, or the constexpr block extended, if they are going to be used. I will update the examples for the next revision.

    @Eric: This isn’t instead of concepts. See Figure 1 — Metaclasses complement (and rely on) concepts and reflection, which are about querying capabilities – based on “does this expression compile” and “does this member/signature exist,” respectively. Metaclasses are about defining types – participating in interpreting the meaning of source code to generate the class definition. Also see section 2.7.

    @Gregory: It gives you the “usual” property syntax modulo writing (). Metaclasses can generate what the language already lets you write, and dropping the () would be an additional language extension (orthogonal to metaclasses).

    @Adam: I think it’s consistent — this to_string is not print all the possible values of the enum, it’s printing all the enumerators.

    And yes, I do need to add a note/discussion that one way to think of this is that it should replace all uses of CRTP, which basically was an early approximation. Curious how this technique is recurring, huh? :)

    @Roland: I didn’t write out the property code but described what it does in the prose. See some of the other examples in the paper for generating functions, such as safe_union.

    @Andreas: Yup, that’s a good summary of benefits of the function style syntax.

  45. Here’s an idea, define a meta-class:
    “fake”: all functions from inherited classes /return default values/are erased(compile time error when called)/…/
    “mock”: /…/
    “test-fixture”: /…/
    “tests”: /…/

    Wonder if/how that will work? :-)
    O my … 6 years to wait for this feature …
    (I wonder if meta-functions are plausible?)

  46. I’m seeing your example of properties.
    Some notes:

    1. You wrote:
    This metaclass implements the following requirements and defaults:
    • Each function’s name must begin with “get” or “set.”
    • T must be copyable.
    • Apply any other Qt property rules

    It’s not true. Getters/Setters may have arbitrary names. We could have read-only property with const ref return type, so it could be non-copyable.

    2. Anyway, I’m just trying to build up more compete example of the property metaclass.
    Using your idea of signal/slot as helper types I came to the following:

    Qt property declaration:
    Q_PROPERTY(bool focus READ hasFocus NOTIFY focusChanged SCRIPTABLE true REVISION 3)

    C++ property declaration:
    property focus {
    getter hasFocus() const { return ::HasFocus(m_widgetHandle); /* ??? */ }
    notifier focusChanged();
    bool scriptable = true;
    int revision = 3;
    };

    Where getter/setter/notifier are helper types to mark or distinguish property attributes (READ/WRITE/NOTIFY etc.)

    Note that line with /* ??? */ is lexically not correct because uses m_widgetHandle from outer class.
    Is this (using outer class members or similar “incorrect” code) will be forbidden?

    I don’t like resulting property metaclass because we express similar things (property attributes like type, name, getter, setter, scriptable etc) using very different C++ constructs (like type name, tag types, return type or names of data members).
    And more over we use redundant words (like “int” for revision) just because of C++ syntax requirement.

    I consider metaclasses as attempt to introduce “programmable” declaration statements into C++.
    And you want to limit it by existing C++ declaration facilities like class/subclass declarations, data and function members.
    It works in simple cases like $interface where you introduce just replacement for “class” keyword.
    But more complex and useful declarative statements looks unnatural at my opinion.

    I’m programming C++ quite a long time (>10 years) and noticed that C++ is too imperative (and thus more verbose) in some cases. We had to invent DSLs (Domain Specific Language) to eliminate writing a lot of required but “stupid” code. Just like interfaces and value classes do in your proposal. You could see as example my project https://github.com/lexxmark/QtnProperty.

    So I wonder at some time metaclasses will replace all these DSLs (I’m consigering Qt classes and WinRT as DSL too). But I think C++ declarative facilities not enough to cover all cases. We need more unified and powerful declarative syntax like JSON or QML.

  47. Herb, it’s a really cool proposal! We can then create DSLs as we like, implemented as a standard c++ library!!

    Would it be possible to implement features like memory safety, thread safety equivalent to how Rust provides?

  48. While this might reduce amount of macros/boilerplate used to generate some classes, my overall impression of defining “interface” and alike is basically this:

    #define BEGIN {
    #define END }
    
  49. Note, that it’s hard to get the difference between creating a class and creating a member (it’s actually specific to the previous definitions)

    vector x{}
    vs
    property y{}

  50. You didn’t say anything about namespacing, it probably better be smth like std::interface etc which looks less nice

  51. Consider this case with a hierarchy of interfaces and implementation classes:

    interface IA {
        void foo();
    };
    
    interface IB : IA {
        void bar();
    };
    
    
    class A : public IA {
    public:
        void foo() override {}
    };
    
    class B : public A, public IB {
    public:
        void bar() override {}
    
        // but now we have to also provide foo override introduced by IB,
        // even though IA has been implemented by base class A
        void foo() override { A::foo(); }
    };
    

    Interfaces would still not act as in other languages, not without an ability to do virtual table fixup, or enforce virtual inheritance with that meta code.

  52. I know that it has been said that these features will not make it into C++20, but we do have a Clang version that implements some of it. When could we expect Visual C++ to have it?

  53. @Alex:

    You wrote: “I don’t like resulting property metaclass because we express similar things (property attributes like type, name, getter, setter, scriptable etc) using very different C++ constructs (like type name, tag types, return type or names of data members).”

    Sure, that’s a reasonable point for the example I gave. The actual Qt property will be more elegant than my initial strawman, which was mostly to demonstrate capability. The flexibility we have gives lots of ways to design a property metaclass… for example, if we changed the requirements to (this again is just a strawman that approximates the published Qt rules, see http://doc.qt.io/qt-5/properties.html):

    • We recognize as a “get” any function that is const and returns T or T const&.
    • We recognize as a “set” any function that is non-const and takes exactly one parameter of type T, T&, or T const&.
    • We recognize as a “notify” any function declaration that is a signal function in the same class.

    then you could write this in your example:

    property<bool> focus : scriptable, revision(3) {
        bool hasFocus() const { return ::HasFocus(m_widgetHandle); /* ??? */ }
        void focusChanged();
    };
    

    or, since we can also reflect over attributes:

    property<bool> focus [[scriptable, revision(3)]] {
        bool hasFocus() const { return ::HasFocus(m_widgetHandle); /* ??? */ }
        void focusChanged();
    };
    

    or many other variations.

    The major point here is that you can match Qt rules pretty much exactly, or WinRT rules, etc. plus have strong expressive power to express an opinion on what your users’ code would best look like — but never straying far from the C++ grammar, since it has to match the C++ grammar, so we couple great flexibility with constraints that help ensure the result still “looks like C++” even though it is expressing things C++ could never express before. As with any (meta)class design, taste comes into play, and the (meta)class author has the flexibility to express their taste as well as express their core requirements and functionality. — Just as with classes, metaclasses offer flexible expressive power to let the (meta)class designer express what they want and make their (meta)class usable in the way that they want. And so I fully expect that we’ll get “Effective C++ Metaclass Design” design guidance just as we have “Effective C++ Class Desigh” guidance (i.e., “just because you can doesn’t mean you should”); we are just now learning those style rules for metaclasses, just as we did with classes in the late 80s and early 90s.

    And regarding the use of m_widgetHandle in the property getter definition, the short answer is “yes that should work” and the hint as to why is “same as in-class member function definitions and non-static data member initializers.” [*]

    Does that help to address some of your concerns, even if it may not yet address all of them completely?

    [*] Far too briefly, but enough that you’ll get the idea if you’re familiar with compilers: In-class member function definitions and member initializers already effectively require two-pass compilation of the class definition — it’s the only place in C++ that requires two-pass today, so that you can refer to something that is not declared until later in the source file. Actual compilers like GCC and Clang implement this by effectively storing up the tokens of the function definition (doing just brace matching) or default initializer (looking for the ;) and then not actually even parsing the contents of those things (never mind doing semantic analysis, name lookup, etc.) until after the class is fully defined. — And that same rule should apply here, so that the function body above would be parsed after the enclosing class is fully defined, which is after its metaclass has run and moved the functions there so they’re in a scope where they can refer to each other naturally. — Two things to note about this: (a) it’s just the same rule C++ already has today for in-class function definitions, and (b) it just does what the user expects in other languages where properties are nested scopes so I don’t expect this to be surprising. Having said that, we absolutely won’t know until we validate the above by implementing it in real compilers and then testing it with real users. That’s real work and design risk, and I’m not minimizing it, just saying why technically we think it’s sound at this point.

  54. @soulformachine: If you want such a system, you can use a metaclass for your derived classes that recognizes interfaces and inherits from them virtually by default. Would that address your concern?

  55. @HerbSutter: I guess that would work, I like the fact that the details would be hidden away behind $class interface and say $class implementation, but what worries me is that this is just a way of providing a nicer syntax for the existing system and not a tool which can change the system so that for example in this case a solution can be produced which is more efficient than virtual inheritance (if there is one). Also it would not be possible to provide interface behavior such as in C# which can provide interface function implementation through a base class which is unrelated to interface but has a function with the same signature:

    interface IA
    {
        void foo();
    }
        
    class B
    {
        public void foo() { }
    }
        
    class A : B, IA
    {
            
    }
        
    public class Program
    {
        public static void Main(string[] args)
        {
            IA a = new A();
            a.foo();
        }
    }
    

    Also, it would still be possible to derive a class from an interface, which would produce different behavior, unless there is a possibility to say “this $class is final unless the deriving $class is implementation”.

  56. Isn’t the interface example which appears in the proposal and which is being discussed here a blatant violation of your own guidelines regarding the “non-virtual interface idiom” (http://www.gotw.ca/publications/mill18.htm), according to which virtual functions in C++ should usually be private and public functions should usually be non-virtual?

  57. @soulformachine Consider that a metaclass can compute anything that is reflectable and constexpr-computable, (although the prototype currently only supports a small subset of that) and all the rules you’d like to express are computable at compile time using reflection… and see also the paper’s safe_union example for a type that completely replaces the declared implementation.

    @Christian The non-virtual interface idiom (NVI) is a recurring and useful idiom, especially for classes that also have implementation, not a do-always rule. Pure public interfaces are a perfectly valid pattern — it’s just until now they have been written entirely by convention without any guard rails where the compiler can help enforce we followed the intent and that it stays that way under maintenance.

  58. Hey, I would like ask a few things about how this proposal and how it will progress in the future.

    I am currently checking out the cppx-compiler and it seems that with the current state of libcppx, it doesn’t provide all features which are needed to eg. imlement the safe_union metaclass.

    The interesting thing here is that members can be removed, injected and iterated through. I guess this isn’t implemented yet? Or has there been changes in this regards?

    I am also wondering though, if you plan to expand the “removal behavior” to more “iterable reflections”, something like removing all base classes with “$metaclass.bases().clear()”.

  59. I am wondering though, how is clear implemented? Would this require a “constexpr vector” implementation? If so, how is the “constexpr vector” implemented?

  60. I’ve watched your presentation just yesterday, without any knowledge of what it would be about.
    Beside my eyes shined even more at each new slide, it’s just like you’ve answered all C++ issues with this killer feature. I’m C++ developers for 10 years now, and metaclasses bring to life all features I could dream of for C++.

    For the examples you mentioned in your presentation, my only feedback is: yes we wanted that because they were missing features that are present on other languages. The good point is that we will be able to make our own semantic by ourselves. Taking the property example, there are plenty ways to design a property like object with more or less features. And instead of waiting several years for the committee to validate a proposition, we will have the tools to implement it directly.

    Here are few “issues” that could be solve using metaclasses:
    – dot operator dilemma, for any forward wrapper like reference_wrapper, one of the main issue using dot operator is: does it grant you access to the wrapper functions? the T functions? or both? With a reference_wrapper metaclass we don’t care anymore since we will have the choice to implement it like we want. In this case: for all functions of T, add a function inside reference_wrapper which just forward the parameters to the T function. But for another kind of forward wrapper, we can imagine different rules.
    – in the same idea: we could wrap a type T to be always contained inside shared_ptr transparently.
    – automatically generate wrappers to another language.
    – automatically generate boilerplate code (factory, etc…)
    – override default historical choices of the C++ like everything is mutable and you have to explicitly apply const whenever it’s needed, compared to Rust where everything is const by default.

    To answer people saying that it will increase the complexity and readibility of a code because it introduce potentially an infinite number of class behavior aside “class” and “struct”, I would say it’s already what templates do except that metaprogramming using templates can become really hard to read and maintain if you’re not an expert. I hope metaclasses will allow us to do more metaprogramming, more easily, more readable / smoother syntax, with less painful error messages than templates.

    Regarding the syntax $class vs a function call: I really prefer the $class syntax because it looks more like a declarative thing. To me, a function is made for executing something (whatever it’s done at compile time or runtime). But here, the metaclass is something that we declare, not something that we compute.

    @Herb
    I got few more questions:
    – Do you expect metaclasses to compile (far more) faster than templates? I guess yes.
    – I don’t know if there are proposals for generalized attributes like C# attributes (inside []) that can be used to flag any class, function or data member – but can we imagine that we could some C#-like attributes to have even better control over what can be generated by metaclasses? For example: tell which members have to be serialized or not.

    Anyway, I definitely cannot wait to gets my hand on. I have many ideas I would like to implement that would benefit the simplicity of metaclasses compared to templates.

    And please, make it available through Visual Studio soon enough :)

    Thanks for this brilliant and revolutionary idea. Looking forward for any updates.

  61. I absolutely love this proposal. When I was done reading it, I was left thinking that the only thing it needs is to be paired with a niladic function proposal like N1611 (Implicitly-Callable Functions). The combination would allow for properties like in other languages, by generating a niladic function from a metaclass that returns a proxy object that provides any appropriate function overloads. It also turns all the difficult questions about property semantics into user-defined code, rather than trying to normalize specific semantics in the core language.

  62. How about using “$this“ instead of the name of metaclass
    or like this:
    “`
    constexpr class interface {
    constexpr {
    compiler.require($this.variables().empty(), “interfaces may not contain data”);
    }
    interface(const decltype(this)& that) = default;
    };
    “`
    Where the type of “this“ (if its used from “constexpr“ block or metaclass) be the type of the class which inherits it.
    Wouldn’t this simplify things?

  63. Hey Herb, I thoroughly enjoy your talks (in video format on YouTube). I’m a full time C# programmer and a part time C++ programmer (mainly embedded systems), watching your C++ talks has essentially pushed me to explore C++ fairly recently…

    On the face of it, I like Meta Classes, but the thought that the word “interface” could mean many things for folks is not really a good thing it is? I addressed this in your talk, but I’m not convinced your argument is valid. Having a standardized meaning of interface has a real value for the language. Giving me to ability to doing other things with MetaClasses is great, but something’s should be addressed by the language as a spec BEFORE we have a clutter of the meaning of “interfaces”.

    So in the case of interface you’re saying this metaclass will be in the language. Ok, but I can have an interface metaclass in a different header file, include that header instead of the standard and now my entire program changes or has compile errors? Can I define my own metaclass for “class”? In C# I don’t use extension methods for similar reasons.
    1. The meaning could change from file to file or project to project
    2. When I switch projects or jobs, what I understand an interface will likely not have the same meaning
    3. When I see “interface” on stack overflow I wouldn’t know what that really means.

    If standardization takes time then well, maybe it needs to be speeded up? :).

  64. One idea to mitigate that could be: Apart from namespaces that can be used to scope them, all meta class names in any user library can be made to require to start with an _ (e.g _interface), except those that are part of the standard library. Much like string literals.

  65. Hi,

    I assisted your Qt Keynote this morning. I like the idea of metaclass a lot, especially in sense that
    it complements very well the notion of templates.

    Concerning the syntax, my first impression was apparently the same as the feedback given
    by the cpp committee: it introduces new symbols and
    keywords (somehow instead of “struct“ we have something else), while at the end what
    we want is to transform a type through a function at compile time. The
    transformed type remains a type, so it would make sense to keep calling it a “struct“ or
    a “class“.

    It also quite similar in principles to `bind` for functions: we create a new function from an existing one,
    but again `bind` returns a new function.

    The committee suggestion to have something like

    meta::type interface(const meta::type source) {
        // … basically same code …
    };
    

    is quite similar to what I was thinking during your keynote. The primary design I have in mind is the metaclass
    concept from Python, which has been slightly modified for python3 but is quite effective and seems
    to now have proven its design:

    * a decorator indicates that a class is a metaclass, and the final executed class will be a transformed version
    of it
    * the fact that a class is meta indicates the compiler/interpreter to apply the meta function on the initial type to get the final type.
    This is the metaclass protocol.
    * the meta function that is called with the reflexion of the class being decorated is an entity that has the same type
    of construct as any other (in python: a class member function, or a free function)
    * the meta function is evaluated at compile time (in python: when the interpreter sees the declaration of the class)

    For the C++ counterpart of that, I believe not introducing any new notion or constructs (`$class`, `constexp{}`)
    would have a better chance to convey the idea of metaclass in a proper way.

    I would suggest this:

    1. the meta function can be a free constexpr function, or a function object with a `static constexpr operator()`.
    Calling this function, or the member function, with the right parameters, is the first step of the meta protocol
    between the type being transformed and the transformation function.
    The advantage would be
    to compose the meta function from eg. a template and member constexpr variables in an easier way for a developer

      meta::type metaFunc1(const meta::type source) {
      };
    
      // here we could imagine meta2 behaving **exactly** as any struct, but with a 
      // required interface to be applied by the compiler
      template <class T>
      struct metaStruct2 {
    
        constexpr auto operator()(const meta::type source) {
          // here the returned type is a new type that is defined nowhere in 
          // the source code, auto make sense
          auto modified_source = source.add_member_function(); // each of your "compiler" command returns a new type or a compile time test
          auto modified_source2 = modified_source.delete_member("x");
          return modified_source2;
        }
      };
      

    2. the decorator would simply be applied with C++ attributes, with the possibility of chaining them. Then they would be applied in order
    Something like

      class [[meta(metaFunc1, metaStruct2<4>)]] 
      some_class_before_transformation{};
      

    In this case, we would apply first metaFunc1, then metaStruct2, etc

    Another possible syntax would be (since attributes can be ignored or are considered as language extensions)
    to reveal the fact that we are applying a function transforming a type that is being declared:

      class metaStruct2<4>(metaFunc1(some_class_before_transformation)) 
      {};
      

    which seems to me clearer and less ambiguous on the orders. It also reveals the fact that we are transforming a type that
    has not yet been fully parsed. This does not really clash with the notion of `final` that you mention in your paper:
    `final` would be an attribute of a returned type, that would prevent its “compiler casting” to “meta::type“ and stop
    the evaluation of the meta functions.

    3. it would be possible to use type alias (w/wo template) with the meta functions:

      using my_augmented_type = metaStruct2<4>(metaFunc1(some_class_before_transformation));
    
      template <int Value>
      using my_augmented_type2 = metaStruct2<Value>(some_class_before_transformation);
      

    One of the possibilities would be to enable or not eg. “some checks” depending on a boolean

      template <bool B = false>
      struct metaStruct3 {
          constexpr auto operator()(const meta::type source) {
            return source; // defaults to identity
          }
      };
    
      template <>
      struct metaStruct3<true> {
          constexpr auto operator()(const meta::type source) {
            // instruments source
            return modified_source;
          }
      };  
      

    Let me know if any of this makes sense :)

    Best,

    (apologies if I posted twice)

  66. Sorry for the delay catching up to comments… I think this batch of replies brings me up to date…

    @Paul: Right, the prototype compiler is still very incomplete and just does the first handful of examples I linked.

    Re removing things: Based on the current feedback at last week’s ISO C++ meeting, we will actually be pursuing a functional style along the lines of the third option in section 5 of the updated paper at https://wg21.link/p0707, where the idea (among other things) is to treat each class as read-only… that is, the source-written class is read-only input, every “metaclass” is defined by a compile-time function that takes a read-only meta::type and builds up a new meta::type which is read-only after it is returned, etc. So think of it as a succession of additions. To remove something, you would copy “everything except that something” into a new meta::type.

    Oh, and yes the committee is actively pursuing constexpr vector. However, this proposal does not rely on that.

    @Tony: I’m hopeful that compile times will be competitive, and better than expressing the same things more convolutedly using today’s TMP and/or extra compilers.

    @moonshadow: Lots of syntax variations are under consideration. I don’t think we’d reuse “$this” in particular beause that would already have a meaning (to reflect on the ‘this’ parameter). But see section 5 of the current paper which mentions an example of where we’d need to say “this_class” or similar.

    @Shiv Kumar: Actually “interface” would be just another type, which we already disambiguate using namespaces. The idea is that if we standardized “interface” it would still be a library, not in the language, but put in namespace std:: along with all the other library types, and then if you want the standard one you’d namespace-qualify “std::interface Shape { /*…* };”.

    @raen: First, thanks! Specific comments follow…

    Re syntax: Agreed, the committee last week gave guidance to steer toward a functional style (see earlier in this reply) which among other things removes much of the need to reflect and therefore to even use the strawman $ notation.

    Re “1. the meta function can be a free constexpr function”: Yes, that’s the direction the committee just encouraged.

    Re “2. the decorator would simply be applied with C++ attributes, with the possibility of chaining them”: I probably wouldn’t use attributes, but in general yes that’s one of the alternatives — see the alternatives showsn in the new section 6 of the paper at https://wg21.link/p0707.

    Thanks all,

    Herb

  67. Thank you Herb for your reply. I see your point, however I still feel we’ll be having conversations like, “is that your interface, my interface or std::interface” :)). The ability to be able to define one’s own keyword (C# vocabulary) is extremely powerful (to say the least) but also a maintenance nightmare, I feel. Further, as regards all of the other things one can do with Meta Classes – for those of us not hardcore C++ programmers a huge learning curve (I feel). Maybe it’s just me.

  68. This looks great! My only problem is understanding where the capabilities of metaclasses (or maybe even metafunctions) stop. Would all of the features below be possible?
    – generating serializer functions automatically (e.g. to_json() or to_protobuf())
    – generating wrappers for other languages like python avoid using awful tools like swig
    – using different defaults for function arguments (e.g. everything const like in rust)
    – solving the nightmare of writing multiple overloads for constructors with string arguments (see Nicolai’s “The Nightmare of Move Semantics for Trivial Classes” cppcon talk) and algorithm interfaces (see David Sankel “Choosing an Abstraction” cppcon talk).

Comments are closed.